Streaming large assets

Archive.read needs a buffer as large as the asset. For a 1 MiB texture that is fine; for a 900 MB video it is not. entryReader reads an entry through a fixed buffer regardless of how large the entry is.

const buffer = try gpa.alloc(u8, zpack.stream_buffer_len);
defer gpa.free(buffer);

var entry_reader = try archive.entryReader(entry, buffer);
const r = entry_reader.reader();

r is a std.Io.Reader that reports end of stream at the entry’s last byte rather than the file’s, decompressing on the way when the entry is deflated.

Sizing the buffer

Entry method	Minimum
deflate	zpack.stream_buffer_len
store	Any size, even smaller than the entry

pub const stream_buffer_len = flate.max_window_len + 4096;  // 69632 bytes

A deflated entry needs flate.max_window_len (64 KiB) for the decompressor’s window - twice deflate’s 32 KiB back-reference history, so matches resolve without shuffling the buffer - plus 4 KiB to hold compressed bytes read from disk. That is where 69632 comes from. Anything smaller is error.BufferTooSmall.

The asymmetry is real and worth knowing - example 04 demonstrates both in the same run:

entryReader with a 1 KiB buffer: BufferTooSmall
stored entry through that same 1 KiB buffer: 4096 bytes

If you do not know the method up front, size for stream_buffer_len and it works either way. A larger buffer does not change correctness, only the number of syscalls.

Hash as you go

entryReader verifies nothing

read checks the entry’s hash before returning. entryReader cannot - the hash covers the entry in full, so it is unknowable until the last byte has gone past. If you stream, you check it yourself.

The pattern, taken from example 04:

var hasher = zpack.format.Hasher.init(zpack.format.hash_seed);
var total: u64 = 0;

while (true) {
    const chunk = r.peekGreedy(1) catch |err| switch (err) {
        error.EndOfStream => break,
        else => |e| return e,
    };

    hasher.update(chunk);
    try consume(chunk);        // upload to the GPU, decode, write out...
    total += chunk.len;

    r.toss(chunk.len);
}

if (total != entry.size) return error.CorruptArchive;
if (hasher.final() != entry.hash) return error.HashMismatch;

Check the length as well as the digest. An entry that stops early would otherwise pass a hash computed over the partial prefix - the length check is what makes truncation detectable.

peekGreedy(1) hands you whatever is buffered rather than a fixed-size chunk, so you process 64 KiB at a time instead of byte by byte. toss advances past what you consumed.

What it actually costs

From example 04, a 1 MiB texture through a 68 KiB buffer:

asset   1048576 bytes (116281 on disk, deflate)
buffer  69632 bytes is 15x smaller than the asset

streamed 1048576 bytes in 32 chunks
largest chunk held at once: 65536 bytes
hash matches the one recorded at pack time

peak memory for the contents: 69632 bytes, not 1048576

Peak memory is the buffer, not the asset. The same 68 KiB streams a 900 MB entry.

When to stream and when not to

Stream when the asset is larger than you want resident, when you are piping it somewhere else anyway (a GPU upload, a decoder, a socket), or when you only need a prefix.

Do not stream for small assets you are going to hold in memory regardless. read is simpler, verifies for you, and for anything that fits in your buffer there is nothing to gain.

Reading just a header is the case where streaming wins outright:

var entry_reader = try archive.entryReader(entry, buffer);
var magic: [8]u8 = undefined;
try entry_reader.reader().readSliceAll(&magic);
// Stop here. The rest of the entry is never decompressed.

Stopping early costs only what you read. There is no way to seek backwards in a deflated entry, though - the history window is built as it goes. Read forward, or use read and index into the result.

Alternatives to hashing every time

Hashing on every stream costs a pass over the data. Two ways out:

Verify once at startup, then stream freely:

try archive.verify();   // every entry, once

Rely on the length check alone where a truncated read is the realistic failure and bit rot is not - a local file you also wrote, for instance. That is a judgement call, not a recommendation.

Do not copy an EntryReader after calling reader()

reader() binds the decompressor to the struct’s own field. Once called, the struct must not be moved or copied - see EntryReader.

Streaming into a file

std.Io composes, so writing an entry straight to disk needs no loop of your own:

const out = try dest.createFile(io, "extracted.raw", .{});
defer out.close(io);

var write_buffer: [64 * 1024]u8 = undefined;
var out_writer = out.writer(io, &write_buffer);

var entry_reader = try archive.entryReader(entry, buffer);
_ = try entry_reader.reader().streamRemaining(&out_writer.interface);
try out_writer.flush();

That skips the hash check. Archive.unpack does the same thing for every entry with the check, so prefer it unless you need one file.