For the limitation:

No compression is supported via the NO_COMPRESSION_* constants as in the above examples. However in these cases the entire contents of each uncompressed file is buffered in memory, and so should not be used for large files. This is because for uncompressed data, its size and CRC32 must be before it in the ZIP file.

To avoid buffering all contents, I'm wondering if we can add a data descriptor after uncompressed data as well so we can simply set 0 in the local file header, just like what we do for compressed files. From the original cpython zipfile implementation, it seems to only determine from whether output is seekable:

https://github.com/python/cpython/blob/96b344c2f15cb09251018f57f19643fe20637392/Lib/zipfile.py#L1610-L1611

So I guess this structure should also be valid?

Hi, first of all thanks for the great work! Is there a way to know the size of the compressed before starting? An example usecase could be the "Content-Length" header in servers, but I'd need it in some other scenarios too (even no-compression would be okay in this case). Thanks :)

Thanks so much for building this package. It is incredibly useful.

One feature that would make this more powerful, in my opinion, is to be able to configure the chunk size you receive from the generator. I noticed that it provides chunks of 65,536 bytes.

The reason is that this tool is particularly useful when uploading zip files by way of multipart uploads. I am personally working on a project that needs to upload upwards of terabyte-sized directories to S3, and I generate URLs for each gigabyte that will be uploaded. I had to concatenate the chunks together until they were a gigabyte before I could run the upload on each multipart URL. This wasn't challenging, but I think it'd be a nice interface improvement.

Maybe there's something I missed in the documentation that does allow for something like this. Please keep me posted.

Thanks again!

This was added in https://github.com/uktrade/stream-zip/commit/62f8cf1ee242f7d9fe1467fe9a93b20164764068, but it might not really have ever been needed, so removing it saves a few bytes. And stronger than that, https://github.com/libarchive/libarchive/issues/1834 gives an argument that it even makes the ZIP file invalid.

To give even more weight to the argument that this is ok, I made a few Zip64 files with data descriptors with InfoZIP. They did not have local Zip64 extra fields. If this were a problem, it probably would have been discovered by now.

This I think is a reasonable way of addressing the need for large files that should both not be buffered in memory, and not compressed, as reported in https://github.com/uktrade/stream-zip/issues/17

This is for clarity, and to allow in future having both NO_COMPRESSION_32 and NO_COMPRESSION_64, to allow clients to fully specify that they do not want any _64 code data.

Note that LibreOffice doesn't support zip64 at the time of writing, so this is still an issue. This is not just for "legacy" software.

Suspect it's possible that some clients don't use sizes of 0xffffffff to activate zip64 mode, but actually use the presence of the zip64 extra field. This is so they would be still able to open files with sizes exactly equal to 0xffffffff created by older zip software.

Realistically wouldn't hit the limits on ZIP64, but it's much more possible to hit the ZIP limits, and so offer a "nicer" Exception interface for this case.