When decompressing a zip file this error is raised:

stream_unzip.UnexpectedSignatureError: b'\xb66\x97\x89'

Part of my code:

def read_in_chunks(infile, chunk_size=UNZIP_CHUNK_SIZE):
    while True:
        chunk = infile.read(chunk_size)
        if chunk:
            yield chunk
        else:
            return<br>
with open(path_to_zip_file, 'rb') as file_object:
    for file_name, file_size, file_chunks in stream_unzip(read_in_chunks(file_object), chunk_size=UNZIP_CHUNK_SIZE):
        print(file_name)
        with open(file_name, 'ab') as file_append:
            for chunk in file_chunks:
                file_append.write(chunk)

Python 3.9 stream-unzip 0.0.70

I can unzip my file with WinRAR and zipfile, but can't with stream-unzip.

Any idea how fix it?

Hi there. From this PR, it seems that encrypted zip files are supported by now, but I found that I cannot unzip a password protected zip file using Deflate64 as the compression method. I am not sure whether it is not implemented yet or it is an unexpected bug. Thanks.

Thanks for the lib. Got ZeroDivisionError: integer division or modulo by zero during processing zip file's last chunk from example code snippet:

Traceback (most recent call last):
  File "***\tmp.py", line 9, in <module>
    for file_name, file_size, unzipped_chunks in stream_unzip(zipped_chunks()):
  File "***\venv_win32_39py\lib\site-packages\stream_unzip.py", line 180, in stream_unzip
    for _ in yield_all():
  File "***\venv_win32_39py\lib\site-packages\stream_unzip.py", line 35, in _yield_all
    offset = (offset + to_yield) % len(chunk)
ZeroDivisionError: integer division or modulo by zero

Code snippet:

import httpx
from stream_unzip import stream_unzip


def zipped_chunks():
    # Any iterable that yields a zip file
    with httpx.stream('GET', 'https://www.gyan.dev/ffmpeg/builds/packages/ffmpeg-4.4-essentials_build.zip') as r:
        yield from r.iter_bytes()


for file_name, file_size, unzipped_chunks in stream_unzip(zipped_chunks()):
    for chunk in unzipped_chunks:
        # print(chunk)
        print(file_name, file_size)

Python 3.9.5 (Windows 10) stream-unzip 0.0.23

When decompressing a Zip64 file this error is raised:

stream_unzip.UnexpectedSignatureError: b'8\\\x12]'

Have you encountered this before?

Thanks,

Rusty

I stumbled on a bug while trying to see if smart_open would work: if you do not iterate all chunks ("unzipped_chunks") there will be a signature error:

from stream_unzip import stream_unzip, UnexpectedSignatureError
import smart_open

url="https://transfer.sh/get/cp4LTN/test.zip"

print("without chunk iteration")
try:
    with smart_open.open(url,'rb') as f_h:
        for file_name, file_size, unzipped_chunks in stream_unzip(f_h):
            print(file_name)
except UnexpectedSignatureError as e:
    print("UnexpectedSignatureError: "+str(e))

print("with chunk iteration")
try:
    with smart_open.open(url,'rb') as f_h:
        for file_name, file_size, unzipped_chunks in stream_unzip(f_h):
            print(file_name)
            for chunk in unzipped_chunks:
                continue
except UnexpectedSignatureError as e:
    print("UnexpectedSignatureError: "+str(e))

result:

$ python .\stream_unzip_test.py
without chunk iteration
b'test.txt'
UnexpectedSignatureError: b'test'
with chunk iteration
b'test.txt'
b'test1.txt'
b'xsubtest.zip'

Hi,

I am trying to create a low-memory stream unzipper that can be ran in AWS Lambda. Currently I can unzip individual files from a .zip package. But the problem I am facing is that I need to read that one file into memory, which for a large file, 15gb, it is being killed by the runtime.

I came across this library, but I am unsure it delivers what I need. Or maybe it does but I don't see how.

This is my example below, using the normal zipfile from the standard lib.

import zipfile 
import s3fs
_zipfile_interface = zipfile.ZipFile
_s3_interface = s3fs.S3FileSystem(anon=False)
_READING_MODE = 'rb'
_WRITING_MODE = 'wb'


def extract_file(filename: str, output_uri: str):
    """
    Extracts filename from zip package into input output S3 URI

    Args:
        filename (str): Name of the file inside zip package
        output_uri (str): Output S3 URI

    Raises:
        exceptions.ExtractionFailure: In case extraction errors out
    """
    with _s3_interface.open(s3_uri, _READING_MODE) as f_in:
        zip_buffer = _zipfile_interface(f_in)

        file_buffer = zip_buffer.open(filename)

        with _s3_interface.open(output_uri, _WRITING_MODE) as f_out:

            f_out.write(file_buffer.read())  # -> the issue lies here

Do ignore S3 interface, for purpose of this example, it's just a normal open().

I have profiled this method here:

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    78     66.2 MiB     66.2 MiB           1       @profile
    79                                             def extract_file(filename: str, output_uri: str):
    80                                                 """
    81                                                 Extracts filename from zip package into input output S3 URI
    82                                         
    83                                                 Args:
    84                                                     filename (str): Name of the file inside zip package
    85                                                     output_uri (str): Output S3 URI
    86                                         
    87                                                 Raises:
    88                                                     exceptions.ExtractionFailure: In case extraction errors out
    89                                                 """
    90     66.2 MiB      0.0 MiB           1           try:
    91     79.1 MiB      2.1 MiB           2               with _s3_interface.open(package.s3_uri, _READING_MODE) as f_in:
    92     73.1 MiB      0.0 MiB           1                   zip_buffer = _zipfile_interface(f_in)  # type: ignore
    93                                         
    94     83.9 MiB     10.8 MiB           1                   file_buffer = zip_buffer.open(filename)
    95                                         
    96     83.9 MiB      0.0 MiB           2                   with _s3_interface.open(output_uri, _WRITING_MODE) as f_out:
    97                                         
    98     83.9 MiB      0.0 MiB           1                       f_out.write(file_buffer.read())  # type: ignore
    99                                         

Line 94 you see the jump in memory, 10MB, from the zip.open(filename) this is when the file is consumed into memory and then I can write it out to the output buffer.

Ideally, I would like that zip.open to iterate chunks of size N that I can just write to the open buffer.

Any knowledge on this?

Thank you, Alex

Consider streaming in a zip file.

def zipped_chunks(zipfile_name: PurePath):
    # Iterable that yields the bytes of a zip file
    with open(zipfile_name, "r+b", buffering=io.DEFAULT_BUFFER_SIZE) as zip_f:
        yield zip_f.read()

I am attempting to unzip a number of large zip files concurrently that are hosted on a slow network drive. Do you see any value in leveraging aiofiles package to stream the read like so?:

async def zipped_chunks(zipfile_name: PurePath):
    # Iterable that yields the bytes of a zip file
    async with aiofiles.open(zipfile_name, "r+b", buffering=io.DEFAULT_BUFFER_SIZE) as zip_f:
        yield await zip_f.read()

async def unzip_tar_files(self, zipfile_name: PurePath):
    chunks: List[bytes] = [data async for data in self.zipped_chunks(zipfile_name)]
    for file_name, tar_file_size, unzipped_chunks in stream_unzip(chunks):
        ....

That seems to work well for me (so far). Do you see any downside?

If not, it might be a nice addition to the README as I have finally come across a program I am writing from scratch that benefits from leveraging an asyincio solution with stream-unzip being a key part of that solution. Took me forever to understand that only list comprehension supports async iteration.

If I try to use smart_open (so that we can e.g. stream from S3 buckets), for example:

with smart_open.open(url,'rb') as f_h:
    for file_name, file_size, unzipped_chunks in stream_unzip(f_h):
        print(file_name)

The code prints one file, and then throws an UnexpectedSignatureError.

We can't up front be sure of the format of the data descriptor in all cases. So we inch our way through the stream checking what we think is the data descriptor against the known compressed and uncompressed size of the data itself. If there is a match: that's "probably" the data descriptor.

The issue is that we could have cases where the we get a match just because the values happen to match. The cases are not likely, but so far I don't see a reason why they're impossible.

This should address issues reported in https://github.com/uktrade/stream-unzip/issues/33

This should fix the issue reported in https://github.com/uktrade/stream-unzip/issues/29

(The Deflate64 thing originally reported I suspect was a red herring. Looked like it’s more to do with the data descriptor)

Addresses some of the concerns at https://github.com/uktrade/stream-unzip/issues/21 (but does not change the API to allow not fully iterating over the bytes of member files)