Output examples

This page contains examples to show how the output of stream-zip can be manipulated for different cases. It is likely you will have to modifiy them for your specific needs.

The examples assume an iterable of member files is available in the variable member_files. See Get started for a guide on how to construct this, or Input examples for examples.

Local file

Saving the ZIP to a local file can be done wtih Python’s built-in open function.

from stream_zip import stream_zip

zipped_chunks = stream_zip(member_files)
with open('my.zip', 'wb') as f:
    for chunk in zipped_chunks:
        f.write(chunk)

File-like object

If you need to output a file-like object rather than an iterable yielding bytes, you can pass the return value of stream_zip through the to_file_like_obj function from to-file-like-obj.

from stream_zip import stream_zip
from to_file_like_obj import to_file_like_obj

zipped_chunks = stream_zip(member_files)
zipped_chunks_obj = to_file_like_obj(zipped_chunks)

Upload to S3

The to_file_like_obj function from to-file-like-obj can be used to upload large ZIP files to S3 with boto3’s upload_fileobj.

import boto3
from boto3.s3.transfer import TransferConfig
from stream_zip import stream_zip
from to_file_like_obj import to_file_like_obj

zipped_chunks = stream_zip(member_files)
zipped_chunks_obj = to_file_like_obj(zipped_chunks)

s3 = boto3.client('s3')
s3.upload_fileobj(
    zipped_chunks_obj, 'mybucket', 'mykey',
    # Since we're streaming the final total size is unknown, so we have to tell boto3 what part
    # size to use to accomodate the entire file - S3 has a hard coded limit of 10000 parts
    # In this example we choose a part size of 200MB, so 2TB maximum final object size
    Config=TransferConfig(multipart_chunksize=1024 * 1024 * 200),
)

Submit your own

Pull requests (PRs) that propose changes to this page are especially welcome. PRs can be made at the source of this page. Submitting a PR requires a GitHub account and knowledge of the GitHub fork and PR process.