A complete Python application to automatize the process of uploading files to Amazon S3

Upload files or folders (even with subfolders) to Amazon S3 in a totally automatized way taking advantage of:

• Amazon S3 Multipart Upload: The uploaded files are processed transparently in parts improving the throughput and the quick recovery from any network issues.

• Resilent Retry System: Intelligent S3 Upload has been built to detect any error during the uploading process and to perform any retries whenever is necessary.

• User Friendly Interface: Just check the demo to see with your own eyes how the upload process is performed.

Checkout the demo

Installation

Clone the repository

git clone https://github.com/polius/intelligent-s3-upload.git

Install the dependencies

python3 -m pip install boto3 --user
python3 -m pip install requests --user

Setup

Before executing the Intelligent S3 Upload, modify the credentials.json file.

{
    "aws_access_key_id": "",
    "aws_secret_access_key": "",
    "region_name": "",
    "bucket_name": "",
    "bucket_prefix": "",
    "storage_class": "",
    "skip_s3_existing_files": true,
    "server_side_encryption": true,
    "slack_url": ""
}

• aws_access_key_id | aws_secret_access_key: Credentials generated by Amazon IAM.
• region_name: The AWS Region Code where the bucket is located.
• bucket_name: The bucket name created by Amazon S3.
• bucket_path: (Optional) The bucket path to store the uploaded objects.
• storage_class: The type of storage to use for the uploaded object. These are the possible values:

• skip_s3_existing_files: Skip uploading objects if these already exists in S3. Possible values: [ true | false ]
• server_side_encryption: Enable Server-side encryption using the Amazon S3 key (SSE-S3). Possible values: [ true | false ]
• slack_url: (Optional) Enter a Webhook URL to send a message to Slack when a upload finishes.

AWS Policy

To be able to run the script check that your policy meets the following requirements:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:ListBucketMultipartUploads",
                "s3:AbortMultipartUpload",
                "s3:ListBucket",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::yourbucket/*",
                "arn:aws:s3:::yourbucket"
            ]
        }
    ]
}

Execution

python3 upload.py --path "{PATH}"

Replace the {PATH} string with the absolute file/folder path.

License

This project is licensed under the MIT license. See the LICENSE file for more info.