A faster collectstatic command.

Anton Agestam

Last update: Dec 27, 2022

Related tags

Overview

Collectfast

A faster collectstatic command.

Features

Efficiently decide what files to upload using cached checksums
Parallel file uploads

Supported Storage Backends

storages.backends.s3boto3.S3Boto3Storage
storages.backends.gcloud.GoogleCloudStorage
django.core.files.storage.FileSystemStorage

Running Django's collectstatic command can become painfully slow as more and more files are added to a project, especially when heavy libraries such as jQuery UI are included in source code. Collectfast customizes the builtin collectstatic command, adding different optimizations to make uploading large amounts of files much faster.

Installation

Install the app using pip:

$ python3 -m pip install Collectfast

Make sure you have this in your settings file and add 'collectfast' to your INSTALLED_APPS, before 'django.contrib.staticfiles':

STATICFILES_STORAGE = "storages.backends.s3boto3.S3Boto3Storage"
COLLECTFAST_STRATEGY = "collectfast.strategies.boto3.Boto3Strategy"
INSTALLED_APPS = (
    # ...
    'collectfast',
)

Note: 'collectfast' must come before 'django.contrib.staticfiles' in INSTALLED_APPS.

Note: The boto strategy will set preload_metadata on the remote storage to True, see #30.

Upload Strategies

Collectfast Strategy	Storage Backend
collectfast.strategies.boto3.Boto3Strategy	storages.backends.s3boto3.S3Boto3Storage
collectfast.strategies.gcloud.GoogleCloudStrategy	storages.backends.gcloud.GoogleCloudStorage
collectfast.strategies.filesystem.FileSystemStrategy	django.core.files.storage.FileSystemStorage

Custom strategies can also be made for backends not listed above by implementing the collectfast.strategies.Strategy ABC.

Usage

Collectfast overrides Django's builtin collectstatic command so just run python manage.py collectstatic as normal.

You can disable Collectfast by using the --disable-collectfast option or by setting COLLECTFAST_ENABLED = False in your settings file.

Setting Up a Dedicated Cache Backend

It's recommended to setup a dedicated cache backend for Collectfast. Every time Collectfast does not find a lookup for a file in the cache it will trigger a lookup to the storage backend, so it's recommended to have a fairly high TIMEOUT setting.

Configure your dedicated cache with the COLLECTFAST_CACHE setting:

CACHES = {
    'default': {
        # Your default cache
    },
    'collectfast': {
        # Your dedicated Collectfast cache
    },
}

COLLECTFAST_CACHE = 'collectfast'

If COLLECTFAST_CACHE isn't set, the default cache will be used.

Note: Collectfast will never clean the cache of obsolete files. To clean out the entire cache, use cache.clear(). See docs for Django's cache framework.

Note: We recommend you to set the MAX_ENTRIES setting if you have more than 300 static files, see #47.

Enable Parallel Uploads

The parallelization feature enables parallel file uploads using Python's concurrent.futures module. Enable it by setting the COLLECTFAST_THREADS setting.

To enable parallel uploads, a dedicated cache backend must be setup and it must use a backend that is thread-safe, i.e. something other than Django's default LocMemCache.

COLLECTFAST_THREADS = 20

Debugging

By default, Collectfast will suppress any exceptions that happens when copying and let Django's collectstatic handle it. To debug those suppressed errors you can set COLLECTFAST_DEBUG = True in your Django settings file.

Contribution

Please feel free to contribute by using issues and pull requests. Discussion is open and welcome.

Testing

The test suite is built to run against live S3 and GCloud buckets. You can disable live tests by setting SKIP_LIVE_TESTS=true or running make test-skip-live. To run live tests locally you need to provide API credentials to test against. Add the credentials to a file named storage-credentials in the root of the project directory:

export AWS_ACCESS_KEY_ID='...'
export AWS_SECRET_ACCESS_KEY='...'
export GCLOUD_CREDENTIALS='{...}'  # Google Cloud credentials as JSON

Install test dependencies and target Django version:

python3 -m pip install -r test-requirements.txt
python3 -m pip install django==2.2

Run test suite:

make test

Code quality tools are broken out from test requirements because some of them only install on Python >= 3.7.

python3 -m pip install -r lint-requirements.txt

Run linters and static type check:

make lint

License

Collectfast is licensed under the MIT License, see LICENSE file for more information.

Comments

AttributeError: 'Bucket' object has no attribute 'Object'

When i try to do an upload of a file to s3 and get its etag, i get this error

  File "/venv3/lib/python3.5/site-packages/collectfast/management/commands/collectstatic.py", line 84, in get_boto3_etag
    return self.storage.bucket.Object(path).e_tag
AttributeError: 'Bucket' object has no attribute 'Object'

Django = 1.10.1 python = 3.5

opened by andy-io 14

Support for Google Cloud Storage
Currently this project does not work with Google Cloud Storage. This is because GoogleCloudStorage:

has a different method of getting a remote file's md5 hash, and it

does not have the location property

These changes resolve this two issues.
opened by jpnauta 9

Collectfast breaks with local filesystem storage even when disabled

Hi,

Thanks for publishing this project. It is very helpful :-)

I've encountered the following issue:

When using Collectfast in disabled mode (for local development without S3), running ./manage.py collectstatic breaks even with --disable-collectfast or COLLECTFAST_ENABLED = False. The reason seems to be the following:

https://github.com/antonagestam/collectfast/blob/ced213e20c8905700c4b7a7e949d8cc7b37e20da/collectfast/management/commands/collectstatic.py#L35

When using filesystem storage, collectstatic crashes with the following:

Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/johananl/tmp/collectfast_test/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
    utility.execute()
  File "/Users/johananl/tmp/collectfast_test/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 365, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/johananl/tmp/collectfast_test/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 216, in fetch_command
    klass = load_command_class(app_name, subcommand)
  File "/Users/johananl/tmp/collectfast_test/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 37, in load_command_class
    return module.Command()
  File "/Users/johananl/tmp/collectfast_test/venv/lib/python3.6/site-packages/collectfast/management/commands/collectstatic.py", line 36, in __init__
    if self.storage.preload_metadata is not True:
  File "/Users/johananl/tmp/collectfast_test/venv/lib/python3.6/site-packages/django/utils/functional.py", line 216, in inner
    return func(self._wrapped, *args)
AttributeError: 'StaticFilesStorage' object has no attribute 'preload_metadata'

This happens because StaticFilesStorage doesn't have an attribute called preload_metadata (I believe it is S3-specific).

To reproduce this issue, simply run ./manage.py collectstatic with collectfast in INSTALLED_APPS and a default storage configuration.

I believe this can be solved by something similar to the following:

from storages.backends.s3boto3 import S3Boto3Storage
...
if isinstance(self.storage, S3Boto3Storage) and self.storage.preload_metadata is not True:
    ...

The following is also possible to avoid the import:

if (type(self.storage).__name__ == 'storages.backends.s3boto3.S3Boto3Storage'
    and self.storage.preload_metadata is not True):

What do you think?

Thanks, Johanan

bug

opened by johananl 9

Postprocess fails because upload happens after collect command.
Running into error during post-processing

django/contrib/staticfiles/storage.py", line 101, in hashed_name: raise ValueError("The file '%s' could not be found with %r." % (filename, self))

I believe this happens because the underlying collect() command also runs post-process. But because this library overrides collects such that uploads don't happen until the underlying collection function completes. The image has not actually been uploaded yet.

Happy to submit PR to delay the post-process function
bug
opened by justinwang189 8
Doesn't work for gzipped items

If AWS_IS_GZIPPED is set to True, then this will check the hash of the un-gzipped item against the etag of the gzipped item, which will obviously not match (though once upon a time it used to, but amazon fixed that).

This means all js and css files on a set-up using gzipping will be copied on every collect.
feature request

opened by manterfield 7
Set preload_metadata on the fly

Setting AWS_PRELOAD_METADATA = True is not always an option, because it makes some commands in the S3BotoStorage really slow. But when it's not enabled, the command does not work. So, preload_metadata should be set when it's not set or set to false when running the command.

Relevant: https://bitbucket.org/david/django-storages/issue/106/preload-breaks-things-like-exist https://github.com/AGoodId/django-s3-collectstatic/issues/2

opened by Gwildor 7
Major cleanup
Drop support for Python 3.5

Drop support for Django 1.11

Drop support for storages.backends.s3boto.S3BotoStorage (collectfast.strategies.boto.BotoStrategy)

Drop support for guessing strategies, e.g. require COLLECTFAST_STRATEGY to be set

Replace comment syntax with type annotation syntax

Migrate to setup.cfg

Start packaging typing information

Todo

~~Stop special-casing Python 3.8 in travis conf~~ #169

~~Consolidate test-requirements and lint-requirements~~ #170
opened by antonagestam 6
sdist tests not runnable
The source distribution isnt testable as-is.

Ideally one of the following should work.

python setup.py test python `which django-admin` test

I see this project uses tox and runtests.py.

The files to support that are not included in the sdist.
feature request
opened by jayvdb 6

Error on collectstatic using django 1.11 python 3.6

Hi ,

I'm trying to move my Django project (which uses collectstatic) from python 2.7 to python 3.6.

When running collectstatic from the command line I get an error (see further). As everything else works fine, I'm a bit lost on what went wrong here.

I did not change my settings.py and all worked fine under python 2.7

Thanks for pointing me in the right direction!

Regards, Joris

The error:

(platform)baze@baze-mintbox:~/dev/www/projects/ieper/platform > python manage.py collectstatic

Traceback (most recent call last):
  File "manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
    utility.execute()
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/core/management/__init__.py", line 355, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/management/commands/collectstatic.py", line 70, in handle
    super(Command, self).handle(**options)
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 199, in handle
    collected = self.collect()
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/management/commands/collectstatic.py", line 61, in collect
    ret = super(Command, self).collect()
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 124, in collect
    handler(path, prefixed_path, storage)
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/management/commands/collectstatic.py", line 109, in copy_file
    self.do_copy_file(args)
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/management/commands/collectstatic.py", line 86, in do_copy_file
    self.storage, path, prefixed_path, source_storage):
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/etag.py", line 93, in should_copy_file
    remote_storage, source_storage, path, prefixed_path):
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/etag.py", line 83, in has_matching_etag
    storage_etag = get_etag(remote_storage, path, prefixed_path)
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/etag.py", line 58, in get_etag
    etag = get_remote_etag(storage, prefixed_path)
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/etag.py", line 38, in get_remote_etag
    normalized_path = storage._normalize_name(
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/utils/functional.py", line 239, in inner
    return func(self._wrapped, *args)
AttributeError: 'StaticFilesStorage' object has no attribute '_normalize_name'\

For reference, my pip freeze output:

boto==2.46.1 boto3==1.4.4 botocore==1.5.40 certifi==2017.4.17 chardet==3.0.4 Collectfast==0.5.2 defusedxml==0.5.0 dj-database-url==0.4.2 Django==1.11 django-allauth==0.32.0 django-postgrespool==0.3.1 django-storages==1.5.2 docutils==0.13.1 futures==3.1.1 gunicorn==19.7.1 idna==2.5 jmespath==0.9.2 newrelic==2.86.3.70 oauthlib==2.0.2 psycopg2==2.7.1 python-dateutil==2.6.0 python3-openid==3.1.0 pytz==2017.2 requests==2.17.3 requests-oauthlib==0.8.0 s3transfer==0.1.10 six==1.10.0 SQLAlchemy==1.1.10 urllib3==1.21.1

opened by ImplicitCoder 6

0.6.2 CI now fails

Re-running CI on 0.6.2 results in errors

https://travis-ci.org/jayvdb/collectfast/builds/495173846

======================================================================
ERROR: collectfast.tests.test_command (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: collectfast.tests.test_command
Traceback (most recent call last):
  File "/opt/python/3.6.3/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/opt/python/3.6.3/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/home/travis/build/jayvdb/collectfast/collectfast/tests/test_command.py", line 7, in <module>
    from .utils import test, clean_static_dir, create_static_file, override_setting
  File "/home/travis/build/jayvdb/collectfast/collectfast/tests/utils.py", line 10, in <module>
    import boto
  File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/site-packages/boto/__init__.py", line 1216, in <module>
    boto.plugin.load_plugins(config)
  File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/site-packages/boto/plugin.py", line 93, in load_plugins
    _import_module(file)
  File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/site-packages/boto/plugin.py", line 75, in _import_module
    return imp.load_module(name, file, filename, data)
  File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/imp.py", line 235, in load_module
    return load_source(name, filename, file)
  File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/imp.py", line 172, in load_source
    module = _load(spec)
  File "/usr/lib/python2.7/dist-packages/google_compute_engine/boto/compute_auth.py", line 19, in <module>
    from google_compute_engine import logger
ModuleNotFoundError: No module named 'google_compute_engine'
======================================================================
ERROR: collectfast.tests.test_etag (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: collectfast.tests.test_etag
Traceback (most recent call last):
  File "/opt/python/3.6.3/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/opt/python/3.6.3/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/home/travis/build/jayvdb/collectfast/collectfast/tests/test_etag.py", line 7, in <module>
    from storages.backends.s3boto import S3BotoStorage
  File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/site-packages/storages/backends/s3boto.py", line 25, in <module>
    from boto import __version__ as boto_version
  File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/site-packages/boto/__init__.py", line 1216, in <module>
    boto.plugin.load_plugins(config)
AttributeError: module 'boto' has no attribute 'plugin'

When I try to run it on openSUSE build environment, with latest moto, I get different errors:

======================================================================
ERROR: test_basics (collectfast.tests.utils.test_basics)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/moto/core/models.py", line 74, in wrapper
    result = func(*args, **kwargs)
  File "collectfast/tests/utils.py", line 33, in wraps
    boto.connect_s3().create_bucket(django_settings.AWS_STORAGE_BUCKET_NAME)
  File "/usr/lib/python2.7/site-packages/boto/__init__.py", line 141, in connect_s3
    return S3Connection(aws_access_key_id, aws_secret_access_key, **kwargs)
  File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 194, in __init__
    validate_certs=validate_certs, profile_name=profile_name)
  File "/usr/lib/python2.7/site-packages/boto/connection.py", line 569, in __init__
    host, config, self.provider, self._required_auth_capability())
  File "/usr/lib/python2.7/site-packages/boto/auth.py", line 1021, in get_auth_handler
    'Check your credentials' % (len(names), str(names)))
NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV1Handler'] Check your credentials

bug

opened by jayvdb 5

Properly handle non-S3 storage backends

This PR addresses https://github.com/antonagestam/collectfast/issues/120. This bug causes Collectfast to crash when a non-S3 storage backend is configured.

I am not completely sure about the way I've implemented the storage type check, so please feel free to suggest alternative approaches. Regardless and in general - any comments and suggestions are very welcome.

opened by johananl 5

Fix boto3 pre_should_copy_hook closing the connection too soon

I've been testing collecfast for a while and I've spotted an issue with the connection being closed too soon.

After digging around and trying to understand conversations about thread-safe, I've checked about S3Boto3Storage and I can confirm it's thread-safe as well.

You can check the log here https://pypi.org/project/django-storages/ : 1.6.4 (2017-07-27) The conversation I found about it https://github.com/antonagestam/collectfast/issues/107

As S3Boto3Storage is safe to use with thread, we can simply drop the logic about resetting the connection. If you keep this logic, it closes the main connection of s3 client and some files are not getting synced at all. You have to run it multiple time to make it works.

I've tested locally by commenting off the code there, and it was working as expected https://github.com/antonagestam/collectfast/blob/master/collectfast/management/commands/collectstatic.py#L106

Here are my settings:

COLLECTFAST_ENABLED = os.getenv("COLLECTFAST_ENABLED") == "true"

# Default strategy for collectfast
COLLECTFAST_STRATEGY = os.getenv("COLLECTFAST_STRATEGY", "collectfast.strategies.filesystem.FileSystemStrategy")

# Define how much collectfast should store entries in the cache. We assume there are 2400 static file in the project
MAX_ENTRIES = int(os.getenv("MAX_ENTRIES", 300))

# Set which cache to use.
COLLECTFAST_CACHE = os.getenv("COLLECTFAST_CACHE", "default")

# Enable Parallel Uploads. Do not try to exceed this value with AWS. It will close the connection
COLLECTFAST_THREADS = int(os.getenv("COLLECTFAST_THREADS", 10))

COLLECTFAST_DEBUG = os.getenv("COLLECTFAST_DEBUG") == "true"

COLLECTFAST_REDIS_CACHE = {
    "BACKEND": "django_redis.cache.RedisCache",
    "LOCATION": os.environ.get("COLLECTFAST_REDIS_URL", "redis://redis/0"),
    "KEY_PREFIX": "collectfast",
    "OPTIONS": {
        "SOCKET_CONNECT_TIMEOUT": int(os.getenv("REDIS_SOCKET_CONNECT_TIMEOUT", default=2)),
        "SOCKET_TIMEOUT": int(os.getenv("REDIS_SOCKET_TIMEOUT", default=2)),
        "MAX_ENTRIES": MAX_ENTRIES,
        "IGNORE_EXCEPTIONS": os.getenv("COLLECTFAST_IGNORE_EXCEPTIONS") == "true",
    },
}

And my .env

STATICFILES_STORAGE="storages.backends.s3boto3.S3Boto3Storage"
COLLECTFAST_STRATEGY="collectfast.strategies.boto3.Boto3Strategy"
MAX_ENTRIES=3000
COLLECTFAST_ENABLED=true
COLLECTFAST_THREAD=20
COLLECTFAST_DEBUG=true
AWS_DEFAULT_ACL=private

COLLECTFAST_CACHE=collectfast_redis_cache
COLLECTFAST_REDIS_URL="redis://0.0.0.0:6379/0"
COLLECTFAST_IGNORE_EXCEPTIONS="true"

WDYT about that?

opened by vschoener 5

Set the test suite regardless of actual cloud connection.
Now we can run the test suite independently without the actual access key of AWS and GCP.

mock boto3.s3 before starting tests with moto

mock GoogleCloud Client, Bucket, Blob classes.
opened by mode9 2
Make test suite independent of original maintainer

Currently the test suite has been running against real accounts on AWS and Google Cloud. For multiple reasons it would be great to replace this with mocks.

I've attempted to do this for S3 a few years back with moto, and failed, hopefully moto has improved since then and it should be attempted again.

One also needs to look into something equivalent for Google Cloud.

opened by antonagestam 2
⚠️ Looking for maintainers

Hi there 👋

I am looking for a maintainer to take over this project. I am no longer working with projects that uses the collectstatic framework and therefor can't be as invested as a maintainer of this project should preferably be.

Please get in contact if you are willing to take over.

opened by antonagestam 9

Don't suppress errors when using threads

When I was using collectfast in a CI system, some of the assets were not copied. Sometimes it copied all, sometimes just a subset without any errors.

This is what I experienced:

~ $ python manage.py collectstatic --noinput --disable-collectfast
1249 static files copied, 59 unmodified.
~ $
~ $ echo 'Deleting the files from AWS now'
Deleting the files from AWS now
~ $
~ $ python manage.py collectstatic --noinput
790 static files copied.
~ $ python manage.py collectstatic --noinput
518 static files copied.
~ $ python manage.py collectstatic --noinput
0 static files copied.
~ $ python manage.py collectstatic --noinput
0 static files copied.
~ $
~ $ echo 'Deleting the files from AWS now'
Deleting the files from AWS now
~ $
~ $ python manage.py collectstatic --noinput
697 static files copied.
~ $ python manage.py collectstatic --noinput
611 static files copied.
~ $ python manage.py collectstatic --noinput
0 static files copied.
~ $ python manage.py collectstatic --noinput
0 static files copied.
~ $
~ $

It turned out, that the redis cache I used has some connection limits, so in the background I was having redis.exceptions.ConnectionError: max number of clients reached errors without being notified.

The collectstatic build step succeeded all times while having these errors and not copying some files.

The proposed fix simply waits for all the results and re-raises any errors, so the collectstatic command will fail loudly when an error happens in at least one thread.

(I wanted to write a test for this, but wasn't able to get the test suite running so I gave it up.)

opened by lepilepi 3

Releases(2.2.0)

2.2.0(Jun 5, 2020)
Add post_copy_hook and on_skip_hook to collectfast.strategies.base.Strategy.

Add collectfast.strategies.filesystem.CachingFileSystemStrategy.

Fix a bug where files weren't properly closed when read for hashing.

Fix a bug where gzip compression level was inconsistent with S3.

Source code(tar.gz)
Source code(zip)
2.1.0(Mar 2, 2020)
Use concurrent.futures.ThreadPoolExecutor instead of multiprocessing.dummy.Pool for parallel uploads. (#187)

Support post_process() (#188)

Source code(tar.gz)
Source code(zip)
2.0.1(Feb 24, 2020)
Fixes and adds regression test for #178

Adds tests for strictly typed settings (#182)

Source code(tar.gz)
Source code(zip)
2.0.0(Feb 23, 2020)
Drop support for Python 3.5.

Drop support for Django 1.11.

Drop support for storages.backends.s3boto.S3BotoStorage (remove collectfast.strategies.boto.BotoStrategy).

Drop support for guessing strategies, e.g. require COLLECTFAST_STRATEGY to be set.

Package type hints.

Support django-storages 1.9+.

Validate types of settings.

Source code(tar.gz)
Source code(zip)
1.3.2(Feb 16, 2020)
Support Django 3.0

Limit django-storages to <1.9

Source code(tar.gz)
Source code(zip)
1.3.0(Oct 22, 2019)
Fix #157: catch pydoc.ErrorDuringImport

Remove runtests.py and use pytest instead

Add Python 3.8 to test matrix

Update copyright note in readme

Add deprecation warning for using BotoStrategy

Source code(tar.gz)
Source code(zip)
1.2.0(Oct 6, 2019)

Implement strategy for Google Cloud Storage
Source code(tar.gz)
Source code(zip)
1.1.1(Oct 6, 2019)

1.1.0 broke compatibility by not correctly guessing strategy for subclasses of supported storages.
Source code(tar.gz)
Source code(zip)
1.1.0(Oct 6, 2019)
Large rewrite to simplify implementing support for more storage backends. Introduces a Strategy abstraction that can be subclassed to implement support for different storage backends and needs.

Misc.:

Remove --ignore-etag option

Get rid of wrapping quotes in hashes, bump cache key version

Command tests now run with both boto and boto3

Source code(tar.gz)
Source code(zip)
1.0.0(Aug 12, 2019)
Drops support for Python 2.

Source code(tar.gz)
Source code(zip)
0.6.0(Dec 8, 2017)
support for AWS_IS_GZIPPED

use of public API on storage

support for django 1.11 and 2.0

Source code(tar.gz)
Source code(zip)
0.5.2(Feb 8, 2017)
Fixes boto compatibility

Source code(tar.gz)
Source code(zip)
0.5.1(Jan 10, 2017)

fix regression in 0.5.0
Source code(tar.gz)
Source code(zip)
0.5.0(Jan 4, 2017)
Large refactoring and addition of integration tests

A few minor bugfixes

Source code(tar.gz)
Source code(zip)
0.4.1(Dec 30, 2016)

Fixes regression in 0.4.0.
Source code(tar.gz)
Source code(zip)
0.4.0(Dec 17, 2016)

0.4.0 introduces parallel file uploads using Python's multiprocessing module. This enables huge speed-ups for first time uploads. Using testdata with 50 files at 500kb each upload finished in 1m9s using 0.3.1 and in just 10s using 0.4.0.
Source code(tar.gz)
Source code(zip)
0.3.1(Nov 2, 2016)

Source code(tar.gz)
Source code(zip)
0.3.0(Nov 2, 2016)
Added support for boto3

Changed caching mechanism to only store etag, should be lighter on the cache backend

Updated tests to include Django 1.10 and updated docs

Source code(tar.gz)
Source code(zip)
0.2.3(Apr 22, 2015)

Source code(tar.gz)
Source code(zip)
0.2.2(Apr 22, 2015)

Collectfast now runs on Django 1.8 thanks to @aericson.

The test suite is now also run on Python 3.4 for Django 1.7 and 1.8.
Source code(tar.gz)
Source code(zip)
0.2.0(Apr 10, 2014)
Fix normalized_path bug

Better cache documentation

Introduced COLLECTFAST_ENABLED setting

Override AWS_PRELOAD_METADATA if not set to True

Adds testing! :)

Source code(tar.gz)
Source code(zip)
0.1.14(Mar 13, 2014)
Python 3 support (#22)

Configurable cache backend using COLLECTFAST_CACHE (#26)

Support Storage.location by normalizing path names (#25)

Source code(tar.gz)
Source code(zip)
0.1.13(Oct 24, 2013)

Fixes installation issue introduced in the 0.1.12 package.
Source code(tar.gz)
Source code(zip)
0.1.12(Oct 21, 2013)

Should further increase performance.
Source code(tar.gz)
Source code(zip)
0.1.11(Jul 22, 2013)
This release reduces the default verbosity an only outputs a summary with the total number of skipped files and a benchmark of how long time it took to run the collect method. The biggest change is that the Command class now overrides the handle method. This might introduce problems for people using other Django versions than 1.4. Please submit issues if this is the case.

To get all the "skipped file x" logs back you should use:

$ python manage.py collectstatic --verbosity=2
Source code(tar.gz)
Source code(zip)