A faster collectstatic command.

Overview

Collectfast

A faster collectstatic command.

Test Suite Static analysis Test Coverage Maintainability

Features

  • Efficiently decide what files to upload using cached checksums
  • Parallel file uploads

Supported Storage Backends

  • storages.backends.s3boto3.S3Boto3Storage
  • storages.backends.gcloud.GoogleCloudStorage
  • django.core.files.storage.FileSystemStorage

Running Django's collectstatic command can become painfully slow as more and more files are added to a project, especially when heavy libraries such as jQuery UI are included in source code. Collectfast customizes the builtin collectstatic command, adding different optimizations to make uploading large amounts of files much faster.

Installation

Install the app using pip:

$ python3 -m pip install Collectfast

Make sure you have this in your settings file and add 'collectfast' to your INSTALLED_APPS, before 'django.contrib.staticfiles':

STATICFILES_STORAGE = "storages.backends.s3boto3.S3Boto3Storage"
COLLECTFAST_STRATEGY = "collectfast.strategies.boto3.Boto3Strategy"
INSTALLED_APPS = (
    # ...
    'collectfast',
)

Note: 'collectfast' must come before 'django.contrib.staticfiles' in INSTALLED_APPS.

Note: The boto strategy will set preload_metadata on the remote storage to True, see #30.

Upload Strategies
Collectfast Strategy Storage Backend
collectfast.strategies.boto3.Boto3Strategy storages.backends.s3boto3.S3Boto3Storage
collectfast.strategies.gcloud.GoogleCloudStrategy storages.backends.gcloud.GoogleCloudStorage
collectfast.strategies.filesystem.FileSystemStrategy django.core.files.storage.FileSystemStorage

Custom strategies can also be made for backends not listed above by implementing the collectfast.strategies.Strategy ABC.

Usage

Collectfast overrides Django's builtin collectstatic command so just run python manage.py collectstatic as normal.

You can disable Collectfast by using the --disable-collectfast option or by setting COLLECTFAST_ENABLED = False in your settings file.

Setting Up a Dedicated Cache Backend

It's recommended to setup a dedicated cache backend for Collectfast. Every time Collectfast does not find a lookup for a file in the cache it will trigger a lookup to the storage backend, so it's recommended to have a fairly high TIMEOUT setting.

Configure your dedicated cache with the COLLECTFAST_CACHE setting:

CACHES = {
    'default': {
        # Your default cache
    },
    'collectfast': {
        # Your dedicated Collectfast cache
    },
}

COLLECTFAST_CACHE = 'collectfast'

If COLLECTFAST_CACHE isn't set, the default cache will be used.

Note: Collectfast will never clean the cache of obsolete files. To clean out the entire cache, use cache.clear(). See docs for Django's cache framework.

Note: We recommend you to set the MAX_ENTRIES setting if you have more than 300 static files, see #47.

Enable Parallel Uploads

The parallelization feature enables parallel file uploads using Python's concurrent.futures module. Enable it by setting the COLLECTFAST_THREADS setting.

To enable parallel uploads, a dedicated cache backend must be setup and it must use a backend that is thread-safe, i.e. something other than Django's default LocMemCache.

COLLECTFAST_THREADS = 20

Debugging

By default, Collectfast will suppress any exceptions that happens when copying and let Django's collectstatic handle it. To debug those suppressed errors you can set COLLECTFAST_DEBUG = True in your Django settings file.

Contribution

Please feel free to contribute by using issues and pull requests. Discussion is open and welcome.

Testing

The test suite is built to run against live S3 and GCloud buckets. You can disable live tests by setting SKIP_LIVE_TESTS=true or running make test-skip-live. To run live tests locally you need to provide API credentials to test against. Add the credentials to a file named storage-credentials in the root of the project directory:

export AWS_ACCESS_KEY_ID='...'
export AWS_SECRET_ACCESS_KEY='...'
export GCLOUD_CREDENTIALS='{...}'  # Google Cloud credentials as JSON

Install test dependencies and target Django version:

python3 -m pip install -r test-requirements.txt
python3 -m pip install django==2.2

Run test suite:

make test

Code quality tools are broken out from test requirements because some of them only install on Python >= 3.7.

python3 -m pip install -r lint-requirements.txt

Run linters and static type check:

make lint

License

Collectfast is licensed under the MIT License, see LICENSE file for more information.

Comments
  • AttributeError: 'Bucket' object has no attribute 'Object'

    AttributeError: 'Bucket' object has no attribute 'Object'

    When i try to do an upload of a file to s3 and get its etag, i get this error

      File "/venv3/lib/python3.5/site-packages/collectfast/management/commands/collectstatic.py", line 84, in get_boto3_etag
        return self.storage.bucket.Object(path).e_tag
    AttributeError: 'Bucket' object has no attribute 'Object'
    

    Django = 1.10.1 python = 3.5

    opened by andy-io 14
  • Support for Google Cloud Storage

    Support for Google Cloud Storage

    Currently this project does not work with Google Cloud Storage. This is because GoogleCloudStorage:

    1. has a different method of getting a remote file's md5 hash, and it
    2. does not have the location property

    These changes resolve this two issues.

    opened by jpnauta 9
  • Collectfast breaks with local filesystem storage even when disabled

    Collectfast breaks with local filesystem storage even when disabled

    Hi,

    Thanks for publishing this project. It is very helpful :-)

    I've encountered the following issue:

    When using Collectfast in disabled mode (for local development without S3), running ./manage.py collectstatic breaks even with --disable-collectfast or COLLECTFAST_ENABLED = False. The reason seems to be the following:

    https://github.com/antonagestam/collectfast/blob/ced213e20c8905700c4b7a7e949d8cc7b37e20da/collectfast/management/commands/collectstatic.py#L35

    When using filesystem storage, collectstatic crashes with the following:

    Traceback (most recent call last):
      File "./manage.py", line 10, in <module>
        execute_from_command_line(sys.argv)
      File "/Users/johananl/tmp/collectfast_test/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
        utility.execute()
      File "/Users/johananl/tmp/collectfast_test/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 365, in execute
        self.fetch_command(subcommand).run_from_argv(self.argv)
      File "/Users/johananl/tmp/collectfast_test/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 216, in fetch_command
        klass = load_command_class(app_name, subcommand)
      File "/Users/johananl/tmp/collectfast_test/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 37, in load_command_class
        return module.Command()
      File "/Users/johananl/tmp/collectfast_test/venv/lib/python3.6/site-packages/collectfast/management/commands/collectstatic.py", line 36, in __init__
        if self.storage.preload_metadata is not True:
      File "/Users/johananl/tmp/collectfast_test/venv/lib/python3.6/site-packages/django/utils/functional.py", line 216, in inner
        return func(self._wrapped, *args)
    AttributeError: 'StaticFilesStorage' object has no attribute 'preload_metadata'
    

    This happens because StaticFilesStorage doesn't have an attribute called preload_metadata (I believe it is S3-specific).

    To reproduce this issue, simply run ./manage.py collectstatic with collectfast in INSTALLED_APPS and a default storage configuration.

    I believe this can be solved by something similar to the following:

    from storages.backends.s3boto3 import S3Boto3Storage
    ...
    if isinstance(self.storage, S3Boto3Storage) and self.storage.preload_metadata is not True:
        ...
    

    The following is also possible to avoid the import:

    if (type(self.storage).__name__ == 'storages.backends.s3boto3.S3Boto3Storage'
        and self.storage.preload_metadata is not True):
    

    What do you think?

    Thanks, Johanan

    bug 
    opened by johananl 9
  • Postprocess fails because upload happens after collect command.

    Postprocess fails because upload happens after collect command.

    Running into error during post-processing

    django/contrib/staticfiles/storage.py", line 101, in hashed_name:    
    raise ValueError("The file '%s' could not be found with %r." % (filename, self))
    

    I believe this happens because the underlying collect() command also runs post-process. But because this library overrides collects such that uploads don't happen until the underlying collection function completes. The image has not actually been uploaded yet.

    Happy to submit PR to delay the post-process function

    bug 
    opened by justinwang189 8
  • Doesn't work for gzipped items

    Doesn't work for gzipped items

    If AWS_IS_GZIPPED is set to True, then this will check the hash of the un-gzipped item against the etag of the gzipped item, which will obviously not match (though once upon a time it used to, but amazon fixed that).

    This means all js and css files on a set-up using gzipping will be copied on every collect.

    feature request 
    opened by manterfield 7
  • Set preload_metadata on the fly

    Set preload_metadata on the fly

    Setting AWS_PRELOAD_METADATA = True is not always an option, because it makes some commands in the S3BotoStorage really slow. But when it's not enabled, the command does not work. So, preload_metadata should be set when it's not set or set to false when running the command.

    Relevant: https://bitbucket.org/david/django-storages/issue/106/preload-breaks-things-like-exist https://github.com/AGoodId/django-s3-collectstatic/issues/2

    opened by Gwildor 7
  • Major cleanup

    Major cleanup

    • Drop support for Python 3.5
    • Drop support for Django 1.11
    • Drop support for storages.backends.s3boto.S3BotoStorage (collectfast.strategies.boto.BotoStrategy)
    • Drop support for guessing strategies, e.g. require COLLECTFAST_STRATEGY to be set
    • Replace comment syntax with type annotation syntax
    • Migrate to setup.cfg
    • Start packaging typing information

    Todo

    • ~~Stop special-casing Python 3.8 in travis conf~~ #169
    • ~~Consolidate test-requirements and lint-requirements~~ #170
    opened by antonagestam 6
  • sdist tests not runnable

    sdist tests not runnable

    The source distribution isnt testable as-is.

    Ideally one of the following should work.

    python setup.py test
    python `which django-admin` test
    

    I see this project uses tox and runtests.py.

    The files to support that are not included in the sdist.

    feature request 
    opened by jayvdb 6
  • Error on collectstatic using django 1.11 python 3.6

    Error on collectstatic using django 1.11 python 3.6

    Hi ,

    I'm trying to move my Django project (which uses collectstatic) from python 2.7 to python 3.6.

    When running collectstatic from the command line I get an error (see further). As everything else works fine, I'm a bit lost on what went wrong here.

    I did not change my settings.py and all worked fine under python 2.7

    Thanks for pointing me in the right direction!

    Regards, Joris

    The error:

    (platform)baze@baze-mintbox:~/dev/www/projects/ieper/platform > python manage.py collectstatic
    
    Traceback (most recent call last):
      File "manage.py", line 22, in <module>
        execute_from_command_line(sys.argv)
      File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
        utility.execute()
      File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/core/management/__init__.py", line 355, in execute
        self.fetch_command(subcommand).run_from_argv(self.argv)
      File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
        self.execute(*args, **cmd_options)
      File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
        output = self.handle(*args, **options)
      File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/management/commands/collectstatic.py", line 70, in handle
        super(Command, self).handle(**options)
      File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 199, in handle
        collected = self.collect()
      File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/management/commands/collectstatic.py", line 61, in collect
        ret = super(Command, self).collect()
      File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 124, in collect
        handler(path, prefixed_path, storage)
      File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/management/commands/collectstatic.py", line 109, in copy_file
        self.do_copy_file(args)
      File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/management/commands/collectstatic.py", line 86, in do_copy_file
        self.storage, path, prefixed_path, source_storage):
      File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/etag.py", line 93, in should_copy_file
        remote_storage, source_storage, path, prefixed_path):
      File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/etag.py", line 83, in has_matching_etag
        storage_etag = get_etag(remote_storage, path, prefixed_path)
      File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/etag.py", line 58, in get_etag
        etag = get_remote_etag(storage, prefixed_path)
      File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/etag.py", line 38, in get_remote_etag
        normalized_path = storage._normalize_name(
      File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/utils/functional.py", line 239, in inner
        return func(self._wrapped, *args)
    AttributeError: 'StaticFilesStorage' object has no attribute '_normalize_name'\
    

    For reference, my pip freeze output:

    boto==2.46.1 boto3==1.4.4 botocore==1.5.40 certifi==2017.4.17 chardet==3.0.4 Collectfast==0.5.2 defusedxml==0.5.0 dj-database-url==0.4.2 Django==1.11 django-allauth==0.32.0 django-postgrespool==0.3.1 django-storages==1.5.2 docutils==0.13.1 futures==3.1.1 gunicorn==19.7.1 idna==2.5 jmespath==0.9.2 newrelic==2.86.3.70 oauthlib==2.0.2 psycopg2==2.7.1 python-dateutil==2.6.0 python3-openid==3.1.0 pytz==2017.2 requests==2.17.3 requests-oauthlib==0.8.0 s3transfer==0.1.10 six==1.10.0 SQLAlchemy==1.1.10 urllib3==1.21.1

    opened by ImplicitCoder 6
  • 0.6.2 CI now fails

    0.6.2 CI now fails

    Re-running CI on 0.6.2 results in errors

    https://travis-ci.org/jayvdb/collectfast/builds/495173846

    ======================================================================
    ERROR: collectfast.tests.test_command (unittest.loader._FailedTest)
    ----------------------------------------------------------------------
    ImportError: Failed to import test module: collectfast.tests.test_command
    Traceback (most recent call last):
      File "/opt/python/3.6.3/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
        module = self._get_module_from_name(name)
      File "/opt/python/3.6.3/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
        __import__(name)
      File "/home/travis/build/jayvdb/collectfast/collectfast/tests/test_command.py", line 7, in <module>
        from .utils import test, clean_static_dir, create_static_file, override_setting
      File "/home/travis/build/jayvdb/collectfast/collectfast/tests/utils.py", line 10, in <module>
        import boto
      File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/site-packages/boto/__init__.py", line 1216, in <module>
        boto.plugin.load_plugins(config)
      File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/site-packages/boto/plugin.py", line 93, in load_plugins
        _import_module(file)
      File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/site-packages/boto/plugin.py", line 75, in _import_module
        return imp.load_module(name, file, filename, data)
      File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/imp.py", line 235, in load_module
        return load_source(name, filename, file)
      File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/imp.py", line 172, in load_source
        module = _load(spec)
      File "/usr/lib/python2.7/dist-packages/google_compute_engine/boto/compute_auth.py", line 19, in <module>
        from google_compute_engine import logger
    ModuleNotFoundError: No module named 'google_compute_engine'
    ======================================================================
    ERROR: collectfast.tests.test_etag (unittest.loader._FailedTest)
    ----------------------------------------------------------------------
    ImportError: Failed to import test module: collectfast.tests.test_etag
    Traceback (most recent call last):
      File "/opt/python/3.6.3/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
        module = self._get_module_from_name(name)
      File "/opt/python/3.6.3/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
        __import__(name)
      File "/home/travis/build/jayvdb/collectfast/collectfast/tests/test_etag.py", line 7, in <module>
        from storages.backends.s3boto import S3BotoStorage
      File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/site-packages/storages/backends/s3boto.py", line 25, in <module>
        from boto import __version__ as boto_version
      File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/site-packages/boto/__init__.py", line 1216, in <module>
        boto.plugin.load_plugins(config)
    AttributeError: module 'boto' has no attribute 'plugin'
    

    When I try to run it on openSUSE build environment, with latest moto, I get different errors:

    ======================================================================
    ERROR: test_basics (collectfast.tests.utils.test_basics)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/moto/core/models.py", line 74, in wrapper
        result = func(*args, **kwargs)
      File "collectfast/tests/utils.py", line 33, in wraps
        boto.connect_s3().create_bucket(django_settings.AWS_STORAGE_BUCKET_NAME)
      File "/usr/lib/python2.7/site-packages/boto/__init__.py", line 141, in connect_s3
        return S3Connection(aws_access_key_id, aws_secret_access_key, **kwargs)
      File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 194, in __init__
        validate_certs=validate_certs, profile_name=profile_name)
      File "/usr/lib/python2.7/site-packages/boto/connection.py", line 569, in __init__
        host, config, self.provider, self._required_auth_capability())
      File "/usr/lib/python2.7/site-packages/boto/auth.py", line 1021, in get_auth_handler
        'Check your credentials' % (len(names), str(names)))
    NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV1Handler'] Check your credentials
    
    bug 
    opened by jayvdb 5
  • Properly handle non-S3 storage backends

    Properly handle non-S3 storage backends

    This PR addresses https://github.com/antonagestam/collectfast/issues/120. This bug causes Collectfast to crash when a non-S3 storage backend is configured.

    I am not completely sure about the way I've implemented the storage type check, so please feel free to suggest alternative approaches. Regardless and in general - any comments and suggestions are very welcome.

    opened by johananl 5
  • Fix boto3 pre_should_copy_hook closing the connection too soon

    Fix boto3 pre_should_copy_hook closing the connection too soon

    I've been testing collecfast for a while and I've spotted an issue with the connection being closed too soon.

    After digging around and trying to understand conversations about thread-safe, I've checked about S3Boto3Storage and I can confirm it's thread-safe as well.

    You can check the log here https://pypi.org/project/django-storages/ : 1.6.4 (2017-07-27) The conversation I found about it https://github.com/antonagestam/collectfast/issues/107

    As S3Boto3Storage is safe to use with thread, we can simply drop the logic about resetting the connection. If you keep this logic, it closes the main connection of s3 client and some files are not getting synced at all. You have to run it multiple time to make it works.

    I've tested locally by commenting off the code there, and it was working as expected https://github.com/antonagestam/collectfast/blob/master/collectfast/management/commands/collectstatic.py#L106

    Here are my settings:

    COLLECTFAST_ENABLED = os.getenv("COLLECTFAST_ENABLED") == "true"
    
    # Default strategy for collectfast
    COLLECTFAST_STRATEGY = os.getenv("COLLECTFAST_STRATEGY", "collectfast.strategies.filesystem.FileSystemStrategy")
    
    # Define how much collectfast should store entries in the cache. We assume there are 2400 static file in the project
    MAX_ENTRIES = int(os.getenv("MAX_ENTRIES", 300))
    
    # Set which cache to use.
    COLLECTFAST_CACHE = os.getenv("COLLECTFAST_CACHE", "default")
    
    # Enable Parallel Uploads. Do not try to exceed this value with AWS. It will close the connection
    COLLECTFAST_THREADS = int(os.getenv("COLLECTFAST_THREADS", 10))
    
    COLLECTFAST_DEBUG = os.getenv("COLLECTFAST_DEBUG") == "true"
    
    COLLECTFAST_REDIS_CACHE = {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": os.environ.get("COLLECTFAST_REDIS_URL", "redis://redis/0"),
        "KEY_PREFIX": "collectfast",
        "OPTIONS": {
            "SOCKET_CONNECT_TIMEOUT": int(os.getenv("REDIS_SOCKET_CONNECT_TIMEOUT", default=2)),
            "SOCKET_TIMEOUT": int(os.getenv("REDIS_SOCKET_TIMEOUT", default=2)),
            "MAX_ENTRIES": MAX_ENTRIES,
            "IGNORE_EXCEPTIONS": os.getenv("COLLECTFAST_IGNORE_EXCEPTIONS") == "true",
        },
    }
    

    And my .env

    STATICFILES_STORAGE="storages.backends.s3boto3.S3Boto3Storage"
    COLLECTFAST_STRATEGY="collectfast.strategies.boto3.Boto3Strategy"
    MAX_ENTRIES=3000
    COLLECTFAST_ENABLED=true
    COLLECTFAST_THREAD=20
    COLLECTFAST_DEBUG=true
    AWS_DEFAULT_ACL=private
    
    COLLECTFAST_CACHE=collectfast_redis_cache
    COLLECTFAST_REDIS_URL="redis://0.0.0.0:6379/0"
    COLLECTFAST_IGNORE_EXCEPTIONS="true"
    

    WDYT about that?

    opened by vschoener 5
  • Set the test suite regardless of actual cloud connection.

    Set the test suite regardless of actual cloud connection.

    Now we can run the test suite independently without the actual access key of AWS and GCP.

    • mock boto3.s3 before starting tests with moto
    • mock GoogleCloud Client, Bucket, Blob classes.
    opened by mode9 2
  • Make test suite independent of original maintainer

    Make test suite independent of original maintainer

    Currently the test suite has been running against real accounts on AWS and Google Cloud. For multiple reasons it would be great to replace this with mocks.

    I've attempted to do this for S3 a few years back with moto, and failed, hopefully moto has improved since then and it should be attempted again.

    One also needs to look into something equivalent for Google Cloud.

    opened by antonagestam 2
  • ⚠️ Looking for maintainers

    ⚠️ Looking for maintainers

    Hi there 👋

    I am looking for a maintainer to take over this project. I am no longer working with projects that uses the collectstatic framework and therefor can't be as invested as a maintainer of this project should preferably be.

    Please get in contact if you are willing to take over.

    opened by antonagestam 9
  • Don't suppress errors when using threads

    Don't suppress errors when using threads

    When I was using collectfast in a CI system, some of the assets were not copied. Sometimes it copied all, sometimes just a subset without any errors.

    This is what I experienced:

    ~ $ python manage.py collectstatic --noinput --disable-collectfast
    1249 static files copied, 59 unmodified.
    ~ $
    ~ $ echo 'Deleting the files from AWS now'
    Deleting the files from AWS now
    ~ $
    ~ $ python manage.py collectstatic --noinput
    790 static files copied.
    ~ $ python manage.py collectstatic --noinput
    518 static files copied.
    ~ $ python manage.py collectstatic --noinput
    0 static files copied.
    ~ $ python manage.py collectstatic --noinput
    0 static files copied.
    ~ $
    ~ $ echo 'Deleting the files from AWS now'
    Deleting the files from AWS now
    ~ $
    ~ $ python manage.py collectstatic --noinput
    697 static files copied.
    ~ $ python manage.py collectstatic --noinput
    611 static files copied.
    ~ $ python manage.py collectstatic --noinput
    0 static files copied.
    ~ $ python manage.py collectstatic --noinput
    0 static files copied.
    ~ $
    ~ $
    

    It turned out, that the redis cache I used has some connection limits, so in the background I was having redis.exceptions.ConnectionError: max number of clients reached errors without being notified.

    The collectstatic build step succeeded all times while having these errors and not copying some files.

    The proposed fix simply waits for all the results and re-raises any errors, so the collectstatic command will fail loudly when an error happens in at least one thread.

    (I wanted to write a test for this, but wasn't able to get the test suite running so I gave it up.)

    opened by lepilepi 3
Releases(2.2.0)
  • 2.2.0(Jun 5, 2020)

    • Add post_copy_hook and on_skip_hook to collectfast.strategies.base.Strategy.
    • Add collectfast.strategies.filesystem.CachingFileSystemStrategy.
    • Fix a bug where files weren't properly closed when read for hashing.
    • Fix a bug where gzip compression level was inconsistent with S3.
    Source code(tar.gz)
    Source code(zip)
  • 2.1.0(Mar 2, 2020)

    • Use concurrent.futures.ThreadPoolExecutor instead of multiprocessing.dummy.Pool for parallel uploads. (#187)
    • Support post_process() (#188)
    Source code(tar.gz)
    Source code(zip)
  • 2.0.1(Feb 24, 2020)

  • 2.0.0(Feb 23, 2020)

    • Drop support for Python 3.5.
    • Drop support for Django 1.11.
    • Drop support for storages.backends.s3boto.S3BotoStorage (remove collectfast.strategies.boto.BotoStrategy).
    • Drop support for guessing strategies, e.g. require COLLECTFAST_STRATEGY to be set.
    • Package type hints.
    • Support django-storages 1.9+.
    • Validate types of settings.
    Source code(tar.gz)
    Source code(zip)
  • 1.3.2(Feb 16, 2020)

  • 1.3.0(Oct 22, 2019)

    • Fix #157: catch pydoc.ErrorDuringImport
    • Remove runtests.py and use pytest instead
    • Add Python 3.8 to test matrix
    • Update copyright note in readme
    • Add deprecation warning for using BotoStrategy
    Source code(tar.gz)
    Source code(zip)
  • 1.2.0(Oct 6, 2019)

  • 1.1.1(Oct 6, 2019)

  • 1.1.0(Oct 6, 2019)

    Large rewrite to simplify implementing support for more storage backends. Introduces a Strategy abstraction that can be subclassed to implement support for different storage backends and needs.

    Misc.:

    • Remove --ignore-etag option
    • Get rid of wrapping quotes in hashes, bump cache key version
    • Command tests now run with both boto and boto3
    Source code(tar.gz)
    Source code(zip)
  • 1.0.0(Aug 12, 2019)

  • 0.6.0(Dec 8, 2017)

  • 0.5.2(Feb 8, 2017)

  • 0.5.1(Jan 10, 2017)

  • 0.5.0(Jan 4, 2017)

  • 0.4.1(Dec 30, 2016)

  • 0.4.0(Dec 17, 2016)

    0.4.0 introduces parallel file uploads using Python's multiprocessing module. This enables huge speed-ups for first time uploads. Using testdata with 50 files at 500kb each upload finished in 1m9s using 0.3.1 and in just 10s using 0.4.0.

    Source code(tar.gz)
    Source code(zip)
  • 0.3.0(Nov 2, 2016)

    • Added support for boto3
    • Changed caching mechanism to only store etag, should be lighter on the cache backend
    • Updated tests to include Django 1.10 and updated docs
    Source code(tar.gz)
    Source code(zip)
  • 0.2.2(Apr 22, 2015)

  • 0.2.0(Apr 10, 2014)

    • Fix normalized_path bug
    • Better cache documentation
    • Introduced COLLECTFAST_ENABLED setting
    • Override AWS_PRELOAD_METADATA if not set to True
    • Adds testing! :)
    Source code(tar.gz)
    Source code(zip)
  • 0.1.14(Mar 13, 2014)

  • 0.1.13(Oct 24, 2013)

  • 0.1.12(Oct 21, 2013)

  • 0.1.11(Jul 22, 2013)

    This release reduces the default verbosity an only outputs a summary with the total number of skipped files and a benchmark of how long time it took to run the collect method. The biggest change is that the Command class now overrides the handle method. This might introduce problems for people using other Django versions than 1.4. Please submit issues if this is the case.

    To get all the "skipped file x" logs back you should use:

    $ python manage.py collectstatic --verbosity=2
    
    Source code(tar.gz)
    Source code(zip)
A faster pytorch implementation of faster r-cnn

A Faster Pytorch Implementation of Faster R-CNN Write at the beginning [05/29/2020] This repo was initaited about two years ago, developed as the firs

Jianwei Yang 7.1k Jan 1, 2023
A cd command that learns - easily navigate directories from the command line

NAME autojump - a faster way to navigate your filesystem DESCRIPTION autojump is a faster way to navigate your filesystem. It works by maintaining a d

William Ting 14.5k Jan 3, 2023
A command-line based, minimal torrent streaming client made using Python and Webtorrent-cli. Stream your favorite shows straight from the command line.

A command-line based, minimal torrent streaming client made using Python and Webtorrent-cli. Installation pip install -r requirements.txt It use

Jonardon Hazarika 17 Dec 11, 2022
commandpack - A package of modules for working with commands, command packages, files with command packages.

commandpack Help the project financially: Donate: https://smartlegion.github.io/donate/ Yandex Money: https://yoomoney.ru/to/4100115206129186 PayPal:

null 4 Sep 4, 2021
Aiminsun 165 Dec 21, 2022
AML Command Transfer. A lightweight tool to transfer any command line to Azure Machine Learning Services

AML Command Transfer (ACT) ACT is a lightweight tool to transfer any command from the local machine to AML or ITP, both of which are Azure Machine Lea

Microsoft 11 Aug 10, 2022
Quadruped-command-tracking-controller - Quadruped command tracking controller (flat terrain)

Quadruped command tracking controller (flat terrain) Prepare Install RAISIM link

Yunho Kim 4 Oct 20, 2022
Ros command - Unifying the ROS command line tools

Unifying the ROS command line tools One impairment to ROS 2 adoption is that all

null 37 Dec 15, 2022
A Python-based command prompt concept which includes windows command emulation.

PythonCMD A Python-based command prompt concept which includes windows command emulation. Current features: echo: Input your message and it will be cl

null 1 Feb 5, 2022
A faster and highly-compatible implementation of the Python programming language. The code here is out of date, please follow our blog

Pyston is a faster and highly-compatible implementation of the Python programming language. Version 2 is currently closed source, but you can find the

null 4.9k Dec 21, 2022
AWS DeepRacer Free Student Workshop: Run faster by using your custom waypoints

AWS DeepRacer Free Student Workshop: Run faster by using your custom waypoints Reward Function Template for waypoints def reward_function(params):

Yuen Cheuk Lam 88 Nov 27, 2022
Better directory iterator and faster os.walk(), now in the Python 3.5 stdlib

scandir, a better directory iterator and faster os.walk() scandir() is a directory iteration function like os.listdir(), except that instead of return

Ben Hoyt 506 Dec 29, 2022
Python library to make development of portfolio analysis faster and easier

Trafalgar Python library to make development of portfolio analysis faster and easier Installation ?? For the moment, Trafalgar is still in beta develo

Santosh Passoubady 641 Jan 1, 2023
Rotational region detection based on Faster-RCNN.

R2CNN_Faster_RCNN_Tensorflow Abstract This is a tensorflow re-implementation of R2CNN: Rotational Region CNN for Orientation Robust Scene Text Detecti

UCAS-Det 581 Nov 22, 2022
Yahoo! Finance market data downloader (+faster Pandas Datareader)

Yahoo! Finance market data downloader Ever since Yahoo! finance decommissioned their historical data API, many programs that relied on it to stop work

Ran Aroussi 8.4k Jan 1, 2023
Library for faster pinned CPU <-> GPU transfer in Pytorch

SpeedTorch Faster pinned CPU tensor <-> GPU Pytorch variabe transfer and GPU tensor <-> GPU Pytorch variable transfer, in certain cases. Update 9-29-1

Santosh Gupta 657 Dec 19, 2022
Python disk-backed cache (Django-compatible). Faster than Redis and Memcached. Pure-Python.

DiskCache is an Apache2 licensed disk and file backed cache library, written in pure-Python, and compatible with Django.

Grant Jenks 1.7k Jan 5, 2023
LeViT a Vision Transformer in ConvNet's Clothing for Faster Inference

LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference This repository contains PyTorch evaluation code, training code and pretrained

Facebook Research 504 Jan 2, 2023
Making the process of downloading youtube videos faster and more convinient.

Easy-YT Making the process of downloading youtube videos faster and more convinient. What can it do? This python script can be used to download youtub

Meynam 39 Nov 15, 2021