Pipeline is an asset packaging library for Django.

Overview

Pipeline

Jazzband GitHub Actions Coverage Documentation Status

Pipeline is an asset packaging library for Django, providing both CSS and JavaScript concatenation and compression, built-in JavaScript template support, and optional data-URI image and font embedding.

Installation

To install it, simply:

pip install django-pipeline

Documentation

For documentation, usage, and examples, see : https://django-pipeline.readthedocs.io

Comments
  • Load times have increased dramatically since upgrade to 1.4.x

    Load times have increased dramatically since upgrade to 1.4.x

    Hello,

    We just tried upgrading django-pipeline from version 1.3.25 to 1.4.x (multiple minor versions tested, including latest: 1.4.6). Every request rendering static files went from 0.5 sec to about 20 secs to render.

    We tried to debug what led to this increase but could not find it. After downgrading to 1.3 the load times were again at 0.5 secs per page.

    opened by bobvoorneveld 44
  • When less file exists in a different folder depth than output, staticfiles dies looking for images

    When less file exists in a different folder depth than output, staticfiles dies looking for images

    I could swear this worked in 1.1.27...

    When any of my less files specify background: url(''), the url is relative assuming that all compressed CSS will end up in css/*.css, and therefore the image urls being with '../img/'.

    However, it seems that pipeline has passed paths off to django staticfiles that are all relative to their original less files, which leads to the image not being found:

    Traceback (most recent call last):
    ...
      File "/Users/steven/virtualenvs/simple-pipeline/lib/python2.7/site-packages/staticfiles/management/commands/collectstatic.py", line 163, in handle_noargs
        collected = self.collect()
      File "/Users/steven/virtualenvs/simple-pipeline/lib/python2.7/site-packages/staticfiles/management/commands/collectstatic.py", line 120, in collect
        for original_path, processed_path, processed in processor:
      File "/Users/steven/virtualenvs/simple-pipeline/lib/python2.7/site-packages/staticfiles/storage.py", line 270, in post_process
        content = pattern.sub(converter, content)
      File "/Users/steven/virtualenvs/simple-pipeline/lib/python2.7/site-packages/staticfiles/storage.py", line 213, in converter
        hashed_url = self.url(unquote(joined_result), force=True)
      File "/Users/steven/virtualenvs/simple-pipeline/lib/python2.7/site-packages/staticfiles/storage.py", line 160, in url
        hashed_name = self.hashed_name(clean_name).replace('\\', '/')
      File "/Users/steven/virtualenvs/simple-pipeline/lib/python2.7/site-packages/staticfiles/storage.py", line 123, in hashed_name
        (clean_name, self))
    ValueError: The file 'less/img/wilford-brimley-diabeetus-cat.jpg' could not be found with <simple_pipeline.myapp.pipeline_storage.S3PipelineStorage object at 0x1027f2a90>.
    

    No to go figure out if/where this ever worked...

    [1] http://pastebin.com/MGtcj9rA (full stacktrace here)

    question 
    opened by estebistec 27
  • Do not create compiled JS and CSS in source directory

    Do not create compiled JS and CSS in source directory

    If I have a file, 'foo.coffee', when I reference that file through the tag, it creates a file next to it called 'foo.js'. The problem with this is it makes it very difficult to have a version control system ignore these files. It is very unlikely that I would want to blanket ignore .css or .js files, so that isn't an option. Instead, I'm left ignoring files individually when I add a new less or coffee file.

    It would be nice if pipeline placed the files in their own directory, so I could blanket ignore that directory. As an example, if I have '{static}/foo/bar.coffee', pipeline could place the compiled file in '{static}/{pipelined}/foo/bar.js', where "pipelined" could be a configurable setting.

    opened by SoftwareMaven 21
  • FinderStorage don't write where it should

    FinderStorage don't write where it should

    https://github.com/cyberdelia/django-pipeline/blob/master/pipeline/storage.py#L89 <- bug here

    path is string and os.path.dirname(name) is string too.

    Searching substring in this case is so confusing - need more complicated logic.

    bug 
    opened by dreadatour 21
  • Less compiler not respecting relative paths with @import

    Less compiler not respecting relative paths with @import

    I'm running django-pipeline version 1.1.22 (latest), and the following isn't working yet in .less files:

     @import "bootstrap-1.4/bootstrap.less";
    

    The reason one would do this would be to include the Less library and be able to predefine variables and be able to use the mixins exposed by the library in one's own Less code. For instance:

    @basefont: 14px; /* in bootstrap, it's 13 by default, so we override it before it's imported */
    
    @import "bootstrap-1.4/bootstrap.less";
    
    .ghostly { /* we use bootstrap's defined .opacity mixin to create this effect. */
        .opacity(90);
    }
    

    Currently, this doesn't work and yields:

    Environment:
    
    
    Request Method: GET
    Request URL: http://localhost:8000/
    
    Django Version: 1.3
    Python Version: 2.7.1
    Installed Applications:
    ['django.contrib.auth',
     'django.contrib.contenttypes',
     'django.contrib.sessions',
     'django.contrib.sites',
     'django.contrib.messages',
     'django.contrib.admin',
     'django.contrib.admindocs',
     'django.contrib.staticfiles',
     'django_bcrypt',
     'pipeline',
     'reversion',
     'south',
     'myproject.web']
    Installed Middleware:
    ('django.middleware.common.CommonMiddleware',
     'django.contrib.sessions.middleware.SessionMiddleware',
     'django.middleware.csrf.CsrfViewMiddleware',
     'django.contrib.auth.middleware.AuthenticationMiddleware',
     'django.contrib.messages.middleware.MessageMiddleware')
    
    
    Template error:
    In template /home/rfkrocktk/Desktop/myproject/src/myproject/web/templates/desktop/root.html, error at line 6
       Caught CompilerError while rendering: /var/lib/gems/1.8/gems/less-2.0.7/lib/less/loader.rb:46:in `error': file 'bootstrap-1.4/bootstrap.less' wasn't found. (Less::ParseError)
        from at Function.importer (/var/lib/gems/1.8/gems/less-2.0.7/lib/less/js/lib/less/index.js:114:13)
        from at Object.push (/var/lib/gems/1.8/gems/less-2.0.7/lib/less/js/lib/less/parser.js:73:25)
        from at Object.<anonymous> (/var/lib/gems/1.8/gems/less-2.0.7/lib/less/js/lib/less/tree/import.js:30:17)
        from at Object.import (/var/lib/gems/1.8/gems/less-2.0.7/lib/less/js/lib/less/parser.js:953:28)
        from at $ (/var/lib/gems/1.8/gems/less-2.0.7/lib/less/js/lib/less/parser.js:103:24)
        from at Object.directive (/var/lib/gems/1.8/gems/less-2.0.7/lib/less/js/lib/less/parser.js:967:29)
        from at $ (/var/lib/gems/1.8/gems/less-2.0.7/lib/less/js/lib/less/parser.js:103:24)
        from at Object.primary (/var/lib/gems/1.8/gems/less-2.0.7/lib/less/js/lib/less/parser.js:428:80)
        from at $ (/var/lib/gems/1.8/gems/less-2.0.7/lib/less/js/lib/less/parser.js:103:24)
        from at Object.parse (/var/lib/gems/1.8/gems/less-2.0.7/lib/less/js/lib/less/parser.js:264:42)
        from /var/lib/gems/1.8/gems/less-2.0.7/lib/less/parser.rb:65:in `parse'
        from /var/lib/gems/1.8/gems/less-2.0.7/lib/less/parser.rb:13:in `calljs'
        from /var/lib/gems/1.8/gems/less-2.0.7/lib/less/parser.rb:27:in `lock'
        from /var/lib/gems/1.8/gems/less-2.0.7/lib/less/parser.rb:25:in `lock'
        from /var/lib/gems/1.8/gems/less-2.0.7/lib/less/parser.rb:12:in `calljs'
        from /var/lib/gems/1.8/gems/less-2.0.7/lib/less/parser.rb:63:in `parse'
        from /var/lib/gems/1.8/gems/less-2.0.7/bin/lessc:22
        from /usr/local/bin/lessc:19:in `load'
        from /usr/local/bin/lessc:19
    
       1 : {% load static %}
    
    
       2 : {% load compressed %}
    
    
       3 : <!DOCTYPE html>
    
    
       4 : <html>
    
    
       5 :  <head>
    
    
       6 :      {% block stylesheets %} {% compressed_css 'all' %} {% endblock %}
    
    
       7 :      <title>{% block title %}{% endblock %}&nbsp;&raquo;&nbsp;myproject</title>
    
    
       8 :  </head>
    
    
       9 :  <body>
    
    
       10 :         {% block body %}{% endblock %}
    
    
       11 :         {% block scripts %}{% compressed_js 'all' %}{% endblock %}
    
    
       12 :     </body>
    
    
       13 : </html>
    
    
       14 : 
    
    Traceback:
    File "/home/rfkrocktk/Desktop/myproject/eggs/Django-1.3-py2.7.egg/django/core/handlers/base.py" in get_response
      111.                         response = callback(request, *callback_args, **callback_kwargs)
    File "/home/rfkrocktk/Desktop/myproject/eggs/Django-1.3-py2.7.egg/django/views/generic/simple.py" in direct_to_template
      28.     return HttpResponse(t.render(c), mimetype=mimetype)
    File "/home/rfkrocktk/Desktop/myproject/eggs/Django-1.3-py2.7.egg/django/template/base.py" in render
      123.             return self._render(context)
    File "/home/rfkrocktk/Desktop/myproject/eggs/Django-1.3-py2.7.egg/django/template/base.py" in _render
      117.         return self.nodelist.render(context)
    File "/home/rfkrocktk/Desktop/myproject/eggs/Django-1.3-py2.7.egg/django/template/base.py" in render
      744.                 bits.append(self.render_node(node, context))
    File "/home/rfkrocktk/Desktop/myproject/eggs/Django-1.3-py2.7.egg/django/template/debug.py" in render_node
      73.             result = node.render(context)
    File "/home/rfkrocktk/Desktop/myproject/eggs/Django-1.3-py2.7.egg/django/template/loader_tags.py" in render
      127.         return compiled_parent._render(context)
    File "/home/rfkrocktk/Desktop/myproject/eggs/Django-1.3-py2.7.egg/django/template/base.py" in _render
      117.         return self.nodelist.render(context)
    File "/home/rfkrocktk/Desktop/myproject/eggs/Django-1.3-py2.7.egg/django/template/base.py" in render
      744.                 bits.append(self.render_node(node, context))
    File "/home/rfkrocktk/Desktop/myproject/eggs/Django-1.3-py2.7.egg/django/template/debug.py" in render_node
      73.             result = node.render(context)
    File "/home/rfkrocktk/Desktop/myproject/eggs/Django-1.3-py2.7.egg/django/template/loader_tags.py" in render
      127.         return compiled_parent._render(context)
    File "/home/rfkrocktk/Desktop/myproject/eggs/Django-1.3-py2.7.egg/django/template/base.py" in _render
      117.         return self.nodelist.render(context)
    File "/home/rfkrocktk/Desktop/myproject/eggs/Django-1.3-py2.7.egg/django/template/base.py" in render
      744.                 bits.append(self.render_node(node, context))
    File "/home/rfkrocktk/Desktop/myproject/eggs/Django-1.3-py2.7.egg/django/template/debug.py" in render_node
      73.             result = node.render(context)
    File "/home/rfkrocktk/Desktop/myproject/eggs/Django-1.3-py2.7.egg/django/template/loader_tags.py" in render
      64.             result = block.nodelist.render(context)
    File "/home/rfkrocktk/Desktop/myproject/eggs/Django-1.3-py2.7.egg/django/template/base.py" in render
      744.                 bits.append(self.render_node(node, context))
    File "/home/rfkrocktk/Desktop/myproject/eggs/Django-1.3-py2.7.egg/django/template/debug.py" in render_node
      73.             result = node.render(context)
    File "/home/rfkrocktk/Desktop/myproject/eggs/django_pipeline-1.1.22-py2.7.egg/pipeline/templatetags/compressed.py" in render
      30.             package['paths'] = self.packager.compile(package['paths'])
    File "/home/rfkrocktk/Desktop/myproject/eggs/django_pipeline-1.1.22-py2.7.egg/pipeline/packager.py" in compile
      50.         return self.compiler.compile(paths)
    File "/home/rfkrocktk/Desktop/myproject/eggs/django_pipeline-1.1.22-py2.7.egg/pipeline/compilers/__init__.py" in compile
      25.                         compiled_content = compiler.compile_file(content, storage.path(path))
    File "/home/rfkrocktk/Desktop/myproject/eggs/django_pipeline-1.1.22-py2.7.egg/pipeline/compilers/less/__init__.py" in compile_file
      17.         content = self.execute_command(command)
    File "/home/rfkrocktk/Desktop/myproject/eggs/django_pipeline-1.1.22-py2.7.egg/pipeline/compilers/__init__.py" in execute_command
      88.             raise CompilerError(error)
    
    Exception Type: TemplateSyntaxError at /
    Exception Value: Caught CompilerError while rendering: /var/lib/gems/1.8/gems/less-2.0.7/lib/less/loader.rb:46:in `error': file 'bootstrap-1.4/bootstrap.less' wasn't found. (Less::ParseError)
        from at Function.importer (/var/lib/gems/1.8/gems/less-2.0.7/lib/less/js/lib/less/index.js:114:13)
        from at Object.push (/var/lib/gems/1.8/gems/less-2.0.7/lib/less/js/lib/less/parser.js:73:25)
        from at Object.<anonymous> (/var/lib/gems/1.8/gems/less-2.0.7/lib/less/js/lib/less/tree/import.js:30:17)
        from at Object.import (/var/lib/gems/1.8/gems/less-2.0.7/lib/less/js/lib/less/parser.js:953:28)
        from at $ (/var/lib/gems/1.8/gems/less-2.0.7/lib/less/js/lib/less/parser.js:103:24)
        from at Object.directive (/var/lib/gems/1.8/gems/less-2.0.7/lib/less/js/lib/less/parser.js:967:29)
        from at $ (/var/lib/gems/1.8/gems/less-2.0.7/lib/less/js/lib/less/parser.js:103:24)
        from at Object.primary (/var/lib/gems/1.8/gems/less-2.0.7/lib/less/js/lib/less/parser.js:428:80)
        from at $ (/var/lib/gems/1.8/gems/less-2.0.7/lib/less/js/lib/less/parser.js:103:24)
        from at Object.parse (/var/lib/gems/1.8/gems/less-2.0.7/lib/less/js/lib/less/parser.js:264:42)
        from /var/lib/gems/1.8/gems/less-2.0.7/lib/less/parser.rb:65:in `parse'
        from /var/lib/gems/1.8/gems/less-2.0.7/lib/less/parser.rb:13:in `calljs'
        from /var/lib/gems/1.8/gems/less-2.0.7/lib/less/parser.rb:27:in `lock'
        from /var/lib/gems/1.8/gems/less-2.0.7/lib/less/parser.rb:25:in `lock'
        from /var/lib/gems/1.8/gems/less-2.0.7/lib/less/parser.rb:12:in `calljs'
        from /var/lib/gems/1.8/gems/less-2.0.7/lib/less/parser.rb:63:in `parse'
        from /var/lib/gems/1.8/gems/less-2.0.7/bin/lessc:22
        from /usr/local/bin/lessc:19:in `load'
        from /usr/local/bin/lessc:19
    
    bug awaiting response 
    opened by naftulikay 17
  • Compiler @import don't work

    Compiler @import don't work

    With the below situation, our 'base' bundle is still being compressed and written to a file called 'css/base.css' (where it should be 'css/thatguyismystyle.base.min.css' )

    The template tag still writes out the desired filename (which is 'css/thatguyismystyle.base.min.css' )


    Django==1.3
    django-pipeline==1.2.5
    django-staticfiles==1.2.1
    

        from os.path import join
        from settings import SERVER_ROOT
        from collections import OrderedDict as OD
    
        NODE_BIN = join(SERVER_ROOT, 'node_modules/.bin')
    
        #
        # Pipelined CSS Files
        #
    
    
        PIPELINE_CSS = OD((
            ('base', {
                'source_filenames': (
                    'css/base.less',
                ),
    
                'output_filename':
                    'css/thatguyismystyle.base.min.css',
    
                'extra_context': {
                    'media': 'screen',
                },
            }),
            ('theme', {
                'source_filenames': (
                    'css/theme/base.less',
                ),
                'output_filename':
                    'css/theme/thatguyismystyle.theme.min.css',
    
                'extra_context': {
                    'media': 'screen',
                },
            }),
            ('theme-light', {
                'source_filenames': (
                    'themes/light/css/base.less'
                ),
                'output_filename':
                    'themes/thatguyismystyle.colour.light.min.css',
                'extra_context': {
                    "media": "screen",
                }
            }),
            ('theme-dark', {
                'source_filenames': (
                    'themes/dark/css/base.less'
                ),
                'output_filename':
                    'themes/thatguyismystyle.colour.dark.min.css',
                'extra_context': {
                    "media": "screen",
                }
            }),
        ))
    
        #
        # Pipelined JS FILES
        #
    
        BASE_JS_FRAMEWORK = (
            "js/lib/modernizr.min.js",
            "js/lib/jquery.min.1.7.1.js",
            "js/lib/history.js/jquery.history.js",
            "js/lib/moment.min.pretty.js",
            "js/lib/moment.strftime.min.js",
        )
        BOOTSTRAP_JS = (
            'js/lib/bootstrap/bootstrap-alert.js',
            'js/lib/bootstrap/bootstrap-button.js',
            'js/lib/bootstrap/bootstrap-carousel.js',
            'js/lib/bootstrap/bootstrap-collapse.js',
            'js/lib/bootstrap/bootstrap-dropdown.js',
            'js/lib/bootstrap/bootstrap-modal.js',
            'js/lib/bootstrap/bootstrap-popover.js',
            'js/lib/bootstrap/bootstrap-scrollspy.js',
            'js/lib/bootstrap/bootstrap-tab.js',
            'js/lib/bootstrap/bootstrap-tooltip.js',
            'js/lib/bootstrap/bootstrap-transition.js',
            'js/lib/bootstrap-contrib/*.js',
        )
        CKEDITOR_JS = (
            'js/lib/ckeditor/*.js',
        )
        TINYMCS_JS = (
            'js/lib/tinymce/tiny_mce.js',
            'js/lib/tinymce/langs/*.js',
            'js/lib/tinymce/jquery.tinymce.js',
        )
    
    
        PIPELINE_JS = OD((
            ('backend', {
                'source_filenames': (
                    'js/lib/jquery.js',
                ) + BOOTSTRAP_JS
                  + CKEDITOR_JS + (
                    'js/lib/jquery.*.js',
                    'js/lib/thatguyismystyle.*.js',
                    'js/application.js',
                ),
                'output_filename': 'js/.js',
            }),
            ('profile-pages', {
                'source_filenames': (
                ),
                'output_filename': 'js/profile.js'
            })
        ))
        PIPELINE_JS = dict()
    
        PIPELINE_DISABLE_WRAPPER = True
    
        #
        # Compilers
        #
    
        PIPELINE_STORAGE = 'staticfiles.storage.StaticFilesStorage'
        STATICFILES_STORAGE = 'staticfiles.storage.StaticFilesStorage'
        PIPELINE=True
    
        PIPELINE_COMPILERS = (
            'pipeline.compilers.less.LessCompiler',
            'pipeline.compilers.coffee.CoffeeScriptCompiler',
        )
        PIPELINE_LESS_BINARY = join( NODE_BIN, 'lessc' )
    
        #
        # Compressors
        #
        PIPELINE_JS_COMPRESSOR = 'pipeline.compressors.jsmin.JSMinCompressor'
        PIPELINE_CSS_COMPRESSOR = 'pipeline.compressors.cssmin.CssminCompressor'
    
    
        STATICFILES_FINDERS = (
         'staticfiles.finders.AppDirectoriesFinder',
        )
    
        STATICFILES_IGNORE_PATTERNS = (
            "*.txt",
            "tests",
            "css/*.old",
            "/opt/webfiles/common/*.txt",
            "/opt/webfiles/common/temp",
        )
    
    bug 
    opened by airtonix 15
  • Ignore missing files

    Ignore missing files

    In large projects that use dozens of third-party libraries, invariably, some library will reference a url(../../..somefile) which doesn't exist. When running manage.py collectstatic with django-pipeline, this will result in an incredibly unhelpful error like:

    ValueError: The file 'tinymce/js/tinymce/plugins/compat3x/img/icons.gif' could not be found with <pipeline.storage.PipelineCachedStorage object at 0x1ca6750>.
    

    Not only does pipeline not mention which file attempts to include this file, searching for img/icons.gif myself results in 42 matches. Even if I could locate the issue, I can't easily fix the missing file, since it's a third-party library. This means pipeline is effectively blocking all my deployments because of a missing file I don't even use.

    Ideally, pipeline should report which resource is referencing a missing file and allow specifying an option to ignore it. This is very similar to the https://github.com/adamcharnock/django-pipeline-forgiving fork, although I think this would be a good idea for the main branch.

    opened by chrisspen 14
  • 1.4.2 requires absolute paths support from static storage

    1.4.2 requires absolute paths support from static storage

    I'm using the following static storage:

    class S3PipelineStorage(PipelineMixin, S3BotoStorage):
        """S3 storage with pipeline magic"""
        pass
    

    Which is almost the same as the one from pipeline docs (even simpler because it doesnt use CachedFilesMixin) . It's fine with 1.3.x but breaks with 1.4.2 because it tries to run sass compiler using absolute path from s3. S3BotoStorage doesn't support absolute paths and throws this long exception:

    /home/bender/myapps/foo/manage.py in <module>()
          8     from django.core.management import execute_from_command_line
          9 
    ---> 10     execute_from_command_line(sys.argv)
    
    /home/bender/.virtualenvs/foo/lib/python2.7/site-packages/django/core/management/__init__.pyc in execute_from_command_line(argv)
        383     """
        384     utility = ManagementUtility(argv)
    --> 385     utility.execute()
    
    /home/bender/.virtualenvs/foo/lib/python2.7/site-packages/django/core/management/__init__.pyc in execute(self)
        375             sys.stdout.write(self.main_help_text() + '\n')
        376         else:
    --> 377             self.fetch_command(subcommand).run_from_argv(self.argv)
        378 
        379 
    
    /home/bender/.virtualenvs/foo/lib/python2.7/site-packages/django/core/management/base.pyc in run_from_argv(self, argv)
        286         handle_default_options(options)
        287         try:
    --> 288             self.execute(*args, **options.__dict__)
        289         except Exception as e:
        290             if options.traceback or not isinstance(e, CommandError):
    
    /home/bender/.virtualenvs/foo/lib/python2.7/site-packages/django/core/management/base.pyc in execute(self, *args, **options)
        336                     not options.get('skip_checks')):
        337                 self.check()
    --> 338             output = self.handle(*args, **options)
        339             if output:
        340                 if self.output_transaction:
    
    /home/bender/.virtualenvs/foo/lib/python2.7/site-packages/django/core/management/base.pyc in handle(self, *args, **options)
        531         if args:
        532             raise CommandError("Command doesn't accept any arguments")
    --> 533         return self.handle_noargs(**options)
        534 
        535     def handle_noargs(self, **options):
    
    /home/bender/.virtualenvs/foo/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.pyc in handle_noargs(self, **options)
        166             raise CommandError("Collecting static files cancelled.")
        167 
    --> 168         collected = self.collect()
        169         modified_count = len(collected['modified'])
        170         unmodified_count = len(collected['unmodified'])
    
    /home/bender/.virtualenvs/foo/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.pyc in collect(self)
        112             processor = self.storage.post_process(found_files,
        113                                                   dry_run=self.dry_run)
    --> 114             for original_path, processed_path, processed in processor:
        115                 if isinstance(processed, Exception):
        116                     self.stderr.write("Post-processing '%s' failed!" % original_path)
    /home/bender/.virtualenvs/foo/lib/python2.7/site-packages/pipeline/storage.pyc in post_process(self, paths, dry_run, **options)
         28         from pipeline.packager import Packager
         29         packager = Packager(storage=self)
    ---> 30         for package_name in packager.packages['css']:
         31             package = packager.package_for('css', package_name)
         32             output_file = package.output_filename
    
    /home/bender/.virtualenvs/foo/lib/python2.7/site-packages/pipeline/packager.pyc in pack_stylesheets(self, package, **kwargs)
         94         return self.pack(package, self.compressor.compress_css, css_compressed,
         95                          output_filename=package.output_filename,
    ---> 96                          variant=package.variant, **kwargs)
         97 
         98     def compile(self, paths, force=False):
    
    /home/bender/.virtualenvs/foo/lib/python2.7/site-packages/pipeline/packager.pyc in pack(self, package, compress, signal, **kwargs)
        103         if self.verbose:
        104             print("Saving: %s" % output_filename)
    --> 105         paths = self.compile(package.paths, force=True)
        106         content = compress(paths, **kwargs)
        107         self.save_file(output_filename, content)
    
    /home/bender/.virtualenvs/foo/lib/python2.7/site-packages/pipeline/packager.pyc in compile(self, paths, force)
         97 
         98     def compile(self, paths, force=False):
    ---> 99         return self.compiler.compile(paths, force=force)
        100 
        101     def pack(self, package, compress, signal, **kwargs):
    
    /home/bender/.virtualenvs/foo/lib/python2.7/site-packages/pipeline/compilers/__init__.pyc in compile(self, paths, force)
         54         else:
         55             with futures.ThreadPoolExecutor(max_workers=multiprocessing.cpu_count()) as executor:
    ---> 56                 return list(executor.map(_compile, paths))
         57 
         58     def output_path(self, path, extension):
    
    /home/bender/.virtualenvs/foo/lib/python2.7/site-packages/concurrent/futures/_base.pyc in map(self, fn, *iterables, **kwargs)
        578             for future in fs:
        579                 if timeout is None:
    --> 580                     yield future.result()
        581                 else:
        582                     yield future.result(end_time - time.time())
    
    /home/bender/.virtualenvs/foo/lib/python2.7/site-packages/concurrent/futures/_base.pyc in result(self, timeout)
        398                 raise CancelledError()
        399             elif self._state == FINISHED:
    --> 400                 return self.__get_result()
        401 
        402             self._condition.wait(timeout)
    
    /home/bender/.virtualenvs/foo/lib/python2.7/site-packages/concurrent/futures/_base.pyc in __get_result(self)
        357     def __get_result(self):
        358         if self._exception:
    --> 359             reraise(self._exception, self._traceback)
        360         else:
        361             return self._result
    
    /home/bender/.virtualenvs/foo/lib/python2.7/site-packages/concurrent/futures/_compat.pyc in reraise(exc, traceback)
        105     def reraise(exc, traceback):
        106         locals_ = {'exc_type': type(exc), 'exc_value': exc, 'traceback': traceback}
    --> 107         exec('raise exc_type, exc_value, traceback', {}, locals_)
    /home/bender/.virtualenvs/foo/lib/python2.7/site-packages/concurrent/futures/thread.pyc in run(self)
         59 
         60         try:
    ---> 61             result = self.fn(*self.args, **self.kwargs)
         62         except BaseException:
         63             e, tb = sys.exc_info()[1:]
    
    /home/bender/.virtualenvs/foo/lib/python2.7/site-packages/pipeline/compilers/__init__.pyc in _compile(input_path)
         34                 if compiler.match_file(input_path):
         35                     output_path = self.output_path(input_path, compiler.output_extension)
    ---> 36                     infile = self.storage.path(input_path)
         37                     outfile = self.output_path(infile, compiler.output_extension)
         38                     outdated = compiler.is_outdated(input_path, output_path)
    
    /home/bender/.virtualenvs/foo/lib/python2.7/site-packages/django/core/files/storage.pyc in path(self, name)
         85         accessed using open() should *not* implement this method.
         86         """
    ---> 87         raise NotImplementedError("This backend doesn't support absolute paths.")
         88 
         89     # The following methods form the public API for storage systems, but with
    
    NotImplementedError: This backend doesn't support absolute paths
    
    opened by rjagielski 12
  • pipeline.exceptions.CompressorError

    pipeline.exceptions.CompressorError

    I occasionally (once every 10 runs or so?) get this error when django-pipeline is compressing my JS:

    Traceback (most recent call last):
      File "manage.py", line 10, in <module>
        execute_from_command_line(sys.argv)
      File "/home/vagrant/.virtualenvs/VBEZ/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
        utility.execute()
      File "/home/vagrant/.virtualenvs/VBEZ/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
        self.fetch_command(subcommand).run_from_argv(self.argv)
      File "/home/vagrant/.virtualenvs/VBEZ/local/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
        self.execute(*args, **options.__dict__)
      File "/home/vagrant/.virtualenvs/VBEZ/local/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
        output = self.handle(*args, **options)
      File "/home/vagrant/.virtualenvs/VBEZ/local/lib/python2.7/site-packages/django/core/management/base.py", line 415, in handle
        return self.handle_noargs(**options)
      File "/home/vagrant/.virtualenvs/VBEZ/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 173, in handle_noargs
        collected = self.collect()
      File "/home/vagrant/.virtualenvs/VBEZ/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 118, in collect
        for original_path, processed_path, processed in processor:
      File "/home/vagrant/.virtualenvs/VBEZ/local/lib/python2.7/site-packages/pipeline/storage.py", line 39, in post_process
        packager.pack_javascripts(package)
      File "/home/vagrant/.virtualenvs/VBEZ/local/lib/python2.7/site-packages/pipeline/packager.py", line 110, in pack_javascripts
        return self.pack(package, self.compressor.compress_js, js_compressed, templates=package.templates, **kwargs)
      File "/home/vagrant/.virtualenvs/VBEZ/local/lib/python2.7/site-packages/pipeline/packager.py", line 104, in pack
        content = compress(paths, **kwargs)
      File "/home/vagrant/.virtualenvs/VBEZ/local/lib/python2.7/site-packages/pipeline/compressors/__init__.py", line 65, in compress_js
        js = getattr(compressor(verbose=self.verbose), 'compress_js')(js)
      File "/home/vagrant/.virtualenvs/VBEZ/local/lib/python2.7/site-packages/pipeline/compressors/yuglify.py", line 13, in compress_js
        return self.compress_common(js, 'js', settings.PIPELINE_YUGLIFY_JS_ARGUMENTS)
      File "/home/vagrant/.virtualenvs/VBEZ/local/lib/python2.7/site-packages/pipeline/compressors/yuglify.py", line 10, in compress_common
        return self.execute_command(command, content)
      File "/home/vagrant/.virtualenvs/VBEZ/local/lib/python2.7/site-packages/pipeline/compressors/__init__.py", line 237, in execute_command
        raise CompressorError(stderr)
    pipeline.exceptions.CompressorError: Killed
    

    The only way I have found to continue is to restart the vagrant box that I am developing on.

    opened by ErikEvenson 12
  • AttributeError: 'CachedFileFinder' object has no attribute 'storages'

    AttributeError: 'CachedFileFinder' object has no attribute 'storages'

    I'm having the following problem. Not sure if it's a misconfiguration on my end or a bug.

    I'm using Bower to install some components. I've added to STATICFILES_DIRS the directory where the components are installed.

    Here are some relevant parts from settings.py

    STATICFILES_FINDERS = (
        'pipeline.finders.AppDirectoriesFinder',
        'pipeline.finders.FileSystemFinder',
        'pipeline.finders.PipelineFinder',
        'pipeline.finders.CachedFileFinder',
    )
    STATICFILES_STORAGE = 'pipeline.storage.PipelineCachedStorage'
    DEBUG = False
    

    When I run collectstatic, I get the following error:

    AttributeError: 'CachedFileFinder' object has no attribute 'storages'
    

    However, If I remove CachedFileFinder, collectstatic works fine.

    opened by honi 12
  • What should be happening?

    What should be happening?

    I've begun implementing Pipeline in our app. I've added the middleware, items to the requirements.txt file, the compressor variable, and the PIPELINE_JS object.

    PIPELINE_JS = {
        'activity': {
            'source_filenames': (
                'assets/js/libs/src/core.libs.js',
                'assets/js/libs/src/datepicker.js',
            ),
            'output_filename': 'assets/js/libs/activity.js',
        }
    }
    

    I've included the template tag on the page, and output the tag call like so:

    {% compressed_js 'activity' %}

    The page loads without error, but nothing appears to be happening. I wanted to make sure something should be happening at this point. I don't see any outputted files, and viewing the rendered source of the page shows an empty line where the call to the template was.

    So what should I expect is happening?

    Also, as a side note, it would be AWESOME if you could include a super simple app using pipeline as a demo. Something that users could download and run locally to see what happens.

    Looking forward to your response.

    opened by commadelimited 12
  • Add Django 4.1 support

    Add Django 4.1 support

    Only a small change was required in the tests to handle this change about removing the type="text/css" attributes for CSS link tags:

    2022-09-29_12-14


    It might make sense when releasing a new version to drop support for Python 3.6 and Django 2.2, 3.0 and 3.1 since they have all reached end of life. If you want I can push that in this PR or seperately

    opened by kevinmarsh 3
  • swapped the project_infile path with the infile path on the compile_f…

    swapped the project_infile path with the infile path on the compile_f…

    collectstatic static fails with the project_infile variable in the compile_file method because the finder looks for static files in the static directories not from static_root.

    opened by OzzyGrammer 0
  • [Question][Bug] Infile paramater value wrong after updating from 1.6.14 to 2.0.7

    [Question][Bug] Infile paramater value wrong after updating from 1.6.14 to 2.0.7

    We had to upgrade django-pipeline from 1.6.14 to at least 2.0.7 since we are currently in the process of upgrading django to version 3.2. After we made the update, our static files were not behaving the same anymore (css not loading). After a few days of investigating I saw that the infile parameter that is passed into the compile_file function in the LessCompiler class were not the same as the value we usually got before we made the update.

    Seems like the infile value comes from compile function on the Compiler class in the init file.

        def compile(self, paths, compiler_options={}, force=False):
            def _compile(input_path):
                for compiler in self.compilers:
                    compiler = compiler(verbose=self.verbose, storage=self.storage)
                    if compiler.match_file(input_path):
                        try:
                            infile = self.storage.path(input_path)
                        except NotImplementedError:
                            infile = finders.find(input_path)
                        project_infile = finders.find(input_path)
                        outfile = compiler.output_path(infile, compiler.output_extension)
                        outdated = compiler.is_outdated(project_infile, outfile)
                        compiler.compile_file(project_infile, outfile,
                                              outdated=outdated, force=force,
                                              **compiler_options)
    
                        return compiler.output_path(input_path, compiler.output_extension)
                else:
                    return input_path
    

    I see the infile value is never passed into the compile_file function. No matter what happens in the try except, another variable called project_infile is created underneath and passed to compile_file, which in our case has the wrong value. Infile has the correct value. On the older version that we used the project_infile didnt exist and the infile value was passed to compile_file. Thank you in advance.

    opened by Willem-Nieuwoudt 0
  • [Feature Request]: self.request to context in render_js

    [Feature Request]: self.request to context in render_js

    This is a feature request to add self.request to context in render_js on JavascriptNode.

    This would be handy for cases where request context variables need to be rendered in the js template, for example, with django-csp you would be able to add a nonce in a custom template without having to extend JavascriptNode and writing a new tag to use it.

    My suggestion is to add a key to extra_context which you can declare in settings with a boolean value that defaults to false. If true it adds self.request to context which will then be usable in the template.

    If it's something anybody else is interested in, I could create a pull request for it.

    opened by mfvictor 0
  • pipeline compression skips over js when using yuglify

    pipeline compression skips over js when using yuglify

    Hi there,

    I am a bit new to using js. I am having a strange output, where the functions inside a js file are being skipped over during compression using yuglify. I am getting no errors, except when checking the output the functions are not present!

    The file which is being ignored

    // application.js
    function changeValueProp() {
        if (document.getElementById('ShowHideProp').value == 'Show Properties'){
          document.getElementById('ShowHideProp').value = 'Hide Properties';
        }
        else{
          document.getElementById('ShowHideProp').value = 'Show Properties';
        }
      }
    

    In my pipeline config I am using

    {
        "JAVASCRIPT": {
         "application": {"source_filenames":  (f'{APP_STATIC_DIR}/application.js',f'{APP_STATIC_DIR}/application01.js',),
                                 "output_filename":    'compressed/js/application.js'},
                              },
        "YUGLIFY_BINARY": f"{BASE}/node_modules/yuglify/bin/yuglify",
    }
    

    I just checked that if I use "DISABLE_WRAPPER":True, the function from application.js is compressed and put in the output file. In ReadTheDocs, the DISABLE_WRAPPER:False is recommended for improving performance.

    Is it possible to use the wrapper without it skipping over the js described above ?

    BR

    SK

    opened by esskaey 0
Releases(2.0.8)
  • 2.0.8(Jan 10, 2022)

    • Added Django 4.0 compatibility. Thanks to @kevinmarsh (#760)
    • Add tests for Django 4.0, Python 3.9 and Python 3.10. Thank to @kevinmarsh (#739)
    • Introduce CODE_OF_CONDUCT.md for the project. Thank to @hugovk (#758)
    • Add precision in the documentation for PipelineCachedStorage. Thank to @gatsinski (#739)
    • Drop support for slimit compressor (#765) due to package not released an official version for Python 3 and not any new package release from 2013.
    • Edit github actions matrix: django 3.2.9 support python 3.10, remove python 4.0 (doesn't exist) and exclude pypy-3.8 for django-main.
    • Add .pre-commit-config.yaml. Thanks to @hugovk (#762)
    • Update package.json due to CoffeeScript on NPM has moved to "coffeescript"
    • Update setup.py with Django 4.0 and Python 3.10
    Source code(tar.gz)
    Source code(zip)
  • 2.0.7(Oct 21, 2021)

  • 2.0.6(Jan 18, 2021)

  • 2.0.5(Aug 16, 2020)

    • Adding Django 3.1 compatibility.
    • CachedStaticFilesStorage is removed from Django. Add a check of the current version to prevent error while importing. Thank to @vmsp
    • Context in django.template.base is removed from Django and not used anymore in django-pipeline.
    • Fixing widgets tests of django-pipeline due to Media.render_js change in Django. More information in Django ticket #31892
    Source code(tar.gz)
    Source code(zip)
  • 2.0.4(May 29, 2020)

  • 2.0.3(May 23, 2020)

    • Add subclass of ManifestStaticFilesStorage. Thank to @jhpinson
    • Change the documentation to use PipelineManifestStorage in configuration instead of PipelineCachedStorage now deprecated.
    • Change import MutableMapping from collections.abc. Thank to @colons
    • Fix Middleware to properly decode HTML. Thank to @gatsinski
    • Keep mimetypes as str. Thank to @benspaulding
    • Based on #642 add 'NonPackagingPipelineManifestStorage' and update the documentation: storages.rst. Thank to @kronion
    • Remove futures from pipeline setup.py requirements.
    Source code(tar.gz)
    Source code(zip)
  • 2.0.0(May 23, 2020)

    • Definitely drop the support of Python 2.
    • Drop support for Python 3.5 (not compatible with PEP 498).
    • Remove 'decorator.py' how was used for backward compatibility between python 2 and 3 for metaclass inheritance on PipelineFormMedia.
    • Replace 'format' by 'fstring' (PEP 498: Literal String Interpolation).
    • Remove of old imports form 'django.utils.six' and these fixes (1.7.0).
    • Remove tests of uncovered versions of Python and Django.
    • Replace tests for Pypy by Pypy3.
    • Explicitly specify when files are read / write in binary mode.
    • Set opening files for tests to deal with universal newlines.
    • Upgrade documentation version to 2.0 to follow the project version.
    Source code(tar.gz)
    Source code(zip)
  • 1.7.0(Dec 28, 2019)

    Release the last major version of django-pipeline working on Python 2.

    • Thank you for all the modifications made since version 1.6.14, which we cannot quote.
    • Apply an optimization for compilation. Thank to @blankser
    • Edit setup.py to follow the recommendation of the documentation. Thank to @shaneikennedy
    • Add tests for Django 3.0 and Python 3.8
    • Add alternatives imports for django.utils.six, who has been removed in Django 3.0
    Source code(tar.gz)
    Source code(zip)
  • 1.6.13(May 25, 2017)

    • Fix forward-slashed paths on Windows. Thanks to @etiago
    • Fix CSS URL detector to match quotes correctly. Thanks to @vskh
    • Add a compiler_options dict to compile, to allow passing options to custom compilers. Thanks to @sassanh
    • Verify support for Django 1.11. Thanks to @jwhitlock
    Source code(tar.gz)
    Source code(zip)
  • 1.6.12(Mar 13, 2017)

https://django-storages.readthedocs.io/

Installation Installing from PyPI is as easy as doing: pip install django-storages If you'd prefer to install from source (maybe there is a bugfix in

Josh Schneier 2.3k Jan 5, 2023
Pipeline is an asset packaging library for Django.

Pipeline Pipeline is an asset packaging library for Django, providing both CSS and JavaScript concatenation and compression, built-in JavaScript templ

Jazzband 1.4k Jan 3, 2023
Pipeline is an asset packaging library for Django.

Pipeline Pipeline is an asset packaging library for Django, providing both CSS and JavaScript concatenation and compression, built-in JavaScript templ

Jazzband 1.4k Aug 29, 2021
A library which implements low-level functions that relate to packaging and distribution of Python

What is it? Distlib is a library which implements low-level functions that relate to packaging and distribution of Python software. It is intended to

Python Packaging Authority 29 Oct 11, 2022
Alternate Python bindings for the Open Asset Import Library (ASSIMP)

Impasse A simple Python wrapper for assimp using cffi to access the library. Requires Python >= 3.7. It's a fork of PyAssimp, Assimp's official Python

Salad Dais 3 Sep 26, 2022
Python dependency management and packaging made easy.

Poetry: Dependency Management for Python Poetry helps you declare, manage and install dependencies of Python projects, ensuring you have the right sta

Poetry 23.1k Jan 1, 2023
Python PyPi staging server and packaging, testing, release tool

devpi: PyPI server and packaging/testing/release tool This repository contains three packages comprising the core devpi system on the server and clien

null 629 Jan 1, 2023
Python dependency management and packaging made easy.

Poetry: Dependency Management for Python Poetry helps you declare, manage and install dependencies of Python projects, ensuring you have the right sta

Poetry 23.2k Jan 5, 2023
A modern Python application packaging and distribution tool

PyOxidizer PyOxidizer is a utility for producing binaries that embed Python. The over-arching goal of PyOxidizer is to make complex packaging and dist

Gregory Szorc 4.5k Jan 7, 2023
Simplified packaging of Python modules

Flit is a simple way to put Python packages and modules on PyPI. It tries to require less thought about packaging and help you avoid common mistakes.

Thomas Kluyver 1.9k Jan 5, 2023
Solaris IPS: Image Packaging System

Solaris Image Packaging System Introduction The image packaging system (IPS) is a software delivery system with interaction with a network repository

Oracle 57 Dec 30, 2022
A simple weather tool. I made this as a way for me to learn Python, API, and PyPi packaging.

A simple weather tool. I made this as a way for me to learn Python, API, and PyPi packaging.

Clint E. 105 Dec 31, 2022
QPT-Quick packaging tool 前项式Python环境快捷封装工具

QPT - Quick packaging tool 快捷封装工具 GitHub主页 | Gitee主页 QPT是一款可以“模拟”开发环境的多功能封装工具,一行命令即可将普通的Python脚本打包成EXE可执行程序,与此同时还可轻松引入CUDA等深度学习加速库, 尽可能在用户使用时复现您的开发环境。

GT-Zhang 545 Dec 28, 2022
auto packaging for iOS

iOS Auto Packaging iOS自动打包脚本 准备 脚本第一次执行之前 先检查依赖, packaging目录下终端执行 pip3 install -r requirements.txt 运行 cd packaging packaging.py -h <help> -s <scheme>

DeeCo 17 Jul 23, 2022
Automatic packaging of the open-composite libs for OvGME

OvGME Packager for OpenXR – OpenComposite for DCS Note This repository is currently unsupported and needs to be migrated to the upstream OpenComposite

null 12 Nov 3, 2022
This is a repository for the Duke University Cloud Computing course project on Serveless Data Engineering Pipeline. For this project, I recreated the below pipeline.

AWS Data Engineering Pipeline This is a repository for the Duke University Cloud Computing course project on Serverless Data Engineering Pipeline. For

null 15 Jul 28, 2021
SNV calling pipeline developed explicitly to process individual or trio vcf files obtained from Illumina based pipeline (grch37/grch38).

SNV Pipeline SNV calling pipeline developed explicitly to process individual or trio vcf files obtained from Illumina based pipeline (grch37/grch38).

East Genomics 1 Nov 2, 2021
Two phase pipeline + StreamlitTwo phase pipeline + Streamlit

Two phase pipeline + Streamlit This is an example project that demonstrates how to create a pipeline that consists of two phases of execution. In betw

Rick Lamers 1 Nov 17, 2021
Using Data Science with Machine Learning techniques (ETL pipeline and ML pipeline) to classify received messages after disasters.

Using Data Science with Machine Learning techniques (ETL pipeline and ML pipeline) to classify received messages after disasters.

null 1 Feb 11, 2022
Udacity-api-reporting-pipeline - Udacity api reporting pipeline

udacity-api-reporting-pipeline In this exercise, you'll use portions of each of

Fabio Barbazza 1 Feb 15, 2022