Asset management for Python web development.

I want to use the JST filter to generate templates that I can access via JST['template'] in my javascript but I couldn't find documentation on how to actually use the filter, only that it exists.

my projects make use of django.views.static.serve in development mode which makes it really easy to work with static files that live in external dependencies (apps.) the combined and minimized files are built and checked in to the repo with each relevant commit (the web serving nodes aren't capable of running the filters and thus building the files at run time, which i wouldn't want to do even if they could.) i created a helper package, https://github.com/ross/webassets-dynamic, to allow webassets to work with django.views.static.serve based files.

that worked fine with 0.5, but with 0.6 in debug mode there's an existence check in _normalize_source_path, https://github.com/miracle2k/webassets/blob/master/src/webassets/env.py#L304, that's causing problems.

self.abspath(spath) will return something like /.shared-app/js/somefile.js. that path will be handled by a corresponding django.views.static.serve rule in shared-app and works fine, but that file doesn't live on the filesystem in STATIC_ROOT/.shared-app/js/somefile.js.

i assume the existence check is there to help give a better error message for people who are using django staticfiles, but it's getting in the way of the ability to use django.views.static.serve.

i'd be happy to try and work towards a solution in django_assets itself rather than relying on the extra package i created.

thoughts?

i've fixed my projects to version 0.5 for the time being, but i'd prefer not to have to do that long-term.

-rm

Here's an idea I'm currently planning to add as soon as a find the time, to find a common solution to multiple related features that are currently lacking, like cache busting via filename, using hashes to cache-bust (#6), or auto-rebuilding via hash (a TODO in the code).

All this should be possible without complicating things too much for users who don't need any of it. Comments are welcome.

We'll make the following changes:

• A env.versioner attribute defines what should be used as a version. This would be backed with classes like TimestampVersion (the default) or HashVersion, and you'd be free to create your own class, like MyGitRevVersion.
• assets.expire becomes a simple boolean, it controls whether a query string should be used for cache busting. Potentially it should also be renamed to make this clearer. What is added as a query string is determined by the versioner. By default nothing changes, but you can now expire using something else than the timestamp by defining a different versioner (see #6).
• A bundle's output filename now allows a %(version)s placeholder, and this is as an alternative for env.expire. Instead of a querystring for cache-busting, the filename itself would change. At least initially, those files would probably not be managed, so webassets would never delete old versions, although I suppose there's no reason this couldn't be done.

When manually building a bundle, this is what happens:

• If the output filename does not contain a placeholder, everything happens as it currently does.
• If the output filename does contain a placeholder, the versioner would be used to provide the "current" version. Each Version class needs to have this ability. TimestampVersion would simply use the max(mtime) of the bundle's source files. The HashVersion would use the has over all the source files. A GitVersion class might call the "git" executable, or maybe use git-python.
• (Maybe, not sure yet if it's necessary) The version to be built can be specified manually, via the management command, or low-level via an argument to bundle.build(),

The process for automatic (re)building:

• If the output filename does not contain a placeholder, again nothing needs to chance.
• If the output filename of a bundle contains a placeholder, we obviously can't just compare timestamps like we currently do, since we don't know the target output file. The decision to rebuild thus needs to be made by determining the latest version (see above) and checking if the filename exists.
• Thus, when using auto-building AND and filenames with placeholders, you want to make sure to use a versioner that is quick, i.e. maybe not HashVersion.
• "Auto-rebuilding" really is a function of the "version" concept. If a rebuild is required, then that must mean there is a new version. If you use GitVersion, then autobuilding doesn't make sense at all. The two things are related, so the architecture should be designed accordingly.
• I'm thinking renaming env.updater to env.auto_update or env.auto_build for clarification purposes and making it a boolean as well, and then letting the Version class provide the updater to use. There would probably be only a "TimestampUpdater" built in, but you could write your own. If a Version class does not provide an updater, like MyGitRevVersion likely wouldn't, then the env.auto_build option can't be used with that versioner.
• The advanced user could therefore still use HashVersion with expire=True to do hash-based cache busting (again, see #6), while still having automatic rebuilds by timestamp checking.

The existing cache system would be extended/used so that the current version of a bundle can be cached. The HashVersion would need this feature very much, and TimestampVersion would benefit as well (needing no stat calls). Of course, this can only provide a benefit if automatic rebuilding is not wanted.

Either the environment or bundles could probably be "pinned" to a specific version, which would make it possible in some scenarios to run with old version assets.

Hi all,

webassets is now using MEDIA_URL and MEDIA_ROOT settings. The meaning of these settings was changed in django 1.3 and webassets should use STATIC_URL and STATIC_ROOT for django 1.3 (or django < 1.3 + django-staticfiles).

There is a fork by lyschoening (https://github.com/lyschoening/webassets/) with django.contrib.staticfiles-related changes. As I can understand, it uses the following approach: replace MEDIA_* settings with STATIC_* settings and load files using staticfiles loader (source files are compressed, not the collected). This seems backwards-incompatible though.

I think the issue can be resolved this way:

1. Introduce ASSETS_URL and ASSETS_ROOT options. They should be equal to STATIC_ROOT and STATIC_URL by default and fall back to MEDIA_ROOT and MEDIA_URL if STATIC_* options are not defined. django-compressor has similar settings.
2. Do not load files using staticfiles loader, load them from ASSETS_ROOT. There can be 3 situations:

a) Production. Files are collected to STATIC_ROOT and combined by webassets. Staticfiles loader is useless (?) because files are collected and webassets can load them from STATIC_ROOT. b) Development. Files are served uncombined by staticfiles development view. Staticfiles loader is useless because files are not compressed and webassets doesn't have to load files. c) Development. Files are served by staticfiles development view and we want them to be combined. Webassets can't just put combined files to STATIC_ROOT because it is ignored(?) when development staticfiles serving is enabled.

So I think it is easier to always work with collected static files. If (c) support is important it may be added later.

The short version

I think I'm going to take a stab at making AssetsFinder properly serve up bundle files to collectstatic without requiring assets rebuild. What do you think?

The long version

With apologies for a bit of a thought in progress.

I know you're deep in work on the Versioner stuff, but I wanted to write a bit about what I've figured out in the meantime. One thing webassets doesn't do is help with cache busting on static assets other then bundles such as images. It turns out CachedStaticFilesStorage already does this beautifully. Without repeating the docs too much, what it does on collectstatic is:

• Create a copy of each static file renamed to include a hash of the file contents
• Process css files so that any references to other static files use the hashed version

In the short term, I've found that I can actually use this almost out of the box with webassets by running collecstatic again after assets rebuild. It nicely creates hashed/processed versions of the bundle files and saves them into the static folder, ready for uploading/whatever. The following template code ties it all together:

{% assets "site_css" %}
    <link rel="stylesheet" type="text/css" href="{% static ASSET_URL %}">
{% endassets %}

There only real catch was that I had to set ASSETS_ROOT to something other then STATIC_ROOT. Since collectstatic makes copies of any files it finds and AssetsFinder will happily return anything it finds in ASSETS_ROOT, every run of collectstatic would create hashed versions of the already hashed files.

It's not a big deal, but I think this hints at a few issues I'm planning to poke at. I'm not sure of all the issues that might come from these, I'm just thinking out loud.

• Ideally assets rebuild would not be required. Instead collectstatic would just grab all of the bundled files from the AssetsFinder and do its hashing magic. What happens is that collecstatic uses list method to see which files a storage can offer. AssetsFileStorage only returns files that already exist in ASSETS_ROOT. Adding a custom list should fix this.
• Eliminating a standalone command might cause issues with the --parse-templates option. I don't use this, but I suppose you could have a command that wraps collectstatic or something. Not sure about this.

I'm not sure how this would affect (or is affected by) your versioning stuff. It turns out with this workflow I don't actually use the webassets expiring features. I know you support more then just Django, but right now webassets is still missing anything to do with non bundle assets such as images. It might not be too hard to add that functionality (renaming other assets + processing files that reference them), but until you do I'll use what's there.

It seems that the cssrewrite filter doesn't work when put after the scss filter if as_output is set to True. I was able to reproduce with a minimal use-case:

# foo.py
import webassets
from webassets.filter import get_filter

env = webassets.Environment(directory='.', url='/')

scss = get_filter(
    'scss',
    as_output=True,
    load_paths='.',
    style='compressed',
)

cssrewrite = get_filter('cssrewrite', replace={'static/': 'assets/'})

bundle = webassets.Bundle(
    'file1.scss', 'file2.scss',
    filters=(scss, cssrewrite),
    output='packed.css'
)
env.register('my_bundle', bundle)
bundle.build()

// file1.scss
@import "foo";

// foo.scss
body {
  background-image: url("static/paper.gif")
}

// file2.scss
a {
  color: red;
}

Here is what happens when as_output=True:

body{background-image:url("static/paper.gif")}a{color:red}

As you can see the URL is not rewritten to assets/paper.gif.

But with as_output=False:

body{background-image:url("assets/paper.gif")}

a{color:red}

The URL is correctly rewritten but the output is not a one-liner and it's also much slower to compile (not obvious on this small test case but it is on bigger projects).

Am I misunderstanding how this is supposed to work or is it a bug?

I'm using webassets via Flask-Assets. I have:

UglifyJS 2.4.15 webassets 0.11dev.

The UglifyJS filter produces an empty file. Using the uglifyjs tool from the command line produces minified files as expected. Note: jsmin and rjsmin both also works as filters using the same setup. I tried providing the file location in the asset configuration as well, to no avail. :(

Hi,

The bootstrap 2.0 js files minified with rjsmin throw an error: "Uncaught SyntaxError: Unexpected token !"

Problem is the !function() {} syntax

So I'm trying to configure an application that uses webassets so that it can be deployed by multiple Unix users, utilizing group permissions so that all changes from one user are writable by all users in the group.

Running into a roadblock with webassets, because it forces all files created in the webassets-cache directory to be created without group write or read permissions, thus if one user ends up writing a file as part of their webassets build, they will blow up the build for all other users.

I have the group sticky bit set on the webasset-cache directory, which makes all files created by default in that directory have the group read/write permissions enabled. But because of the use of mkstemp when webassets creates the file, it guarantees the group read/write bits will not be set on the resulting file. See: https://github.com/miracle2k/webassets/blob/master/src/webassets/cache.py#L202

I think the solution might be to add a new configurable in the webassets config, which specifies the permissions that you want the webassets cache files to have (default to 0600 as now), and then explicitly calling os.chmod before the rename. That way in my configuration I could set the permissions to 0660 and it'd be fine. Thoughts?

When using with Environment.debug = True it raise an exception like this :

Traceback (most recent call last):
File "zbuilder.py", line 93, in <module>
    print assets_env[item].urls()
File "/home/django/Emencia/parrotzik_project/lib/python2.6/site-packages/webassets/bundle.py", line 662, in urls
    urls.extend(bundle._urls(env, extra_filters, *args, **kwargs))
File "/home/django/Emencia/parrotzik_project/lib/python2.6/site-packages/webassets/bundle.py", line 644, in _urls
    url = env.resolver.resolve_source_to_url(external, org)
File "/home/django/Emencia/parrotzik_project/lib/python2.6/site-packages/webassets/env.py", line 296, in resolve_source_to_url
    return self.query_url_mapping(filepath)
File "/home/django/Emencia/parrotzik_project/lib/python2.6/site-packages/webassets/env.py", line 234, in query_url_mapping
    raise ValueError('Cannot determine url for %s' % filepath)

This is how i init my webassets environment :

assets_env = Environment()
assets_env.debug = True
assets_env.url = '/static/'
assets_env.directory = '_build/static'
assets_env.load_path = ['sources']
assets_env.cache = ".webassets-cache"

Also a directory "webassets-external" is created at '_build/static/webassets-external' filled with some assets.

Starting on line 158 of requirejs.py, the RequireJS filter is attempting to infer the AMD module name.

        # extract the AMD module name
        name = kw.get('source')
        if not name:
            # ...
        kw['modname'] = path.splitext(name)[0]

The result of this is not guaranteed to be accurate. It would depend on how you've structured your project. Unless I'm missing something, this module name is not likely to be correct unless your Javascript application's point of entry is a top level module in static.

Say for example the structure of your static directory looked like mine often do:

/static
    /js
        /main.js
        /config.js
        /build.js
    /vendor
        /underscore
        /etc

The point of entry (main.js) would be interpreted to be js/main where as you may have intended ./js to be the baseUrl causing builds to fail, unless you prepended all your module names with js/. Since command line parameters are being assigned by the filter, the user cannot override these values as RequireJS gives precedence to command line parameters over the build profile.

It would be nice to be able to omit parameters passed to r.js in favor of values set in a build profile, but in this situation it is possible to "adjust" for the filter's command line parameters through the build profile. You can correct the module names / paths to be relative to the /js directory which would make the above directory structure work.

({
    paths: {
        'js': './'
        'underscore': '../vendor/underscore/underscore'
    }
})

There are small typos in:

• docs/upgrading.rst
• src/webassets/filter/init.py
• tests/test_bundle_various.py

Fixes:

• Should read incompatibility rather than incompatiblity.
• Should read existent rather than existant.
• Should read bytestring rather than byestring.

Semi-automated pull request generated by https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md

I'm using webassets, as part of a platform on Flask (CKAN) (so as built-in of Flask? I don't know).

I have a plugin that uses a js library, that uses some css themes, that are initially located relatively to each other: js.js js-themes/ default/ icons.png style.css

style.css references url(icons.png).

Using webassets to bundle style.css, just adding cssrewrite as a filter without config fails with a ValueError in query_url_mapping: "'Cannot determine url for %s' % filepath".

I see in the docs, get_filter('cssrewrite', replace={'old_directory':'/custom/path/'}) replace looks like what I need.

I see there is the documentation for adding a config section to a YAML (if I interpret correctly):

config:
    another_custom_config_value: foo
bundles:
    # ...

So, what should cssrewrite config look like via YAML? Variations on the following stabs-in-the-dark don't work or cause crashes parsing the YAML:

config:
  cssrewrite:
    - replace={'': 'assets/js-themes/default/'}

Given the following mwe SCSS file:

/* /tmp/file.scss: */
$baseFontFamily : "Droid Sans", sans-serif;

body {
  font: 14px/1.5 $baseFontFamily;
}

the following script snippet:

# webassets_bug.py:
from webassets import Bundle, Environment

bundle = Bundle('/tmp/file.scss', filters='scss', output='/tmp/out.css')
env = Environment(directory='/tmp/')
env.register('css', bundle)

print(bundle.urls())

fails with:

$ python webassets_bug.py
Traceback (most recent call last):
  File "webassets_bug.py", line 7, in <module>
    print(bundle.urls())
  File "/usr/lib/python3.7/site-packages/webassets/bundle.py", line 833, in urls
    urls.extend(bundle._urls(new_ctx, extra_filters, *args, **kwargs))
  File "/usr/lib/python3.7/site-packages/webassets/bundle.py", line 768, in _urls
    *args, **kwargs)
  File "/usr/lib/python3.7/site-packages/webassets/bundle.py", line 620, in _build
    force, disable_cache=disable_cache, extra_filters=extra_filters)
  File "/usr/lib/python3.7/site-packages/webassets/bundle.py", line 544, in _merge_and_apply
    kwargs=item_data)
  File "/usr/lib/python3.7/site-packages/webassets/merge.py", line 280, in apply
    return self._wrap_cache(key, func)
  File "/usr/lib/python3.7/site-packages/webassets/merge.py", line 222, in _wrap_cache
    content = func().getvalue()
  File "/usr/lib/python3.7/site-packages/webassets/merge.py", line 255, in func
    getattr(filter, type)(data, out, **kwargs_final)
  File "/usr/lib/python3.7/site-packages/webassets/filter/sass.py", line 148, in input
    self._apply_sass(_in, out, os.path.dirname(source_path))
  File "/usr/lib/python3.7/site-packages/webassets/filter/sass.py", line 142, in _apply_sass
    return self.subprocess(args, out, _in, cwd=child_cwd)
  File "/usr/lib/python3.7/site-packages/webassets/filter/__init__.py", line 529, in subprocess
    stderr.decode('utf-8').strip()))
webassets.exceptions.FilterError: scss: subprocess returned a non-success result code: 65, stdout=, stderr=(sass):3: Invalid CSS after "...ns", sans-serif": expected expression (e.g. 1px, bold), was ";" (Sass::SyntaxError)
	from /usr/lib/ruby/vendor_ruby/sass/scss/parser.rb:1278:in `expected'
	from /usr/lib/ruby/vendor_ruby/sass/script/lexer.rb:231:in `expected!'
	from /usr/lib/ruby/vendor_ruby/sass/script/parser.rb:762:in `assert_done'
	from /usr/lib/ruby/vendor_ruby/sass/script/parser.rb:69:in `parse'
	from /usr/lib/ruby/vendor_ruby/sass/script/parser.rb:228:in `parse'
	from /usr/lib/ruby/vendor_ruby/sass/script.rb:27:in `parse'
	from /usr/lib/ruby/vendor_ruby/sass/engine.rb:1177:in `parse_script'
	from /usr/lib/ruby/vendor_ruby/sass/engine.rb:783:in `parse_variable'
	from /usr/lib/ruby/vendor_ruby/sass/engine.rb:651:in `parse_line'
	from /usr/lib/ruby/vendor_ruby/sass/engine.rb:540:in `build_tree'
	from /usr/lib/ruby/vendor_ruby/sass/engine.rb:559:in `block in append_children'
	from /usr/lib/ruby/vendor_ruby/sass/engine.rb:558:in `each'
	from /usr/lib/ruby/vendor_ruby/sass/engine.rb:558:in `append_children'
	from /usr/lib/ruby/vendor_ruby/sass/engine.rb:417:in `_to_tree'
	from /usr/lib/ruby/vendor_ruby/sass/engine.rb:290:in `render'
	from /usr/lib/ruby/vendor_ruby/sass/exec/sass_scss.rb:396:in `run'
	from /usr/lib/ruby/vendor_ruby/sass/exec/sass_scss.rb:63:in `process_result'
	from /usr/lib/ruby/vendor_ruby/sass/exec/base.rb:52:in `parse'
	from /usr/lib/ruby/vendor_ruby/sass/exec/base.rb:19:in `parse!'
	from /usr/bin/sass:8:in `<main>'

Is there a way, or plans to support, simply copying assets without any processing such as images, fonts etc. This use case allows to fetch to the static folder assets from another folder.

When using the closure_js filter, the resulting JS just throws this error: ReferenceError: $jscomp is not defined

As this appear to be the only JS filter that is still maintained, any help getting this working would be great.

A bug report upstream suggests that using SIMPLE_OPTIMIZATIONS instead of WHITESPACE_ONLY may fix the issue. However, this either doesn't work or I have failed to change the setting (although I've tried several different methods).