Create a "Version" abstraction about webassets HOT 15 CLOSED

This is good, and I think this is the right direction.

I realize now that the GitVersion probably isn't necessary, and that HashVersion is sufficient. The main difference is that, when a new version of the application is released, only the assets that have changed produce new bundles. If one uses the current Git revision, all assets are rebuilt and clients must download everything, even if the source files haven't changed.

from webassets.

Related discussion: http://bjk5.com/post/4918954974/js-css-packaging-to-minimize-requests-and-randomly-evil

from webassets.

One more related discussion: https://groups.google.com/forum/#!topic/django-developers/jlnw-TUwtuk

from webassets.

This was commited:

• https://docs.djangoproject.com/en/dev//ref/contrib/staticfiles/#cachedstaticfilesstorage
• https://bitbucket.org/django/django/changeset/830907436cdd

from webassets.

Interesting, though it doesn't seem to be something that webassets can build upon, since it would be run after "collectstatic".

from webassets.

Maybe webassets will be able to load files from the CachedStaticFilesStorage, combine the files and then save them to the same storage so that files become versioned. There is post_process method (https://docs.djangoproject.com/en/dev//ref/contrib/staticfiles/#django.contrib.staticfiles.storage.StaticFilesStorage.post_process) that can be called manually.

from webassets.

Adding specifically to django-assets to ability to use the Django storage backends in general would be a very cool feature. But is there a reason why would want to write to CachedStaticFilesStorage specifically, if webassets itself could create versioned filenames?

Though I suppose, if Django storage backends are supported, then you'd be free to chose which of the two should version the final files. I have created a ticket for this: #62.

from webassets.

I'd love to see this soon, or at least a content-hash-based filename mechanism. Our asset-generation takes a looooong time :)

from webassets.

I do plan on having this done rather sooner than later, though I'm not sure why you would expect it speed up the build (if that is what you are suggesting).

Thinking about this though... I wonder if parallelisation would yield speed improvements. It probably should, and at least for the command line build, should be easy to implement.

from webassets.

I haven't fully grokked the level at which the versioner works, but it seems great to me. I'm running on Heroku, so my assets aren't anywhere near the app. Right now I'm using a custom Bundle that overrides the urls() method in production to provide a custom cache buster based on my app version without trying to look at the asset files (which don't exist). It's a bit ugly and temporary, but it seems to work for now.

The Bundle class doesn't seem like the appropriate place to deal with this. As I'm understanding things so far, a custom versioner would cover this nicely.

from webassets.

Yes, correct. The default versioner would be able to write a manifest file that needs to ship with your code and contains the versions.

from webassets.

Let me explain why a hash-based version abstraction would benefit our system. (Kinda lengthy, sorry.)

We currently define 283 webasset bundles across 19 different Django modules. The bundles use the SASS, jsmin, cssmin and cssrewrite filters. The bundles are generated in serial and the process takes about 2m30s on our reasonably-hefty VPS takes. We do a rolling restart of the Django app across a handful of frontend servers, but the assets (the whole /media/ directory, actually) are served from memory in our load balancers. Thus, we currently need a way to place the new asset bundles on the media servers while the frontend servers are restarted so that application servers can point to either the new or old bundles.

How is this done? It's a hack, and I'm not proud of it because of a moronic mistake I made, but it works. In our settings.py:

buildrev = os.popen('git rev-parse HEAD').read().strip()[:10]
...
# webassets
ASSETS_DEBUG = False
ASSETS_EXPIRE = 'querystring'
ASSETS_OUTPUT_PREFIX = join('packed', buildrev) + '.' # Not an official setting.               
SASS_BIN = '/opt/bin/sass'                                                                     

We have a few shortcuts for defining bundles, js() and css(). Here's the former:

def js(name, *args, **kwargs):
    """Convenience helper for creating JavaScript bundles."""
    kwargs.setdefault('filters', 'jsmin')                                                      
    kwargs.setdefault('output', settings.ASSETS_OUTPUT_PREFIX + name + '.js')                  
    return register(name + '_js', Bundle(*args, **kwargs))                                     

After the push processes' preliminary build step, our media/packed/ directory looks like:

9f05409944.base.css
...
9f05409944.team.css
2dfc5b1998.base.css
...
2dfc5b1998.team.css

...where ... represents 281 other bundles. This allows us to rsync the entire directory to the media servers.

There are problems, and this is my Moronic Mistake:

1. Every bundle needs to generated every push, even if the content hasn't changed. (Slows down the push process.)
2. Tying the output file name to the Git revision forces users to re-download assets after a push. (Slows down the user experience.)

A proper version abstraction, with the ability to use a content hash as a version, could use the checksum of the combined source files as part of the output path. I think. Maybe.

Another thought: Actually, after writing all of this, I realize I could do the hashing myself and craft a filename using the checksum of all of the source files instead of ASSETS_OUTPUT_PREFIX. That will make the devserver start a bit slower, and I'm not sure if webassets regenerates output bundles if the destination exists. I'll give it a try.

from webassets.

Hash-based versioning works best because you don't have to worry about mtime being different across servers/OSs. It Just Works™.

from webassets.

This is pretty much done now. If people want to test it, they shouldn't run into many issues (automatic rebuilding having trouble while using versions='hash' will be fixed soon - a manual build helps)

Usage would look something like this:

 env.versions = 'hash:10'                    # length of hash
 env.manifest = 'file:manifest.pickle'       # non-default filename
 env.url_expire = False                      # version is in filename already
 bundle.output = '%(version)s.team.css'

The manifest file needs to be shipped with the app.

I'll have to put together some proper documentation, then I'll merge it into master.

from webassets.

Merged. The docs are here:

http://readthedocs.org/docs/webassets/en/latest/expiring.html

from webassets.