Comments (20)
Since support for this would probably have to be added to webassets core anyway, using PyFilesystem might be an option.
from webassets.
+1
If you are not already working on this I could potentially give it a go. Let me know if you are interested.
from webassets.
Hey Niels, missed your comment. I'm not currently working on it.
This isn't that hard a thing to implement in general, but the devil is in the details. Some thoughts.
-
Should any support for Django storages be based on PyFilesystem integration in webassets core? (i.e. wrapping Django storages in PyFilesystem storages?) I would say no. It would add additional complexity, make PyFilesystem a dependency, and also be more work, thus holding the whole thing up.
-
Obviously, all filesystem interactions of webassets need to be wrapped.
-
There is a very new
Resolver
class used to "find" bundle contents, which will be a lot of help here. For example, it could get a new open() method that returns a file-like object, and then django-assets could overwrite this method to open files from a storage. -
Besides reading and writing files, in a number of places webassets looks at timestamps. This might be the hardest design challenge, because a) not all storages may support getting the last modified time, and b) in some cases this may be too slow, thus defeating the purpose. For example, with auto_build the timestamp of the output file is checked on every request, but if the output file is on S3, you probably wouldn't want that.
Of course, for output files an obvious solution is caching the last known modified timestamp. For input files, the solution might be to simply not check the timestamp and thus not recognizing changes for files that are on remote storages like S3.
from webassets.
Thanks for getting back to me. I'm not using webassets for my project anymore, so I'm not motivated to work on it currently. Sorry.
from webassets.
Having remote options would be great for deploying Python applications to hosts like Heroku, that allow applications to write to /tmp but don't guarantee that they will still be available on the next request. Being able to have bundle outputs stored on S3 to be accessed through Cloudfront or some other CDN that would be awesome.
from webassets.
One other alternative for consideration. After browsing through the code I think this would be a fairly quick implementation.
- Enable the Django staticfiles resolver even if
DEBUG
isTrue
. - Add an
ASSETS_OUTPUT_PATH
to the settings so that when assets are built, they all go to this directory. - Add a staticfiles finder which searches
ASSETS_OUTPUT_PATH
so these files are picked up whencollectstatic
is run.
Under this setup, you would just configure your storage backend for staticfiles, and collectstatic
would pick up all the built assets. I'm sure there's some caveats I'm not seeing, but these seems much simpler than making the library storage-aware.
from webassets.
Note that bad an idea, Michael. It's not exactly "nice" having to use yet another directory for the webassets output, but it would be easy.
It might even be already possible to configure it along those lines: ASSETS_ROOT should work for a custom output directory (staticfiles would still be used for source files).
Do people push files to S3 even during development?
from webassets.
Thinking about it a bit more, I think we could just use STATIC_ROOT
with a subfolder since that folder shouldn't be used for external storages. You might be right about things already working, the only piece that's missing then would be to enable the finder when DEBUG
is disabled.
I personally don't push to S3 during development, but I do in our staging environment.
from webassets.
I just got back to this and I found a solution that's working great for me. I have a dummy app webassets
that just contains __init__.py
and a .gitignore
to ignore .webassets-cache
and static
. Then I added this to INSTALLED_APPS
and set ASSETS_ROOT
to webassets/static
.
First running assets build
will build the assets and place them in webassets/static
. Then when I run collectstatic
, it will pick up those files along with all others. This simply requires use_staticfiles
in DjangoResolver
to return True
. Perhaps this could be a setting?
For me just having a setting to enable this and some documentation similar to what I described above completely solves the issue for me.
from webassets.
Just an update that I realized using webassets
for the dummy app name has the potential to cause naming conflicts in certain scenarios. Really any name can be used here. I decided to just prepend the project name.
from webassets.
I'm not currently developing under Django, but I'd really have webassets modified to allow using PyFilesystem to access storage. I suppose the main modification would to abstract filesystem access routine by replacing os.*, FileHunk and open() built-ins with an extensible interface. I'm willing to help. Would you consider evaluating a patch? Any advice?
from webassets.
I'm also having a problem with this. Django-assets keeps looking for assets in STATIC_ROOT
but on the local filesystem instead of the actual filesytem (S3 in my case). Everything else seems to work.
Wouldn't it suffice then to change the AssetsFileStorage
base class from FileSystemStorage
to DefaultStorage
?
from webassets.
IMO, you still want django-assets
looking on the local filesystem. No need to download from S3 to generate static assets when you should have the files sitting right there. I'm just using the AppDirectoriesFinder
and not the AssetsFinder
at all. I haven't experienced any issues thus far.
from webassets.
I prefer to not have any static files on my heroku slugs, but I could live with a small assets folder for this. I understand this might not be ideal for everyone. Making the storage backend configurable via ASSETS_STORAGE
for instance would work.
If only that would solve my problem.
Even though I have ASSETS_AUTO_BUILD = False
, I still get an bundle error stating that the compiled files can't be found . The path shown is the ASSETS_ROOT
. I'm guessing this is to check the md5 checksum of the compiled files. WIth heroku, I can't have a compiled file in the /static/ directory since the webservers keep cloning themselves from git. So I really need it to be checked on my storage backend.
from webassets.
I haven't gotten a solution I'm happy with for Heroku yet either. Are you using the %(version)s
placeholder in your output filenames? If you're not, then you should be fine. You could also just commit your compiled static files to the repository, but that's obviously not ideal.
from webassets.
@Koed00 A traceback would be illuminating. ASSETS_AUTO_BUILD should indeed disable any file system access. Obviously that means that it needs to be able to construct the url to the files with checking any files, but that is only a problem if you want the url to contain a version. The error you should be seeing in such a case is BundleError("Cannot find version of ...")
from webassets.
I gisted a stacktrace https://gist.github.com/Koed00/6842666
I gather the following happens; Webassets will look for the version number in the .webassets-cache
folder This folder is not found on the local filesystem on Heroku since it lives in the STATIC_ROOT
on S3 , so webassets looks for the bundles output file to determine the hash or file date,which is of course not found either.
I'll try ASSETS_VERSIONS = False
on the next deploy
from webassets.
That would make it worse :) You need to either
-
disable ASSETS_URL_EXPIRE and make sure that no
%(version)s
placeholder is used in output files, or -
Upload a manifest file that contains the version (
ASSETS_VERSION='manifest'
). I've tried to write up the issue here.
It would probably be useful though if webassets could get the version from the files on S3 by itself initially (and then maybe store them somewhere for speed), so this ticket does relate.
from webassets.
@miracle2k works now without the versioning. Manifest file didn't work, cause it looks for it on the local filesystem. This would mean I'd have to commit the manifest to git and manually upload the compiled bundles. This is far from ideal with the dozen bundles I've already got. I'll do without versioning for now.
A solution I've seen in other django packages is to allow the user to select a different storage handler for the django-assets part via a setting. This would probably make things work, as long as Auto Build is off , so there's no webassets code used.
I'm willing to give it a go, just be aware that I've only been using Python and Django for about 3 months ;)
from webassets.
Is this still in the pipeline? And does it extend to non-django projects? I'm using webassets standalone and can use this!
from webassets.
Related Issues (20)
- jsmin/rjsmin removes blanks in ES6 Template Strings HOT 1
- Fail when packing an external resource HOT 4
- YAMLLoadWarning yaml.load is deprecated
- Bundles cannot be replaced/re-added
- --line-comments is deprecated
- LibSASS - Unable to use other compile options like source_comments
- Version 2: Couldn't find preset HOT 3
- Threading.isAlive was removed in Python 3.9
- Load_builtin_filters doesn't work as expected with PyInstaller
- Deprecation warning due to invalid escape sequences in Python 3.7
- Fix simple typo: ultimatily -> ultimately
- Closure: ReferenceError: $jscomp is not defined
- Copy assets
- SCSS filter runs command sass without --scss HOT 2
- Add a tool to remove unused css
- How to configure filters using the YAML config option? Specifically cssrewrite. HOT 1
- Jinja2 Base template loaded twice when upgrading werkzeug package
- Add a way to get all bundle names
- 'out' is deprecated and will stop functioning in TypeScript 5.5
- mode merge=False not implemented in flask_assets
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from webassets.