Suggestion: Switch to dom-based building about roots HOT 18 CLOSED

woah this is crazy but maybe outside the scope of this project

thoughts @Jenius

from roots.

Yeah wow same thoughts in general - this is crazy. Great work on this project first of all, clearly you guys have poured a ton of thought and effort into this. I think that rather than trying to squash the entire project into roots, perhaps it might be better to steal small parts of it and include them in the pipeline. What I especially liked and what roots does not yet have I'll go over below:

• The image optimization. still worried about the external dependencies, but I like how you guys did it and the tailored npm packages you have made to integrate optipng, pngcrush, jpegtran etc. This should definitely be something that happens on compile IF you have the binaries installed, but I definitely don't want it to raise a fuss if you don't because relying on entirely external dependencies that must be separately installed for me is a little whack.
• Your use of require.js - I think we can definitely do better with this. I've been thinking about the best way to roll JS dependencies together and I think roots' philosophy is currently a bit undecided. I have the asset pipeline in there thanks to snockets, so if you want you can define a single application.js (or .coffee) and require all your others there. But I also have require.js included by default so you can dictate which sections of your code depend on which libraries and make it work efficiently there. The whole coffeescript issue is a separate thing - with coffee's automatic closures, it makes sharing code between included separate files super difficult. Having a system where you could export and require like commonjs on the client side I think would be ideal.
• The options for putting your assets out to a CDN. For a static site, yeah, probably not crucial. But nice to have the option there anyway.
• Assetgraph itself. I love this: "if you change the url of an asset, all relations pointing at it are automatically updated" - brilliant. Getting that integrated would definitely be something I'm interested in, although how feasible it is without increasing the project complexity is a major consideration - keeping things clean and simple is a big concern for us.
• Inlining small images. This should definitely be done. I know stylus has support for this within the css - I should work on including that. Having it happen within <img> tags would be an added benefit.

Overall I feel like pulling in AssetGraph itself and working with that directly rather than trying to pull assetgraph-build might be a better approach.

It seems like we have pretty similar goals in general with these two projects though, and clearly you guys have done some great work. I would definitely welcome you jumping in and contributing to roots, if that's something you're interested in!

from roots.

Thanks for all the positive feedback. I'll see if I can address some of the points you raised.

• Image optimization. The ideal goal here is to be completely independent of external binaries of the operating system. We are depending on optipng-bin and jpegtran-bin, that try to put the binaries in npm, thus making the dependency chain better. Also it should just ignore a missing binary and skip the optimization if the tools isn't available. Optional was the word I was looking for :)
• require.js. Assetgraph doesn't depend on it. It just understands it, since it seems to be a pretty widespread dependency manager. And of course the authors use it, so it was the first thing to get in there. Other dependency managers might find their way into assetgraph as well. Bundling strategy is as of yet also an undetermined strategy for assetgraph-builder. For now it squashes everything together if the require is at the root level.
• Coffeescript. There is some support for it in there. I haven't used it as I am not a big fan. Assetgraph mainly concerns itself with postprocessing of pure browser readable inputs. But that usually fits pretty well into most workflows as long as you set up preprocessors yourselt. Or in this case, roots does that.
• Closures, scoping and exports. Sound like you want to take a look at substacks browserify and yarnify. Also a preprocessing step
• Adding assetgraph and complexity. Ideally assetgraph could at some point replace some of the existing build flows on roots, thus balancing out the complexity or even simplifying. However assetgraph is mainly developed towards static pages or web apps, so if roots has a lot of users that use on the fly generated html from a dynamic back end, then it's probably not the best fit.

I very much like your idea of using Assetgraph more low level. The whole point of it is to be a optimization tookit. And since you are building an optimization/workflow framework it fits quite well. The low level version has quite a steep learning curve and is very much geared towards tool builders, not end users, so that's why we're not trying to sell it as much :)

I have also tried getting assetgraph into Yeoman. They like grunt a lot and just want plugins for it that they can preconfigure instead of them writing their own thing. So I created the grunt-reduce wrapper for assetgraph-builder for them.

I'd like to help implement assetgraph, and I would of course also handle all support that may come in from end users if you point them at the assetgraph issues when they have problems with it. I don't think I'll be a regular contributer to roots just yet though. For now my primary goal is to evangelize assetgraph. So it's mostly out of self interest to get peopel to use our awesome idea. But of course that doesn't have to be all bad :)

from roots.

So I keep coming back to look at this and thinking about it throughout the refactor process, and I think it would be incredibly awesome to build some pieces of this idea into the project.

One instance where it recently came up was with selectively compiling on save. Right now, roots compiles the entire project when you run roots watch, pops up the site in your browser, then watches the files and compiles individually changed files to save time. But the problem arises when a dependency that is not specifically compiled (such as a layout file or partial) is changed. Since this file isn't compiled, passing the filename through roots throws an error. And since I don't have an "AST" that knows what depends on what, I can't selectively compile only the files that change as a result of the original file changing. My temporary solution is that when an ignored file is compiled, the whole project gets recompiled, but obviously this isn't ideal.

I feel like having an assetgraph would help with situations like this and a bunch of others. The only question I have here for you @Munter is whether assetgraph only works with vanilla html, css, and javascript. I feel like the real benefits would be to have assetgraph chew through the jade, stylus, and coffeescript files in order to figure out the relations there, before they are compiled. If not, would it be a monumental task to build adapters for other languages?

from roots.

We've had thoughts along the same lines. @papandreou created assetgraph-middleware with exactly this purpose.

Assetgraph already has stylus and coffeescript support, mostly because those where pretty easy to integrate since they are node modules.

There is no doubt that the main purpose of assetgraph has primarily been post processing, not preprocessing. One you get into the realm of preprocessing you have to configure dependencies and workflows manually with manifests, configuration etc. We're not really big fans of expanding assetgraph in that direction.

That said though, assetgraph integrates well with other preprocessors, as long as the output is supported and saved to disk in the location your web application is including it from. This seems to be the typical use case.

from roots.

This is excellent - just a couple questions:

• Does it have jade support? This is fairly important : )
• Like you said, this is not set up really well for pre-processing. While this module would be easy to build in to the roots watch server, the end goal is for roots to spit out nice, compressed, and organized html, css, and js to a public folder - your app is then run as a static site from this folder. Would it be extremely difficult to pull this module out so that it could be used in node manually rather than only as middleware? If not, I'll start tinkering with it!

from roots.

There is no jade support. I guess there might be some day. We haven't had any requests yet.

I don't think there is an easy way to do dependency based continuous updates like you describe. But if I understand you correctly you want to do an actual production build with full optimizations based on file updates on disk? If so I disagree with the idea. A properly minified web application doesn't have the same assets in the output as in the input, so you can map 1:1 and just do a partial build on the assets that changed.

A live updated directory like that should only serve for development, and then it should not be minified at all so development is easier.

from roots.

I'm going to close this just because I don't think it's viable at the moment. It would require a major rewrite of the core of roots, which I'm not prepared to do at the moment, and it would require quite a bit of external support to get everything wired up correctly.

I do like this idea and would very much like to have it integrated at some point in the future. I'll keep it in the back of my mind, but I don't want to have this stagnant issue sitting around as open just because I want to get the issues section focused down to things we're working on right now.

Thanks so much @Munter for all your answers and support here, and don't doubt that someday we will be back here probably asking more questions 😄

from roots.

Ask away :)

Thanks for considering this. I completely understand your decision.

You also might be interested in following tkellens node-task project. It's an attempt at formalizing some of the concepts that experience has shown to be needed in grunt. I feel that roots, assetgraph, grunt and node-task are all basicly working on the same problems from different angles, so there might be potential for collaboration, or at least some good architectural discussions.

from roots.

Definitely agreed, and I'll make sure to check out node-task. Thanks again @Munter!

from roots.

I'm going to reopen this because I am doing a major rewrite of the roots core and like @Jenius said, assetgraph looks very useful. I think at the bare minimum, we can use it for:

• processing the compiled HTML files to determine their dependencies (to optimize compiling)
• or, using it to rewrite dependencies in such a way that the paths to .coffee and .stylus files could be used directly in Jade, meaning that the user wouldn't need to even think about the location of compiled assets in the ./public folder.

Also, if assetgraph is capable (or nearly capable) of optimizing multipage sites, it could be used for general post-compile optimizations like image optimizations & inlining, css/js compression and in-lining, HTML compression... ect. Perhaps we could even use it to pretty-print HTML and other stuff if it provides a easy to work with representation of the HTML page and its assets. Then we can disable those in-language features like image-inlining that stylus has, or the compression that coffeescript offers, and just use asset-graph for all of that. We could even fork the compilers of some of the more popular languages and gut those features out.

I am not interested in having asset-graph itself do any sort of compilation because I think that is way outside of the scope of that project. Instead, I think that roots should provide a more usable interface for assetgraph, and a way to manage all the compilation, that comes before assetgraph.

from roots.

processing the compiled HTML files to determine their dependencies (to optimize compiling)

As long as the html is valid this will work out of the box.

using it to rewrite dependencies in such a way that the paths to .coffee and .stylus files could be used directly in Jade, meaning that the user wouldn't need to even think about the location of compiled assets in the ./public folder.

This one might be a bit more tricky. The way Assetgraph works best is by having a programmatic api, like the DOM or an AST, to match patterns that represent relations. I'm not sure how much access you have to that with Jade. Of course you could just do like we do with Less assets, don't try and model relations, but just compile it and then use the output as an asset to model relations on. We haven't focused that much on the preprocessing stage yet, since there is very little work done in defining standards for setting up relations of things that need to be preprocessed. Maybe @papandreou has some input here.

Also, if assetgraph is capable (or nearly capable) of optimizing multipage sites, it could be used for general post-compile optimizations like image optimizations & inlining, css/js compression and in-lining, HTML compression

Multi page sites are supported by Assetgraph. In the command line tool you can add paths to multiple html-pages or use a minimatch pattern to glob all html-files. The default setup is to bundle static assets per page, meaning you might have to redownload bundles if there are minor differences across pages. An alternative bundling strategy is available, which tries to bundle based on identical assets across pages. However there is the caveat that the assets need to be included in the same order on the pages. That caveat is of course mitigated if these pages are all using the same basic template to include css and javascript assets.

Of course you wouldn't be using the command line tool if this goes into tighter integration with the core. But the principle is simple. Assetgraph also gives special meaning to files that are a part of this initial collection f files that the graph starts out with. They keep their name intact after static assets have been minified, bundles and renamed for far future cache expiry. So on multi page sites you really want all your public facing html-pages to be part of the graph seeding.

from roots.

thanks for the explanation... and wow... just wow. I finished reading your presentation ... it's just so beautiful - exactly the kind of abstraction that roots needs

from roots.

Well, I've been looking at assetgraph for a bit, though the documentation in the readme appears to be outdated and the documentation everywhere else is very sparse.

Any chance you could work on fixing that up?

from roots.

Ok, the more I look at this API, the more I think I should just rewrite it. It seems like AssetGraph is meant for a sequence of optimization-related tasks to be performed. Whereas roots needs an AssetGraph that can be passed around and hold everything... also there are a lot of features that we don't need.

So: the plan going forward -
I'm just going to start rewriting it, and if it turns out that I just totally misunderstood the AG API, then I'll happily ditch that project and use the main AG repo. It is not without some remorse that I make this decision, because I really hate it when I can't use an existing solution, or at least just build on top of it. However, I really don't think that it's feasible to reorganize the existing API. At least I will be able to use a lot of the code from the original repo... just a lot of the underlying structure will need to be changed

from roots.

Yeah, the API documentation has not been prioritized yet. Primarily because there were some rewrites coming up that would change it. I'm pretty sure those changes are made now though and that the API is stable. There is a beginning documentation effort here: http://gofish.dk/assetgraph/api/

Assetgraph was originally thought up as a build system, but we ended up finding an abstraction that has the model where assetgraph itself is just a platform that could potentially be used as a build system. The build system itself is in assetgraph-builder.

However an assetgraph instance can still be passed around, manipulated on the fly, queried on demand etc. So from what you've said so far I would think it covers your use case. It's true that there are some optimization things in there as well, but they're not really adding any weight or extra requirements, so I don't see your point of it being in the way.

I'm pretty sure @papandreou would be very happy to hear your thoughts about what is missing or needs to be changed for assetgraph to become a candidate for inclusion into roots

from roots.

Yeah, I've been reading that one for a while but it's far from complete. It is starting to make more sense, and I'm leaning away from writing my own as I learn more. And yeah, if @papandreou is willing to listen to my complaints then that'd be great!

The issue I was having before is if I initialize an instance of asset graph and run something it doesn't run on its own... I think I'm supposed to be calling .run(), but here's what I'd expect to be able to do:

assetGraph = new AssetGraph();
assetGraph.loadAssets('*.html');
assetGraph.findAssets().length !== 0; // but this fails because it's empty :(

Now I think I know how I'm really supposed to do it, but calling run() every time is still kinda awkward.

from roots.

Sorry for continuing to open and close this, but since this issue has been stagnant for months and I'm trying to really focus down on the next couple releases, I'm going to close this again. As soon as @slang800 is back to working on this actively, we can re-open : )

from roots.