google / module-server Goto Github PK

module-server

License: Apache License 2.0

JavaScript 97.27% HTML 2.73%

module-server's Introduction

Module Server

Module server is a system for efficient serving of CommonJS modules to web browsers. The core feature is that it supports incremental loading of modules and their dependencies with exactly 1 HTTP request per incremental load.

This is a reference implementation that has not been battle-tested in production use. See our presentation from JSConf EU 2012 for more details.

The serving system implements the following constraints:

Requesting a module initiates exactly 1 HTTP request
This single requests contains the requested module and all its dependencies.
Incremental loading (every module request after the first one) of additional modules only downloads dependencies that have not been requested already.
The client does not need to download a dependency tree to decide which additional dependencies to download.

For many web applications serving all JavaScript in a single compiled binary may be a good enough, simple solution, more complex apps with large JS code bases will profit from only downloading code when it is needed. While AMD loaders such as require.js implement incremental loading as well, they often do so through recursive downloading of dependencies which may significantly degrade latency.

Closure compiler supports both compilation of CommonJS and AMD modules. It should thus be possible to use this system as a production frontend for projects that use other systems such as require.js or browserify today.

Source Maps

By default all JS responses support source maps for optimal debugging with Chrome Dev Tools (Don't forget to activate source map support in the Dev Tools settings). We recommend to deactivate this for production use, if you only want to provide clients access to obfuscated JS

Setup

See demo-server.js for an example server. You may want to adapt this to your individual serving stack (such as as for use within express). We recommend doing actual serving through a caching reverse proxy CDN network for minimal latency.

Demo

Run node demo-server.js then drag clients/test/demo.html to a browser window.

Client

clients/module-client.js provides the in-browser module loader. It depends on $LAB.js but should be easily adaptable to most JS loaders.

This will get you started:

<script src="../third-party/LABjs/LAB.src.js"></script>
<script src="../module-client.js"></script>

<script>
window.loadModule = ModuleServer('http://127.0.0.1:1337/');
</script>

Whenever you want to do an incremental load of a module, replace require('foo') with loadModule('foo', function(foo) { … }) and you are all set.

Compiler

module-compiler/bin.js is a wrapper around closure compiler for compiling JS for serving with Module Server. Run with --help for usage. It supports watching a directory tree for automatic compilation when you change your sources and it ships with closure compiler for easy installation.

Make sure you have the java binary in your path :)

Example:

node module-compiler/bin.js  --module_path=./test/fixtures/sample-module --entry_module=app --output_path=../build/

See the INSTALL instructions.

Compilation

Create a file called app.js (or whatever you like) and require all of your top-level modules in it (the ones you actually want to request from your application). This will ensure that everything gets compiled in one go.

Fine print

Pull requests are very much appreciated. Please sign the Google Code contributor license agreement (There is a convenient online form) before submitting.

Author: Malte Ubl (Google Inc.)
Copyright: Copyright © 2012 Google, Inc.
License: Apache 2.0

module-server's People

Contributors

Stargazers

Watchers

module-server's Issues

Update the GitHub repo description

More that just "module-server"

Project status

Hello there.

First of all, this project seems to be amazing idea, and it looks like that it is a 'tiny' missing bit in modules management process. I have started playing around with it, it's really really awesome. I've managed to make it works with few component.io components, now i'll try to run it with browserify modules and perhaps some of r.js dedicated ones, i can see right now that there is a little bit more work on this project, to make it really usable but my question is: Is this project abandoned already?

Best regards,
Chris

npm

It is ultra awesome idea.
Is someone working on regular npm module?

Security Policy violation Binary Artifacts

This issue was automatically created by Allstar.

Security Policy Violation
Project is out of compliance with Binary Artifacts policy: binaries present in source code

Rule Description
Binary Artifacts are an increased security risk in your repository. Binary artifacts cannot be reviewed, allowing the introduction of possibly obsolete or maliciously subverted executables. For more information see the Security Scorecards Documentation for Binary Artifacts.

Remediation Steps
To remediate, remove the generated executable artifacts from the repository.

Artifacts Found

module-compiler/third-party/closure-compiler/compiler.jar

Additional Information
This policy is drawn from Security Scorecards, which is a tool that scores a project's adherence to security best practices. You may wish to run a Scorecards scan directly on this repository for more details.

Allstar has been installed on all Google managed GitHub orgs. Policies are gradually being rolled out and enforced by the GOSST and OSPO teams. Learn more at http://go/allstar

This issue will auto resolve when the policy is in compliance.

Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.

Feature consideration - predefined loadable subgraphs

Hi, I just watched the jsconfeu presentation. I was eager to see your approach so I could compare it to my own. I also have an optimizing module server of sorts. My team and I went down the same sort of road. First started loading files as needed. Then we concatenated everything upfront. Then we hit the 1MB download and knew we had to break it apart. I had actually already done the work of making a dependency graph in order to do the single script concatenation so that I could correctly order the files in the concat script.

This is where our approaches diverge. I had considered doing something similar to your approach. What I dislike about it was the combinatorial explosion. I had gotten used to deploying with a single static script file. There are obvious advantages. Deployment is simpler, the scripts themselves can go anywhere, and you can even use a CDN. Browser caching is potentially much better because order of interaction doesn't change the scripts loaded.

In our approach, instead of allowing for the loading of arbitrary dependencies, certain dependencies were marked as roots of a new dependency subgraph. To get a big picture on the structure, in a very large app like ours, the dependency graph becomes a tree of subgraphs. The tree structure assumes that parents are always loaded before children. Dependencies can only belong to a single subgraph, but if a dependency is needed by multiple subgraphs, they are moved to the closest parent subgraph. As opposed to a module-server, we have what we call a "module compiler". It does many other things in addition to the dependency graph stuff, but that dependency graph is the heart of it. Anyway, at the very end of module compilation, the end result is a static js file for each subgraph, using a naming convention of the path to the module separated by underscores.

Navigating around the app results in dynamic script loads that might look something like:

main.js
main_child3.js
main_child2.js
main_child2_gchild1.js
main_child2_gchild3.js
main_child1.js
main_child1_gchild4.js

Where main, childX, gchildX are just example names for the root dependency of that subgraph. Notice that parents are always loaded before children, but siblings can be loaded in any order.

In our app, this would basically be main loaded initially in the html, and childX being based on first level nav, gchildX based on 2nd level nav. That works best for us, but obviously an app with more functionality per screen and fewer screens would use it differently.

I hope this was helpful. I know its a different direction than you are currently going, but seeing as I'm not able to contribute my own code to open source ATM, I was hoping to spread a little of my own experience/knowledge this way.