Git Product home page Git Product logo

cached-module-loader's Introduction

cached-module-loader

Caching bundler and loader of Node.js modules which uses the cachedData feature in the vm module (available since Node.js 6).

Don't use this.

Installation

$ npm install --save cached-module-loader

Usage

Generate a cached bundle:

const { join } = require('path')
const { writeFileSync } = require('fs')
const cachedModuleLoader = require('cached-module-loader')

cachedModuleLoader.bundleDependencies('babel-core').then(bundle => {
  writeFileSync(join(__dirname, '.cached-data.bin'), cachedData)
  writeFileSync(join(__dirname, '.cached-code.js'), code)
})

Read a bundle and load the module:

const { join } = require('path')
const { readFileSync } = require('fs')
const cachedModuleLoader = require('cached-module-loader')

const cachedData = readFileSync(join(__dirname, '.cached-data.bin'))
const code = readFileSync(join(__dirname, '.cached-code.js'))

const babel = cachedModuleLoader.loadInThisContext(require.resolve('babel-core'), module, {
  moduleId: 'babel-core',
  cachedData,
  code
})

How it works

First cached-module-loader records every .js dependency of the requested module. These dependencies (and the module itself) are concatenated into one bundle. require() calls are rewritten to use absolute paths. The dependencies are wrapped in Node.js' standard module wrapper, and when loading modules, a modified require() function is passed. This function can load modules from the bundle rather than from disk.

A new vm.Script is created with the bundle as the script source, and the produceCachedData option enabled. This generates cachedData, which should be saved on the file system. Using both the source and cachedData, new vm.Script instances can be quickly created, making it much faster to load cached modules from the bundle.

Performance tests

$ node perf-tests/create-cache-data.js

$ time node perf-tests/require-with-cached-data.js
$ time node perf-tests/raw-require.js

cached-module-loader's People

Contributors

jamestalmage avatar novemberborn avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

Forkers

jamestalmage

cached-module-loader's Issues

Simplify API

I think the API for that module could be simple and export only one factory function, which returns a modified require():

const cachedRequire = require('cached-module-loader'){
  cachePath: join(__dirname, '.module-cache'),
  ignore: [/regex/, /to/, /ignore/]
});

const babel = cachedRequire('babel-core');
const got = cachedRequire('got');

cachedRequire()'s source would be something along:

function cachedRequire(moduleId) {
  if (!isCacheExists(moduleId)) {
    bundleDependencies(moduleId);
  }

  return loadInThisContext(moduleId);
}

What do you think?

Caching strategies

Module caches may include lots of files. These are dependent on file system location, Node.js version, and indeed package version. This package should explain appropriate caching strategies.

I reckon that the cache should be discarded when the Node.js version changes. Perhaps we should generate a cache key that takes the version into account, allowing caches for different versions to live side-by-side.

The cache contains hardcoded file paths, so we need an easy way to verify the presumed location of the sources.

Integration with package-hash should be easy.

Perhaps we can resolve the npm modules the cache depends on. Then the respective package.json files could be hashed and verified before using the cache.

Needs tests

  • From the perspective of the module, it shouldn't really be able to tell whether it was loaded from a cache or through Node.js module system. This means that all properties on the module object must be the same (aside from module.parent).
  • Circular dependencies must work as expected
  • Depending on built-in modules must work as expected
  • Dynamic requires are not cached, but must still resolve
  • โ€ฆ?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.