Git Product home page Git Product logo

hard-source-webpack-plugin's Introduction

HardSourceWebpackPlugin

Build Status Build status

HardSourceWebpackPlugin is a plugin for webpack to provide an intermediate caching step for modules. In order to see results, you'll need to run webpack twice with this plugin: the first build will take the normal amount of time. The second build will be significantly faster.

Install with npm install --save-dev hard-source-webpack-plugin or yarn add --dev hard-source-webpack-plugin. And include the plugin in your webpack's plugins configuration.

// webpack.config.js
var HardSourceWebpackPlugin = require('hard-source-webpack-plugin');

module.exports = {
  context: // ...
  entry: // ...
  output: // ...
  plugins: [
    new HardSourceWebpackPlugin()
  ]
}

You can optionally set where HardSource writes and reads its cache to and from, and the hash values that determine when it creates new caches.

new HardSourceWebpackPlugin({
  // Either an absolute path or relative to webpack's options.context.
  cacheDirectory: 'node_modules/.cache/hard-source/[confighash]',
  // Either a string of object hash function given a webpack config.
  configHash: function(webpackConfig) {
    // node-object-hash on npm can be used to build this.
    return require('node-object-hash')({sort: false}).hash(webpackConfig);
  },
  // Either false, a string, an object, or a project hashing function.
  environmentHash: {
    root: process.cwd(),
    directories: [],
    files: ['package-lock.json', 'yarn.lock'],
  },
  // An object.
  info: {
    // 'none' or 'test'.
    mode: 'none',
    // 'debug', 'log', 'info', 'warn', or 'error'.
    level: 'debug',
  },
  // Clean up large, old caches automatically.
  cachePrune: {
    // Caches younger than `maxAge` are not considered for deletion. They must
    // be at least this (default: 2 days) old in milliseconds.
    maxAge: 2 * 24 * 60 * 60 * 1000,
    // All caches together must be larger than `sizeThreshold` before any
    // caches will be deleted. Together they must be at least this
    // (default: 50 MB) big in bytes.
    sizeThreshold: 50 * 1024 * 1024
  },
}),

Some further configuration is possible through provided plugins.

  plugins: [
    new HardSourceWebpackPlugin(),

ExcludeModulePlugin

    // You can optionally exclude items that may not be working with HardSource
    // or items with custom loaders while you are actively developing the
    // loader.
    new HardSourceWebpackPlugin.ExcludeModulePlugin([
      {
        // HardSource works with mini-css-extract-plugin but due to how
        // mini-css emits assets, assets are not emitted on repeated builds with
        // mini-css and hard-source together. Ignoring the mini-css loader
        // modules, but not the other css loader modules, excludes the modules
        // that mini-css needs rebuilt to output assets every time.
        test: /mini-css-extract-plugin[\\/]dist[\\/]loader/,
      },
      {
        test: /my-loader/,
        include: path.join(__dirname, 'vendor'),
      },
    ]),

ParallelModulePlugin

    // HardSource includes an experimental plugin for parallelizing webpack
    // across multiple processes. It requires that the extra processes have the
    // same configuration. `mode` must be set in the config. Making standard
    // use with webpack-dev-server or webpack-serve is difficult. Using it with
    // webpack-dev-server or webpack-serve means disabling any automatic
    // configuration and configuring hot module replacement support manually.
    new HardSourceWebpackPlugin.ParallelModulePlugin({
      // How to launch the extra processes. Default:
      fork: (fork, compiler, webpackBin) => fork(
        webpackBin(),
        ['--config', __filename], {
          silent: true,
        }
      ),
      // Number of workers to spawn. Default:
      numWorkers: () => require('os').cpus().length,
      // Number of modules built before launching parallel building. Default:
      minModules: 10,
    }),
  ]

Options

cacheDirectory

The cacheDirectory is where the cache is written to. The default stores the cache in a directory under node_modules so if node_modules is cleared so is the cache.

The cacheDirectory has a field in it [confighash] that is replaced by the configHash option when webpack is started. The [confighash] field is here to help with changes to the configuration by the developer or by a script. For example if the same webpack configuration is used for the webpack cli tool and then the webpack-dev-server cli tool, they will generate different configuration hashes. webpack-dev-server adds plugins for its reloading features, and the default hash function produces a different value with those plugins added.

configHash

configHash turns a webpack configuration when a webpack instance is started and is used by cacheDirectory to build different caches for different webpack configurations.

Configurations may change how modules are rendered and so change how they appear in the disk cache hard-source writes. It is important to use a different cache per webpack configuration or webpack cli tool. webpack and webpack-dev-server for example needed separate caches, configHash and [confighash] in the cacheDirectory will create separate caches due to the plugins and configuration changes webpack-dev-server makes.

The default value for configHash is

configHash: function(webpackConfig) {
  return require('node-object-hash')({sort: false}).hash(webpackConfig);
}

This uses the npm node-object-hash module with sort set to false to hash the object. node-object-hash hashes as much as it can but may have issue with some plugins or plugins and loaders that load an additional configuration file like a babel rc file or postcss config. In those cases you can depend on node-object-hash and extend what it hashes to best cover those changes.

configHash can also be set to a string or it can be a function that generates a value based on other parts of the environment.

configHash: function() {
  return process.env.NODE_ENV + '-' + process.env.BABEL_ENV;
}

environmentHash

When loaders, plugins, other build time scripts, or other dynamic dependencies change, hard-source needs to replace the cache to make sure the output is correct. The environmentHash is used to determine this. If the hash is different than a previous build, a fresh cache will be used.

The default object

environmentHash: {
  root: process.cwd(),
  directories: [],
  files: ['package-lock.json', 'yarn.lock']
}

hashes the lock files for npm and yarn. They will both be used if they both exist, or just one if only one exists. If neither file is found, the default will hash package.json and the package.json under node_modules.

You can disable the environmentHash by setting it to false. By doing this you will manually need to delete the cache when there is any dependency environment change.

info

Control the amount of messages from hard-source.

mode

Sets other defaults for info. Defaults to 'test' when NODE_ENV==='test'.

level

The level of log messages to report down to. Defaults to 'debug' when mode is 'none'. Defaults to 'warn' when mode is 'test'.

For example 'debug' reports all messages while 'warn' reports warn and error level messages.

cachePrune

hard-source caches are by default created when the webpack configuration changes. Each cache holds a copy of all the data to create a build so they can become quite large. Once a cache is considered "old enough" that it is unlikely to be reused hard-source will delete it to free up space automatically.

maxAge

Caches older than maxAge in milliseconds are considered for automatic deletion.

sizeThreshold

For caches to be deleted, all of them together must total more than this threshold.

Troubleshooting

Configuration changes are not being detected

hard-source needs a different cache for each different webpack configuration. The default configHash may not detect all of your options to plugins or other configuration files like .babelrc or postcss.config.js. In those cases a custom configHash is needed hashing the webpack config and those other values that it cannot normally reach.

Hot reloading is not working

webpack-dev-server needs a different cache than webpack or other webpack cli tools. Make sure your cacheDirectory and configHash options are hashing the changes webpack-dev-server makes to your webpack config. The default hard-source values should do this.

Multiple webpack processes at the same time are getting bad results

If you are using multiple webpack instances in separate processes make sure each has its own cache by changing cacheDirectory or configHash.

Rebuilds are slower than the first build during dev-server

This is can be due to module context dependencies. require.context or loaders that watch folders use this webpack feature so webpack rebuilds when files or folders are added or removed from these watched directories. Be careful about using require.context or context aware loaders on folders that contain a lot of items. Both require.context and context loaders depend on those folders recursively. hard-source hashes every file under a require.context or context loader folder to detect when the context has changed and dependent modules are rebuilt.

webpack-dev-server build loops continuously

Make sure you don't have a require.context or context loader on the root of your project. Such a context module means webpack is watching the hard source cache and when the cache is written after a build, webpack will start a new build for that module. This normally does not happen with webpack-dev-server because it writes the built files into memory instead of the disk. hard-source cannot do that since that would defeat its purpose as a disk caching plugin.

Please contribute!

If you encounter any issues or have an idea for hard-source-webpack-plugin could be better, please let us know.

hard-source-webpack-plugin's People

Contributors

abogaart avatar allenfang avatar cjweigle avatar dariusk avatar ettavolt avatar faceyspacey avatar filipesilva avatar greenkeeper[bot] avatar jlhwung avatar kadamwhite avatar koningskristof avatar kristenmills avatar muzea avatar mzgoddard avatar nikhilmat avatar nomcopter avatar piecyk avatar rowanoulton avatar sevinf avatar shawnbot avatar strate avatar swernerx avatar waldyrious avatar wtgtybhertgeghgtwtg avatar zry656565 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

hard-source-webpack-plugin's Issues

missing css file output from extract-text-webpack-plugin

I've tried adding this to a webpack config using ExtractTextPlugin. It runs to completion, but only output the JS module, and no CSS files.

Using:

"webpack": "^1.13.0",
"extract-text-webpack-plugin": "^1.0.1",

Input and config:

// index.js
require('./style.css')


// webpack.config.js
var webpackConfig = {
  entry: {
    app: ["./index.js"]
  },
  module: {
    loaders: [
      {
        test: /\.css$/,
        loader: ExtractTextPlugin.extract("style-loader", "css-loader")
      },
    ]
  },
  recordsPath: path.resolve(__dirname, 'records.json'),
  plugins: [
     new HardSourceWebpackPlugin({
       cacheDirectory: 'cache'
     }),
    new ExtractTextPlugin("styles.css")
  ],
  output: {
    path: path.resolve(__dirname, "build"),
    publicPath: "/assets/",
    filename: "bundle.js"
  }
};

Output shows that styles were removed, but there is only 1 file in output folder

// bundle.js
// ...some lines
/* 2 */
/***/ function(module, exports) {

        // removed by extract-text-webpack-plugin

/***/ }

Non-extracted styles all work fine. And on the plus side, build time went from 30+ seconds down to ~5!

README: Which options are optional?

Hi there. The README left me a little bit puzzled which of the plugin's options I actually have to set to get going. All? None?

A minimal usage example could help here, I think :)

Btw: I really like the idea of this plugin!

Cheerio

Does not seems to work with webpack dllplugin

I am trying to improve performance by using your awesome plugin and at the same time webpack dllplugin.
But from my tests, when dllplugin (and dllpluginreference) are used, it take the same time as if this hard-source plugin was not enabled!
Some pseudo bench

  • nothing: around 20s
  • hard source: around 5s
  • dllplugin: around 10s
  • hard source + dll plugin: around 10s too ! => looks like your plugin is not enabled in this case

Any idea on this?

Support glob patterns in `environmentPaths`

We have quite a few files that, when changed, should make the cache invalid. Right now I have to list all those files inside environmentPaths.files. If we could use glob patterns instead, we would be less vulnerable to stale caches when new files are added.

Here's our current config:

        files: [
          'package.json',
          'webpack.config.js',
          'webpack.config.edge.js',
          'webpack.config.production.js',
          'webpack.config.staging.js',
          'webpack.config.happo.js',
          'webpack.config.common.js',
        ],

With glob patterns, this would become

        files: [
          'package.json',
          'webpack.config.*',
        ],

this.cacheItem.warnings.map is undefined

Just noticed this with 0.2.2:

  this.warnings = this.cacheItem.warnings.map(deserializeError(ModuleWarning, state), this);
                                         ^

TypeError: Cannot read property 'map' of undefined
    at HardModule.build (/Users/alexose/repo/node_modules/hard-source-webpack-plugin/l
ib/hard-module.js:185:42)
    at Compilation.buildModule (/Users/alexose/repo/node_modules/webpack/lib/Compilati
on.js:126:9)
    at /Users/alexose/repo/node_modules/webpack/lib/Compilation.js:309:10
    at onDoneResolving (/Users/alexose/repo/node_modules/webpack/lib/NormalModuleFactory.js:36:13)
    at /Users/alexose/repo/node_modules/hard-source-webpack-plugin/index.js:741:22
    at fromCache (/Users/alexose/repo/node_modules/hard-source-webpack-plugin/index.js
:682:18)

Haven't dug into it yet, but it only happens when reading from an existing cache.

Does not work with Webpack-2.2.0-rc.4

There seems to be an issue with NullDependencyTemplate which was reworked by the Webpack people. The actual template class is now stored in require(webpack/lib/dependencies/NullDependency).Template - the previous file webpack/lib/dependencies/NullDependencyTemplate is not available anymore.

Webpack 2 and Hot module replacement

We're migrating to webpack 2 at Phenomic but HMR isn't working for us. Did you test with webpack 2 and HMR ?

Our branch is here https://github.com/MoOx/phenomic/tree/webpack-2

Our dev-server is an express instance with webpack-dev-middleware and webpack-hot-middleware

Behaviours:

  • Route /**.hot-update.json not found
  • Client doesn't fetch the new hot-update hash correctly.
    • If I start the dev server, make a change to trigger HMR. Via /__webpack_hmr I can see that the new hash for hot-update is A. Client fetches /A.hot-update.json but it's a route not found as I said above.
    • Next, I make another update and get hash B. Client still fetches A.hot-update.json
    • Restart dev server, make a changes, now the client fetches hash B.

Support first party plugins and loaders

Support all plugins and loaders under https://github.com/webpack. Support can be not caching but making sure builds are correct and complete or fail as expected.

  • * Resolution to this module is cached (in addition to modules depending on this being cached)
  • ** Modules depending on this are cached

Under https://github.com/webpack/webpack

  • ContextModule
  • DllPlugin
  • DllReferencePlugin
  • DelegatedModule *
  • ExternalPlugin
  • ExternalModule **
  • RawModule
  • CommonsChunkPlugin
  • DedupePlugin
  • UglifyJsPlugin
  • HotModuleReplacementPlugin
  • IgnorePlugin (Fix: #92, Related: #91, #72 (comment))
  • NamedModulesPlugin
  • HashedModuleIdsPlugin

Under https://github.com/webpack-contrib

  • extract-text-webpack-plugin *
  • mini-css-extract-plugin
  • style-loader
  • css-loader
  • mocha-loader
  • null-loader
  • imports-loader
  • json-loader
  • url-loader
  • file-loader
  • raw-loader
  • less-loader
  • exports-loader
  • html-loader
  • coffee-loader
  • expose-loader
  • json5-loader
  • transform-loader
  • coffee-redux-loader
  • bundle-loader
  • script-loader
  • jshint-loader
  • i18n-webpack-plugin
  • compression-webpack-plugin
  • coverjs-loader
  • multi-loader
  • i18n-loader
  • source-map-loader
  • worker-loader

Alternative environment hash option

First off, great module, this works flawlessly with my dev environment ๐Ÿ˜„

The issue I'm having is getting it to work on our CI server, as the mtimes of the node_modules directory and package.json etc will always change, hence this plugin won't work on a CI server. Calculating the md5 hash of every file in the node_modules directory isn't feasible for performance reasons, however now that yarn exists, you can just compute a hash of the yarn.lock file, plus package.json and the webpack config file contents, which should give you a performant and accurate environment hash (providing you use yarn to install your dependencies)

There are 2 possible options I propose to handle this:

  • add a contentsHash: true (there's probably a better name) option to the environmentPaths config that will calculate the md5 of each listed files contents, rather than using the mtime of each file.
  • add an alternative environmentHash option that accepts a function that returns the environment hash. The user can then use whatever logic they prefer to calculate the environment hash.

Opening this up for discussion first before I start building a pull request for this, let me know what you'd prefer ๐Ÿ˜„

Options to include or exclude modules

Provide some option set that lets users control modules that get cached. This can help in situations like #70 where a plugin or loader does not work with hard-source temporarily because of a bug or without some kind of plugin for hard-source to help it understand that external plugin or loader.

One example of a bug-less situation could be a loader that builds in a resource from a server that without a plugin to hard-source to understand a way to check for rebuilds. This option set could provide an easier path for hard-source users to let the two collaborate.

Option to disable timestamp comparison

Part of how hard-source-webpack-plugin invalidates the cache is to check the timestamps of the source files. This is all well and good on a single machine, but it prevents CI environments from being able to successfully share a build cache (barring some trickery).

Would it make more sense to store a hash of each file's contents rather than timestamps?

Unhandled rejection TypeError: Cannot read property 'forEach' of undefined

I just upgraded from v.28 to v.32 and got this

Unhandled rejection TypeError: Cannot read property 'forEach' of undefined
    at walkDependencyBlock (/.../phenomic/node_modules/hard-source-webpack-plugin/index.js:290:27)
    at /.../phenomic/node_modules/hard-source-webpack-plugin/index.js:307:9
    at Array.forEach (native)
    at /.../phenomic/node_modules/hard-source-webpack-plugin/index.js:299:32
    at tryCatcher (/.../phenomic/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/.../phenomic/node_modules/bluebird/js/release/promise.js:510:31)
    at Promise._settlePromise (/.../phenomic/node_modules/bluebird/js/release/promise.js:567:18)
    at Promise._settlePromise0 (/.../phenomic/node_modules/bluebird/js/release/promise.js:612:10)
    at Promise._settlePromises (/.../phenomic/node_modules/bluebird/js/release/promise.js:691:18)
    at Promise._fulfill (/.../phenomic/node_modules/bluebird/js/release/promise.js:636:18)
    at PromiseArray._resolve (/.../phenomic/node_modules/bluebird/js/release/promise_array.js:125:19)
    at PromiseArray._promiseFulfilled (/.../phenomic/node_modules/bluebird/js/release/promise_array.js:143:14)
    at Promise._settlePromise (/.../phenomic/node_modules/bluebird/js/release/promise.js:572:26)
    at Promise._settlePromise0 (/.../phenomic/node_modules/bluebird/js/release/promise.js:612:10)
    at Promise._settlePromises (/.../phenomic/node_modules/bluebird/js/release/promise.js:691:18)
    at Promise._fulfill (/.../phenomic/node_modules/bluebird/js/release/promise.js:636:18)

What can I give you to help more?

Record resources like fileDependencies

NormalModules have a main file resource that is resolved from the final part of a require() request. The NormalModule will record a fileDependency if a non-pitch loader is applied to it or if the pitch loader manually records the resource as a file dependency. This means there are cases where NormalModules don't include their resource as a fileDependency. Given that we need track resources like fileDependencies so we build timestamps for them since we consider the resource for whether a dependency is valid or not.

Currently this means require()ing a module that doesn't record that file dependency is being invalidated if applied loaders aren't depending on the resource.

Replace LevelDB with something simpler

Hi!

The plugin is awesome, but it uses LevelDB as a dependency. Since it's implemented in native code, it has to be built and requires python and another build tooling to be available.

Is it possible to replace LevelDB with something NodeJS-based?

Thanks!

hardsource fails with out of memory error on windows 10

I am running node v4.4.3 (npm 2.15.8) with
node --max_old_space_size=4096 node_modules/webpack-dev-server/bin/webpack-dev-server.js

(my app has around 5000 modules)

Here's what I get:

HardSourceWebpackPlugin is writing to a new confighash path for the first time: C:\code\hana_epm_fpa\webpack\build-cache\hardsource\start
  build [==============      ] 69%
<--- Last few GCs --->

  247449 ms: Mark-sweep 3965.0 (4197.1) -> 3965.0 (4197.1) MB, 298.2 / 0 ms [allocation failure] [scavenge might not succeed].
  247730 ms: Mark-sweep 3965.0 (4197.1) -> 3965.0 (4197.1) MB, 281.3 / 0 ms [allocation failure] [scavenge might not succeed].
  248013 ms: Mark-sweep 3965.0 (4197.1) -> 3964.3 (4197.1) MB, 282.9 / 0 ms [last resort gc].
  248296 ms: Mark-sweep 3964.3 (4197.1) -> 3965.0 (4197.1) MB, 282.2 / 0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 00000082FE6B4639 <JS Object>
    1: stringify [native json.js:157] [pc=0000000EED9385E4] (this=00000082FE6AD511 <a JSON with map 00000322AE3614E1>,u=000003FB5633AB51 <an Object with map 00000322AE3C1D59>,v=00000082FE6041B9 <undefined>,I=00000082FE6041B9 <undefined>)
    2: arguments adaptor frame: 1->3
    3: /* anonymous */ [C:\code\hana_epm_fpa\webpack\node_modules\hard-source-webpack-plugin\index.js:796] [pc=0000000EF...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

TypeError: Cannot read property 'dependencies' of null

I get this error when I forgot to clean the cache (eg: Checkout back to a old commit)

/home/khoa/web/nkpb/node_modules/hard-source-webpack-plugin/index.js:351
            block.dependencies.forEach(callback);
                 ^

TypeError: Cannot read property 'dependencies' of null
    at walkDependencyBlock (/home/khoa/web/nkpb/node_modules/hard-source-webpack-plugin/index.js:351:18)
    at /home/khoa/web/nkpb/node_modules/hard-source-webpack-plugin/index.js:368:13
    at Array.forEach (native)
    at /home/khoa/web/nkpb/node_modules/hard-source-webpack-plugin/index.js:360:36
    at FSReqWrap.oncomplete (fs.js:123:15)

Webpack config: https://github.com/thangngoc89/electron-react-app/blob/cf73f6e6e3da84757898c01e38d25a5ce573a036/webpack.config.js

AMD modules are not rebuilding correctly

example: https://github.com/spalger/reproduce-issues/tree/master/hard-source-webpack-plugin/amd

We have a few AMD modules in our codebase, and they don't get recreated correctly after a hard-reset. The build output includes the source, but the module wrapper does not include the __webpack_require__ variable even though the __WEBPACK_AMD_DEFINE_RESULT__ wrapper expects it.

(super excited to start using this btw, taking the 30-40 second build time for https://github.com/elastic/kibana to 2 seconds is going to be killer!)

undefined property

Trying out hard-source-webpack-plugin with Gatsby and am running into this error on one of my projects. Not really sure how to further debug this but would to love help track this down. I'm using version 0.1.0.

HardSourceWebpackPlugin is writing to a new confighash path for the first time: /Users/kylemathews/programs/blog/.intermediate-representation/cache/develop
/Users/kylemathews/programs/gatsby/node_modules/hard-source-webpack-plugin/index.js:599
          var identifier = identifierPrefix + result.request;
                                                    ^

TypeError: Cannot read property 'request' of undefined
  at /Users/kylemathews/programs/gatsby/node_modules/hard-source-webpack-plugin/index.js:599:53
  at /Users/kylemathews/programs/gatsby/node_modules/hard-source-webpack-plugin/index.js:551:22

Sometimes assets are not emitted

I have a situation where some assets required via js (require(./some-file.jpg) and consumed via file-loader are not correctly emitted. If I disable the cache, files are emitted correctly.
I was able to reproduce 100% of the time, but since I disabled and renabled, the cache has changed and I don't have the issue anymore.

Any idea about this issue?

Getting infinite hot reloading with this plugin enabled

And not when commenting out this plugin.

screen shot 2016-08-24 at 17 00 32

This is essentially what my config looks like (after some transformations):

const autoprefixer = require('autoprefixer');
const CircularDependencyPlugin = require('circular-dependency-plugin');
const HardSourceWebpackPlugin = require('hard-source-webpack-plugin');
const ManifestPlugin = require('webpack-manifest-plugin');
const path = require('path');
const webpack = require('webpack');

const config = {
  plugins: [
    new CircularDependencyPlugin({
      exclude: /node_modules\/url-parse/,
    }),
    new ManifestPlugin(),
    new HardSourceWebpackPlugin({
      cacheDirectory: path.resolve('./.hard-source-cache'),
      environmentPaths: {
        files: [
          'package.json',
          '.webpack.config.base.js',
          '.webpack.config.dev.js',
          '.webpack.config.static.js'
        ],
      },
    }),
    new webpack.DefinePlugin({
      '__DEV__': true,
    }),
    new webpack.HotModuleReplacementPlugin(),
    new webpack.NoErrorsPlugin(),
    new JasmineWebpackPlugin({
      htmlOptions: {
        chunks: ['tests'],
        filename: 'index.html',
      },
    }),
  ],
  devtool: 'eval-source-map',
  recordsPath: path.resolve('./public/client/records.json'), // For HardSourceWebpackPlugin to work.
  devServer: { hot: true, host: '0.0.0.0' },
  module: {
    loaders: [
      {
        test: /\.js$/,
        include: [
          path.resolve(__dirname, 'client/components'),
        ],
        loaders: ['react-hot'],
      },
      {
        test: /\.js$/,
        include: [
          path.resolve('./client'),
          path.resolve('./node_modules/twkb'), // Because of https://github.com/TWKB/twkb.js/commit/80ecfee82aa9a3259bb67627a03c2c47122b26a5#commitcomment-17942804
        ],
        exclude: [
          /libraries/,
        ],
        loader: 'babel',
        query: {
          cacheDirectory: '.babel-cache',
          plugins: [],
        },
      },
      {
        test: /\.less$/,
        include: path.resolve('./client'),
        loaders: [
          'style', // Inject into HTML (bundles it in JS).
          'css?localIdentName=[path][name]--[local]--[hash:base64:10]', // Resolves url() and :local().
          'postcss-loader', // Autoprefixer (see below at `postcss()`).
          'less', // LESS preprocessor.
        ],
      },
      {
        test: /\.json$/,
        include: path.resolve('./client'),
        loaders: [
          'json-loader',
        ],
      },
      {
        test: /\.(jpe?g|png|gif)$/i,
        include: path.resolve('./client'),
        loaders: [
          'url?limit=10000', // Inline small images, otherwise create file.
          'img?progressive=true', // Minify images.
        ],
      },
      {
        test: /\.geojson$/, // GeoJSON fixtures
        loader: 'json',
      },
      {
        test: /\.svg$/,
        loader: 'raw',
      },
    ],
  },
  postcss() {
    return [autoprefixer];
  },
  entry: {
    build: [
      'webpack/hot/dev-server'
      'webpack-dev-server/client?http://localhost:8080',
      'client/build',
    ],
    tests: ['client/tests'],
    happo: ['client/happo'],
  },
  resolve: {
    alias: {
      client: path.resolve('./client'),
    },
    modulesDirectories: [
      'node_modules',
    ],
    extensions: [ // Default extensions.
      '',
      '.js',
    ],
  },
  output: {
    path: 'public/client', // In case this is run without webpack-dev-server.
    publicPath: 'http://localhost:8080/',
    filename: '[name].js',
  },
};

Doesn't appear to be working with spike

Hey there,

So I know you might not really care about this, but figured I'd at least post it here in case you might see a quick solution or have the interest to walk me through a way to dive deeper. I am more than happy to get my hands dirty, but would need a little hand-holding at the beginning as I'm not nearly as familiar with webpack's internals or cache structure as you are.

I work on spike, which is essentially a static website generator that uses webpack as its core compiler. So we route not only js, but also html and css through webpack's compiler, and have a plugin to inject and extract files from the pipeline so it behaves as expected. This tool is still in pre-release and hasn't seen any publicity yet, but we are gearing up to make a publicity push sometime in the next couple of months and are expecting decent adoption.

Especially for static sites, something like this plugin would be a game-changer. If we could get this working, someone would be able to build out even very large and content-heavy sites as completely static. So you could take, say, a wordpress site with a couple thousand posts, add the wordpress api plugin, hook up the API to spike, and have it generate the site static. Then take a webhook from wordpress and link it up to your host to have it rebuild when a new post goes up.

At the moment, having a persistent cache between builds is the only thing that stands in the way of making this tool able to easily and quickly handle any size/complexity of website, so needless to say we are really excited to try to get it integrated.

So on to the actual issue! I hooked up the plugin and ran it with a test site in which I generated a couple hundred view files with some resource-heavy transforms. Initial compile was ~30s, and subsequent compiles averaged out around ~24s. I'm not sure how much reduction I should expect, but presumably it would be a little more than this?

I peeked into the records.json and cache/resolve.json files to ensure that all the files were being tracked, as spike does do a little extra manipulation with webpack's pipeline outside the ordinary, but it does appear as if they are all on there. Here's a sample project that demonstrates the issue. You can run a compile using npm start if you are testing.

Thanks so much for working on this incredible plugin, and for taking the time to read this little novel I've written! ๐Ÿ’–

Compiler Stats - errors and warnings - dep.loc.start undefined.

I get this error from HtmlWebpackPlugin, but I looked through HtmlWebpackPlugin doesn't seem like any of the options avoid this line because it happens within the emit step.

Error from second compilation

Unhandled rejection TypeError: Cannot read property 'line' of undefined
    at ./node_modules/webpack/lib/Stats.js:224:60
    at Array.map (native)
    at fnModule (./node_modules/webpack/lib/Stats.js:211:7)
    at Array.map (native)
    at Stats.toJson (./node_modules/webpack/lib/Stats.js:278:58)
    at Compiler.<anonymous> (./node_modules/html-webpack-plugin/index.js:68:59)
    at Compiler.applyPluginsAsync (./node_modules/tapable/lib/Tapable.js:71:13)
    at Compiler.emitAssets (./node_modules/webpack/lib/Compiler.js:226:7)
    at Watching.<anonymous> (./node_modules/webpack/lib/Compiler.js:54:18)
    at ./node_modules/webpack/lib/Compiler.js:403:12
    at Compiler.next (./node_modules/tapable/lib/Tapable.js:67:11)
    at Compiler.<anonymous> (./node_modules/webpack/lib/CachePlugin.js:40:4)
    at Compiler.next (./node_modules/tapable/lib/Tapable.js:69:14)
    at ./node_modules/hard-source-webpack-plugin/index.js:712:7
From previous event:
    at Compiler.<anonymous> (./node_modules/hard-source-webpack-plugin/index.js:710:6)
    at Compiler.applyPluginsAsync (./node_modules/tapable/lib/Tapable.js:71:13)
    at Compiler.<anonymous> (./node_modules/webpack/lib/Compiler.js:400:9)
    at Compilation.<anonymous> (./node_modules/webpack/lib/Compilation.js:577:13)
    at Compilation.applyPluginsAsync (./node_modules/tapable/lib/Tapable.js:60:69)
    at Compilation.<anonymous> (./node_modules/webpack/lib/Compilation.js:572:10)
    at Compilation.applyPluginsAsync (./node_modules/tapable/lib/Tapable.js:60:69)
    at Compilation.<anonymous> (./node_modules/webpack/lib/Compilation.js:567:9)
    at Compilation.next (./node_modules/tapable/lib/Tapable.js:67:11)
    at ExtractTextPlugin.<anonymous> (./node_modules/extract-text-webpack-plugin/index.js:309:4)
    at Compilation.applyPluginsAsync (./node_modules/tapable/lib/Tapable.js:71:13)
    at Compilation.<anonymous> (./node_modules/webpack/lib/Compilation.js:563:8)
    at Compilation.next (./node_modules/tapable/lib/Tapable.js:67:11)
    at ExtractTextPlugin.<anonymous> (./node_modules/extract-text-webpack-plugin/index.js:285:5)
    at ./node_modules/async/lib/async.js:52:16
    at Object.async.forEachOf.async.eachOf (./node_modules/async/lib/async.js:236:30)
    at Object.async.forEach.async.each (./node_modules/async/lib/async.js:209:22)
    at ExtractTextPlugin.<anonymous> (./node_modules/extract-text-webpack-plugin/index.js:237:10)
    at Compilation.applyPluginsAsync (./node_modules/tapable/lib/Tapable.js:71:13)
    at Compilation.seal (./node_modules/webpack/lib/Compilation.js:525:7)
    at Compiler.<anonymous> (./node_modules/webpack/lib/Compiler.js:397:15)
    at ./node_modules/tapable/lib/Tapable.js:103:11
    at Compilation.<anonymous> (./node_modules/webpack/lib/Compilation.js:445:10)
    at ./node_modules/webpack/lib/Compilation.js:417:12
    at ./node_modules/webpack/lib/Compilation.js:332:10

Versions

"hard-source-webpack-plugin": "0.0.42"
"html-webpack-plugin": "2.22.0",
"webpack": "1.13.2"

The error actually comes from webpack/lib/Stats.js because options.reasons defaults to true

    var showReasons = d(options.reasons, !forToString);

I think these are stats of errors and warnings. Eg. compilation.errors compilation.warnings

Webpack's failing block

    obj.reasons = module.reasons.filter(function(reason) {
        return reason.dependency && reason.module;
    }).map(function(reason) {
        var obj = {
            moduleId: reason.module.id,
            moduleIdentifier: reason.module.identifier(),
            module: reason.module.readableIdentifier(requestShortener),
            moduleName: reason.module.readableIdentifier(requestShortener),
            type: reason.dependency.type,
            userRequest: reason.dependency.userRequest
        };
        var dep = reason.dependency;
        if(dep.templateModules) obj.templateModules = dep.templateModules.map(function(module) {
            return module.id;
        });
        if(typeof dep.loc === "object") obj.loc = dep.loc.start.line + ":" + dep.loc.start.column + "-" +
            (dep.loc.start.line !== dep.loc.end.line ? dep.loc.end.line + ":" : "") + dep.loc.end.column;
        return obj;
    }).sort(function(a, b) {
        return a.moduleId - b.moduleId;
    });

Essentially, dep.loc exists, but doesn't have the start/end line/column properties. To avoid this we either need to cache the locations or they should be removed to avoid this. Below is what the compilation object looks like on the second run. If instead loc was undefined or loc.start and loc.end were undefined it wouldn't fail.

compilation = {
    errors: [{
        loc: {
            start: undefined,
            end: undefined,
        },
    }],
    warnings: [{
        loc: {
            start: undefined,
            end: undefined,
        }
    }]
}

I only get into this error due to HtmlWebpackPlugin, but I don't think it's avoidable by configuration because it happens right after the emit step (see below). Looking through the previous issues, I know some people have been using this plugin successfully with it. Perhaps this is by just not having errors or warnings in the compilation object?

  compiler.plugin('emit', function (compilation, callback) {
    var applyPluginsAsyncWaterfall = Promise.promisify(compilation.applyPluginsAsyncWaterfall, {context: compilation});
    // Get all chunks
    var chunks = self.filterChunks(compilation.getStats().toJson(), self.options.chunks, self.options.excludeChunks);

I'd be happy to fix it if someone could point me in the right direction. My guess was that the best way would just to be to modify serializeDependencies in https://github.com/mzgoddard/hard-source-webpack-plugin/blob/1f5618cf358d274e9777b67ea5dba97db0989284/index.js, but I wanted to confirm that.

In case anyone else is running into this, my workaround was the following. Obviously this could indirectly create some other problems for a plugin that actually needs the stats and doesn't specify it, but this worked for me.

var WebpackStats = require('webpack/lib/Stats');
  var wsToJson = WebpackStats.prototype.toJson;
  WebpackStats.prototype.toJson = function toJson() {
    var args = [].slice.call(arguments);
    if (!args.length) {
      args.push({ reasons: false });
    }
    return wsToJson.apply(this, args);
  };

Thanks a lot by the way, I think this plugin is going to make us a lot more productive.

HMR does not work if included in plugins

Hello,

Awesome project!

I've noticed an issue when using this in combination with webpack.HotModuleReplacementPlugin, namely that starting the dev server will succeed once and then fail upon a rebuild (i.e. starting webpack-dev-server again) if the HMR plugin was included in plugins in the webpack config. However, if this is removed and HMR is enabled through the --hot flag, there are no issues.

If it's included as a plugin, the following error is produced on a rebuild:

Unhandled rejection TypeError: Cannot read property 'line' of undefined
    at /ProjectDir/node_modules/webpack/lib/Stats.js:224:60
    at Array.map (native)
    at fnModule (/ProjectDir/node_modules/webpack/lib/Stats.js:211:7)
    at Array.map (native)
    at Stats.toJson (/ProjectDir/node_modules/webpack/lib/Stats.js:278:58)
    at Server.<anonymous> (/ProjectDir/node_modules/webpack-dev-server/lib/Server.js:34:39)
    at Compiler.applyPlugins (/ProjectDir/node_modules/tapable/lib/Tapable.js:26:37)
    at Watching._done (/ProjectDir/node_modules/webpack/lib/Compiler.js:78:17)
    at Watching.<anonymous> (/ProjectDir/node_modules/webpack/lib/Compiler.js:61:18)
    at MemoryFileSystem.writeFile (/ProjectDir/node_modules/memory-fs/lib/MemoryFileSystem.js:314:9)
    at Compiler.writeFile (/ProjectDir/node_modules/webpack/lib/Compiler.js:295:25)
    at Compiler.<anonymous> (/ProjectDir/node_modules/webpack/lib/Compiler.js:291:13)
    at MemoryFileSystem.(anonymous function) [as mkdirp] (/ProjectDir/node_modules/memory-fs/lib/MemoryFileSystem.js:283:10)
    at Compiler.emitRecords (/ProjectDir/node_modules/webpack/lib/Compiler.js:289:24)
    at Watching.<anonymous> (/ProjectDir/node_modules/webpack/lib/Compiler.js:58:19)
    at /ProjectDir/node_modules/webpack/lib/Compiler.js:275:11
From previous event:
    at Compiler.<anonymous> (/ProjectDir/node_modules/hard-source-webpack-plugin/index.js:710:6)

If you need more information about our setup to debug this, please let me know. :)

Inavlidate cache when webpack config changes

Have been looking at this plugin to reduce my team's rebuild times, but was wondering if cache invalidation might be more trouble than it's worth, especially in a medium sized team.

Since this plugin has access to the webpack configuration object, shouldn't it be easy for the plugin to store and compare hashes of the object and invalidate cache automatically?

Agreed that this may not handle all cases - for eg. when you update versions of the plugin, but those change very infrequently compared to the webpack config

Error after switching branches: TypeError: Cannot read property 'dependencies' of null at walkDependencyBlock

I added some new files in my branch, committed them then switched back to master (my branch has yet to be merged) and this error occurred:

Unhandled rejection TypeError: Cannot read property 'dependencies' of null
    at walkDependencyBlock (/my/project/node_modules/hard-source-webpack-plugin/index.js:233:16)
    at /my/project/node_modules/hard-source-webpack-plugin/index.js:250:11
    at Array.forEach (native)
    at /my/project/node_modules/hard-source-webpack-plugin/index.js:242:34
    at tryCatcher (/my/project/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/my/project/node_modules/bluebird/js/release/promise.js:504:31)
    at Promise._settlePromise (/my/project/node_modules/bluebird/js/release/promise.js:561:18)
    at Promise._settlePromise0 (/my/project/node_modules/bluebird/js/release/promise.js:606:10)
    at Promise._settlePromises (/my/project/node_modules/bluebird/js/release/promise.js:681:18)
    at Async._drainQueue (/my/project/node_modules/bluebird/js/release/async.js:138:16)
    at Async._drainQueues (/my/project/node_modules/bluebird/js/release/async.js:148:10)
    at Immediate.Async.drainQueues [as _onImmediate] (/my/project/node_modules/bluebird/js/release/async.js:17:14)
    at tryOnImmediate (timers.js:543:15)
    at processImmediate [as _immediateCallback] (timers.js:523:5)

Support automatic deletion of obsolete cache directories

On a medium-sized project with several bundled assets, the hard sorce cache directory for each configuration hash ends up being around 50mb. This isn't a lot on its own, but after a couple weeks of rapid iteration the overall disk footprint for the parent cache directory ballooned to almost a gig.

This may be out of scope for the plugin, and cleanup steps may certainly be enacted outside of webpack, but I wanted to suggest that it could be helpful to be able to pass a isCacheStale method to the plugin, which could be passed information about the cache file such as how recently a particular cache folder was used or how many have been created since, and return a boolean to be used to remove the cache. That would let us choose whether we want to remove caches after a week, or only keep the latest 3, or some other metric specific to a given project.

TypeError: Cannot read property 'start' of undefined

I get the following error when running this a second time:

~/node_modules/webpack/lib/ModuleDependencyWarning.js:9
        this.message = loc.start.line + ":" + loc.start.column + " ";
                   ^
TypeError: Cannot read property 'start' of undefined
    at new ModuleDependencyWarning (~/node_modules/webpack/lib/ModuleDependencyWarning.js:9:20)

More...

~/node_modules/webpack/lib/ModuleDependencyWarning.js:9
        this.message = loc.start.line + ":" + loc.start.column + " ";
                   ^
TypeError: Cannot read property 'start' of undefined
    at new ModuleDependencyWarning (~/node_modules/webpack/lib/ModuleDependencyWarning.js:9:20)
    at ~/node_modules/webpack/lib/Compilation.js:643:20
    at Array.forEach (native)
    at ~/node_modules/webpack/lib/Compilation.js:642:14
    at Array.forEach (native)
    at ~/node_modules/webpack/lib/Compilation.js:639:22
    at Array.forEach (native)
    at Compilation.reportDependencyWarnings (~/node_modules/webpack/lib/Compilation.js:638:9)
    at Compilation.<anonymous> (~/node_modules/webpack/lib/Compilation.js:497:8)
    at Array.forEach (native)
    at Compilation.finish (~/node_modules/webpack/lib/Compilation.js:496:15)
    at Compiler.<anonymous> (~/node_modules/webpack/lib/Compiler.js:434:15)
    at ~/node_modules/tapable/lib/Tapable.js:152:11
    at Compilation.<anonymous> (~/node_modules/webpack/lib/Compilation.js:444:10)
    at ~/node_modules/webpack/lib/Compilation.js:420:12
    at ~/node_modules/webpack/lib/Compilation.js:332:10
    at ~/node_modules/async/lib/async.js:52:16
    at done (~/node_modules/async/lib/async.js:246:17)
    at ~/node_modules/async/lib/async.js:44:16
    at ~/node_modules/webpack/lib/Compilation.js:332:10
    at ~/node_modules/async/lib/async.js:52:16
    at done (~/node_modules/async/lib/async.js:246:17)
    at ~/node_modules/async/lib/async.js:44:16
    at ~/node_modules/webpack/lib/Compilation.js:332:10
    at ~/node_modules/async/lib/async.js:52:16
    at done (~/node_modules/async/lib/async.js:246:17)
    at ~/node_modules/async/lib/async.js:44:16
    at ~/node_modules/webpack/lib/Compilation.js:332:10
    at ~/node_modules/async/lib/async.js:52:16
    at done (~/node_modules/async/lib/async.js:246:17)
    at ~/node_modules/async/lib/async.js:44:16
    at ~/node_modules/webpack/lib/Compilation.js:332:10
    at ~/node_modules/async/lib/async.js:52:16
    at done (~/node_modules/async/lib/async.js:246:17)
    at ~/node_modules/async/lib/async.js:44:16
    at ~/node_modules/webpack/lib/Compilation.js:332:10
    at ~/node_modules/async/lib/async.js:52:16
    at done (~/node_modules/async/lib/async.js:246:17)
    at ~/node_modules/async/lib/async.js:44:16
    at ~/node_modules/webpack/lib/Compilation.js:332:10
    at ~/node_modules/async/lib/async.js:52:16
    at done (~/node_modules/async/lib/async.js:246:17)
    at ~/node_modules/async/lib/async.js:44:16
    at ~/node_modules/webpack/lib/Compilation.js:332:10
    at ~/node_modules/async/lib/async.js:52:16
    at done (~/node_modules/async/lib/async.js:246:17)
    at ~/node_modules/async/lib/async.js:44:16
    at ~/node_modules/webpack/lib/Compilation.js:332:10
    at ~/node_modules/async/lib/async.js:52:16
    at done (~/node_modules/async/lib/async.js:246:17)
    at ~/node_modules/async/lib/async.js:44:16
    at ~/node_modules/webpack/lib/Compilation.js:332:10
    at ~/node_modules/async/lib/async.js:52:16
    at done (~/node_modules/async/lib/async.js:246:17)
    at ~/node_modules/async/lib/async.js:44:16
    at ~/node_modules/webpack/lib/Compilation.js:332:10
    at ~/node_modules/async/lib/async.js:52:16
    at done (~/node_modules/async/lib/async.js:246:17)
    at ~/node_modules/async/lib/async.js:44:16
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickDomainCallback (internal/process/next_tick.js:122:9)

What can I provide to make this more useful? (I'm using the example on the homepage, nearly verbatim)

Also, this project has:

  • Code-splitting via require.ensure.
  • require.context in several places.
  • babel-loader, and a few others.

When checking out records.json, I'd wager that the files referenced via require.context strike me as the outliers:

{
  "modules": {
    "byIdentifier": {
      "node_modules/react/react.js": "./node_modules/react/react.js",
      "src/routes/docs /\\.js$": 0,
      "multi client": 1,
      "multi widgets": 2,
      "src/components /\\.fixture\\.js$": 3,
      "src/components /\\/[A-Z]\\w+\\.js$": 4,
      // ...a bunch of /node_module/something: node_modules/something
    },
    "usedIds": {
      "0": 0,
      "1": 1,
      "2": 2,
      "3": 3,
      "4": 4,
      "5": 5,
      "./node_modules/react/react.js": "./node_modules/react/react.js",
      // a bunch of ./some/path.js: ./some/path.js
    }
  },
  "chunks": {
    "byName": {
      "client": 5,
      "widgets": 6
    },
    "byBlocks": {
      "node_modules/babel-loader/index.js?{\"cacheDirectory\":true,\"env\":{\"development\":{\"plugins\":[[\"react-transform\",{\"transforms\":[{\"transform\":\"react-transform-hmr\",\"imports\":[\"react\"],\"locals\":[\"module\"]}]}]]}}}!node_modules/eslint-loader/index.js!src/routes/index.js:0/1": 0,
      "node_modules/babel-loader/index.js?{\"cacheDirectory\":true,\"env\":{\"development\":{\"plugins\":[[\"react-transform\",{\"transforms\":[{\"transform\":\"react-transform-hmr\",\"imports\":[\"react\"],\"locals\":[\"module\"]}]}]]}}}!node_modules/eslint-loader/index.js!src/routes/index.js:1/1": 1,
      "node_modules/babel-loader/index.js?{\"cacheDirectory\":true,\"env\":{\"development\":{\"plugins\":[[\"react-transform\",{\"transforms\":[{\"transform\":\"react-transform-hmr\",\"imports\":[\"react\"],\"locals\":[\"module\"]}]}]]}}}!node_modules/eslint-loader/index.js!src/routes/components/index.js:0/0": 2,
      "node_modules/babel-loader/index.js?{\"cacheDirectory\":true,\"env\":{\"development\":{\"plugins\":[[\"react-transform\",{\"transforms\":[{\"transform\":\"react-transform-hmr\",\"imports\":[\"react\"],\"locals\":[\"module\"]}]}]]}}}!node_modules/eslint-loader/index.js!src/routes/dashboard/index.js:0/0": 3,
      "node_modules/babel-loader/index.js?{\"cacheDirectory\":true,\"env\":{\"development\":{\"plugins\":[[\"react-transform\",{\"transforms\":[{\"transform\":\"react-transform-hmr\",\"imports\":[\"react\"],\"locals\":[\"module\"]}]}]]}}}!node_modules/eslint-loader/index.js!src/routes/login/index.js:0/0": 4
    },
    "usedIds": {
      "0": 0,
      "1": 1,
      "2": 2,
      "3": 3,
      "4": 4,
      "5": 5,
      "6": 6
    }
  },
  "prepreHash": "x",
  "preHash": "bb85840e4b6505f91209",
  "hash": "806005adce5d53614746",
  "moduleHashs": {

Doesn't work w/ webpack 2.2 rc.0

/Users/aaronjensen/Source/app/node_modules/hard-source-webpack-plugin/index.js:939
          return stat(resource)
                 ^

TypeError: stat is not a function
    at /Users/aaronjensen/Source/app/node_modules/hard-source-webpack-plugin/index.js:939:18
    at /Users/aaronjensen/Source/app/node_modules/hard-source-webpack-plugin/index.js:949:12
    at /Users/aaronjensen/Source/app/node_modules/webpack/lib/NormalModuleFactory.js:59:4
    at /Users/aaronjensen/Source/app/node_modules/webpack/lib/NormalModuleFactory.js:234:3
    at /Users/aaronjensen/Source/app/node_modules/tapable/lib/Tapable.js:168:11
    at NormalModuleFactory.<anonymous> (/Users/aaronjensen/Source/app/node_modules/hard-source-webpack-plugin/index.js:1043:39)
    at /Users/aaronjensen/Source/app/node_modules/tapable/lib/Tapable.js:170:14
    at NormalModuleFactory.<anonymous> (/Users/aaronjensen/Source/app/node_modules/webpack/lib/NormalModuleReplacementPlugin.js:26:11)
    at /Users/aaronjensen/Source/app/node_modules/tapable/lib/Tapable.js:170:14
    at NormalModuleFactory.<anonymous> (/Users/aaronjensen/Source/app/node_modules/webpack/lib/NormalModuleReplacementPlugin.js:26:11)
    at NormalModuleFactory.applyPluginsAsyncWaterfall (/Users/aaronjensen/Source/app/node_modules/tapable/lib/Tapable.js:172:13)
    at NormalModuleFactory.create (/Users/aaronjensen/Source/app/node_modules/webpack/lib/NormalModuleFactory.js:218:8)
    at iteratorFactory (/Users/aaronjensen/Source/app/node_modules/webpack/lib/Compilation.js:213:11)
    at /Users/aaronjensen/Source/app/node_modules/async/dist/async.js:3025:16
    at eachOfArrayLike (/Users/aaronjensen/Source/app/node_modules/async/dist/async.js:941:9)
    at eachOf (/Users/aaronjensen/Source/app/node_modules/async/dist/async.js:991:5)

Cannot read property 'module' of undefined

I get this on my second (cached) build:

/Users/aaronjensen/Source/the-link/node_modules/webpack/lib/dependencies/HarmonyExportImportedSpecifierDependency.js:26
        if(!this.importDependency.module || !used || !active) return null;
                                 ^

TypeError: Cannot read property 'module' of undefined
    at HardHarmonyExportImportedSpecifierDependency.HarmonyExportImportedSpecifierDependency.getReference (/Users/aaronjensen/Source/the-link/node_modules/webpack/lib/dependencies/HarmonyExportImportedSpecifierDependency.js:26:27)
    at processDependency (/Users/aaronjensen/Source/the-link/node_modules/webpack/lib/FlagDependencyUsagePlugin.js:55:44)
    at /Users/aaronjensen/Source/the-link/node_modules/webpack/lib/FlagDependencyUsagePlugin.js:42:5
    at Array.forEach (native)
    at processDependenciesBlock (/Users/aaronjensen/Source/the-link/node_modules/webpack/lib/FlagDependencyUsagePlugin.js:41:26)
    at processModule (/Users/aaronjensen/Source/the-link/node_modules/webpack/lib/FlagDependencyUsagePlugin.js:37:4)
    at processDependency (/Users/aaronjensen/Source/the-link/node_modules/webpack/lib/FlagDependencyUsagePlugin.js:62:5)
    at /Users/aaronjensen/Source/the-link/node_modules/webpack/lib/FlagDependencyUsagePlugin.js:42:5
    at Array.forEach (native)
    at processDependenciesBlock (/Users/aaronjensen/Source/the-link/node_modules/webpack/lib/FlagDependencyUsagePlugin.js:41:26)
    at processModule (/Users/aaronjensen/Source/the-link/node_modules/webpack/lib/FlagDependencyUsagePlugin.js:37:4)
    at processDependency (/Users/aaronjensen/Source/the-link/node_modules/webpack/lib/FlagDependencyUsagePlugin.js:62:5)
    at /Users/aaronjensen/Source/the-link/node_modules/webpack/lib/FlagDependencyUsagePlugin.js:42:5
    at Array.forEach (native)
    at processDependenciesBlock (/Users/aaronjensen/Source/the-link/node_modules/webpack/lib/FlagDependencyUsagePlugin.js:41:26)
    at processModule (/Users/aaronjensen/Source/the-link/node_modules/webpack/lib/FlagDependencyUsagePlugin.js:37:4)
    at processDependency (/Users/aaronjensen/Source/the-link/node_modules/webpack/lib/FlagDependencyUsagePlugin.js:62:5)
    at /Users/aaronjensen/Source/the-link/node_modules/webpack/lib/FlagDependencyUsagePlugin.js:42:5
    at Array.forEach (native)
    at processDependenciesBlock (/Users/aaronjensen/Source/the-link/node_modules/webpack/lib/FlagDependencyUsagePlugin.js:41:26)
    at processModule (/Users/aaronjensen/Source/the-link/node_modules/webpack/lib/FlagDependencyUsagePlugin.js:37:4)
    at /Users/aaronjensen/Source/the-link/node_modules/webpack/lib/FlagDependencyUsagePlugin.js:20:6
    at Array.forEach (native)
    at Compilation.<anonymous> (/Users/aaronjensen/Source/the-link/node_modules/webpack/lib/FlagDependencyUsagePlugin.js:18:23)
    at Compilation.applyPluginsBailResult (/Users/aaronjensen/Source/the-link/node_modules/webpack/node_modules/tapable/lib/Tapable.js:52:27)
    at Compilation.seal (/Users/aaronjensen/Source/the-link/node_modules/webpack/lib/Compilation.js:543:8)
    at Compiler.<anonymous> (/Users/aaronjensen/Source/the-link/node_modules/webpack/lib/Compiler.js:436:15)
    at /Users/aaronjensen/Source/the-link/node_modules/webpack/node_modules/tapable/lib/Tapable.js:152:11
    at Compilation.<anonymous> (/Users/aaronjensen/Source/the-link/node_modules/webpack/lib/Compilation.js:444:10)
    at /Users/aaronjensen/Source/the-link/node_modules/webpack/lib/Compilation.js:420:12

Doesn't work with html-webpack-plugin

On the second run, compilation fails with following error (stack shortened for brevity):

[Compilation]
  Error: /app/guide/guide.html.ejs:519
  /******/  modules[moduleId].call(module.exports, module, module.exports, hotCreateRequire(moduleId));
                             ^
  TypeError: Cannot read property 'call' of undefined

  - guide.html.ejs:519 __webpack_require__
    /app/guide/guide.html.ejs:519:30

  - index.js:225 HtmlWebpackPlugin.evaluateCompilationResult
    [html-webpack-plugin]/index.js:225:26

It does not depend on the html template used. It fails even if template is empty.

From what I've gathered, html-webpack-plugin does a "child compilation" and evaluates it's result.

@ampedandwired, can you shed some light on this?

Need an approach to invalidate built modules

8d35ff0 shows one of the ways we need to bust cached modules. In it cached modules are busted when their dependencies no longer point an existing file. To do so it uses an approach to preemptively bust modules when checked fileDependencies no longer exist on the system. Preemptively busting modules is the best way but we likely need an way to bust after a module has been fully thawed and added to a compilation. That or some kind of hybrid approach.

One current issue is for webpack 2 support we disable tree shaking that is supported through Module's isUsed method. This way all rendered modules include all exports. If dependents change their needs the functions are available. This plugin will be best when it can support at minimum webpack's built-in features and output the same results as if you weren't using it.

The issue with solving this flaw for webpack 2 is when we can know that what a cached module needs to change. The obvious way is to wait for a point in the build that the used exports are different. Actually fulfilling that idea itself may be difficult as we need to add some callback just before webpack's "make" step completes. We can't modify the compilation in the way we need once that completes. There is no hook for this. A second issue with this approach is we can't remove modules in an existing compilation. Most we can, its just hard to make sure we replace them correctly. However we can't replace entry modules. This relates to the first issue there is no hook for this. I'm not sure if we'd want one.

One possible method will be to run a childCompiler building up the real tree. We can then check the module info of the cached modules and bust ones whose render no longer fits their shape. Then in the real build the up to date modules could be used. A custom child compiler could be made to do this quickly, do just the "make" step, and its modules used like those used in the webpack watch cache.

Strange caching-related errors

I use hot reloading and this plugin, and every so often I get an error like this after an update:

Uncaught TypeError: Cannot call a class as a function
    at _classCallCheck (webpack:///./~/redux-form/es/Field.js?:12:99)
    at Field (webpack:///./~/redux-form/es/Field.js?:39:7)

It always happened with redux-form so far. When I trace through the problem, there is a mixup of exports. The webpack_requires are supposed to import one export, but in fact return another export of the same module.

image
Note how it passes export b, hoping for reduxForm, but in fact getting Field

I don't change the redux-form code, so it's quite mysterious as to why it suddenly changes export.

The only way to make it go away is to clear the hard-source-webpack-plugin cache.

Broken with latest webpack 2.1.0-beta23+

When cache is off, build is ok.
When cache is on I am getting random "Module parse failed" "_this.parser.parse is not a function" on a lot of file. Maybe all of them.

Any idea?

ModuleParseError: Module parse failed: /Users/moox/Sync/Development/phenomic/lib/redux/createStore/index.js _this.parser.parse is not a function
You may need an appropriate loader to handle this file type.
TypeError: _this.parser.parse is not a function
    at /Users/moox/Sync/Development/phenomic/node_modules/webpack/lib/NormalModule.js:199:17
    at /Users/moox/Sync/Development/phenomic/node_modules/webpack/lib/NormalModule.js:164:10
    at /Users/moox/Sync/Development/phenomic/node_modules/loader-runner/lib/LoaderRunner.js:365:3
    at iterateNormalLoaders (/Users/moox/Sync/Development/phenomic/node_modules/loader-runner/lib/LoaderRunner.js:206:10)
    at /Users/moox/Sync/Development/phenomic/node_modules/loader-runner/lib/LoaderRunner.js:197:4
    at Storage.finished (/Users/moox/Sync/Development/phenomic/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:39:16)
    at /Users/moox/Sync/Development/phenomic/node_modules/graceful-fs/graceful-fs.js:78:16
    at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:445:3)
    at /Users/moox/Sync/Development/phenomic/node_modules/webpack/lib/NormalModule.js:207:34
    at /Users/moox/Sync/Development/phenomic/node_modules/webpack/lib/NormalModule.js:164:10
    at /Users/moox/Sync/Development/phenomic/node_modules/loader-runner/lib/LoaderRunner.js:365:3
    at iterateNormalLoaders (/Users/moox/Sync/Development/phenomic/node_modules/loader-runner/lib/LoaderRunner.js:206:10)
    at /Users/moox/Sync/Development/phenomic/node_modules/loader-runner/lib/LoaderRunner.js:197:4
    at Storage.finished (/Users/moox/Sync/Development/phenomic/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:39:16)
    at /Users/moox/Sync/Development/phenomic/node_modules/graceful-fs/graceful-fs.js:78:16
    at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:445:3) 

Structure

This a step towards getting this plugin to become a spec or plugin-able plugin. In general we need to shape the contents to expose some common flows. We also need to do some work to be a good neighbor.

  • Rewrite async use to promises (more of a personal organizational decision). 01d0df6
  • Dependencies and Modules should be their own subclass of the appropriate type and named to help indicate where they came from (this plugin). bad1aab
  • Separate devtool obj generator (have in its own file to help track changes to it outside of the index). f719b44
  • Separate cache reading and writing (possibly multiple caches). ea8dc24
  • Split out sub-plugins.

Support webpack 2.2

Webpack 2.2-rc recently released. Could you please update peerDependencies?

Breaks with webpack v2 beta28

There seems to be a new issue with the just released webpack v2 beta:

/Users/sebastian/Workspace/sebastian-software/advanced-boilerplate/node_modules/hard-source-webpack-plugin/index.js:130
        harmonyName: dep.describeHarmonyExport().exportedName,
                         ^

TypeError: dep.describeHarmonyExport is not a function

When changing these lines it seems to work:

    if (!cacheDep && dep.originModule) {
      console.log("ID:", dep.id, !!dep.describeHarmonyExport)
      cacheDep = {
        harmonyExport: true,
        harmonyId: dep.id,
        harmonyName: dep.describeHarmonyExport && dep.describeHarmonyExport().exportedName,
        harmonyPrecedence: dep.describeHarmonyExport && dep.describeHarmonyExport().precedence,
      };
    }

I am still wondering at this part because there are a lot of dependencies which have neither an ID or harmonyName:

...
ID: createConnect true
ID: undefined true
ID: undefined false
ID: whenMapDispatchToPropsIsFunction true
ID: whenMapDispatchToPropsIsMissing true
ID: whenMapDispatchToPropsIsObject true
ID: undefined true
ID: undefined false
ID: whenMapStateToPropsIsFunction true
ID: whenMapStateToPropsIsMissing true
ID: undefined true
ID: undefined false
ID: defaultMergeProps true
ID: wrapMergePropsFunc true
ID: whenMergePropsIsFunction true
ID: whenMergePropsIsOmitted true
ID: undefined true
ID: undefined false
ID: impureFinalPropsSelectorFactory true
ID: pureFinalPropsSelectorFactory true
ID: finalPropsSelectorFactory true
ID: undefined false
ID: verifySubselectors true
ID: undefined false
ID: undefined false
ID: Subscription true
ID: undefined false
ID: shallowEqual true
ID: undefined false
ID: undefined true
...

This seems to be somehow wrong.

BUG: Does not work with hot dev server + karma

I set up HardSourceWebpackPlugin in both my karma.conf.js and in my webpack.config.hot-dev-server.js. Using the plugin in each config file works just fine on its own, but only one of these can be active at the same time, otherwise the second (post-cache) build fails with:

/Users/David/dev/poc-oliver/client/node_modules/source-list-map/lib/fromStringWithSourceMap.js:11
	var sources = map.sources;
	                 ^

TypeError: Cannot read property 'sources' of null
    at fromStringWithSourceMap (/Users/David/dev/poc-oliver/client/node_modules/source-list-map/lib/fromStringWithSourceMap.js:11:19)
    at HardSource.listMap (/Users/David/dev/poc-oliver/client/node_modules/hard-source-webpack-plugin/lib/hard-source.js:57:10)
    at /Users/David/dev/poc-oliver/client/node_modules/webpack-sources/lib/ConcatSource.js:51:17
    at Array.forEach (native)
    at ConcatSource.listMap (/Users/David/dev/poc-oliver/client/node_modules/webpack-sources/lib/ConcatSource.js:47:16)
    at /Users/David/dev/poc-oliver/client/node_modules/webpack-sources/lib/ConcatSource.js:51:17
    at Array.forEach (native)
    at ConcatSource.listMap (/Users/David/dev/poc-oliver/client/node_modules/webpack-sources/lib/ConcatSource.js:47:16)
    at /Users/David/dev/poc-oliver/client/node_modules/webpack-sources/lib/ConcatSource.js:51:17
    at Array.forEach (native)
    at ConcatSource.listMap (/Users/David/dev/poc-oliver/client/node_modules/webpack-sources/lib/ConcatSource.js:47:16)
    at /Users/David/dev/poc-oliver/client/node_modules/webpack-sources/lib/ConcatSource.js:51:17
    at Array.forEach (native)
    at ConcatSource.listMap (/Users/David/dev/poc-oliver/client/node_modules/webpack-sources/lib/ConcatSource.js:47:16)
    at ConcatSource.proto.sourceAndMap (/Users/David/dev/poc-oliver/client/node_modules/webpack-sources/lib/SourceAndMapMixin.js:23:16)
    at CachedSource.sourceAndMap (/Users/David/dev/poc-oliver/client/node_modules/webpack-sources/lib/CachedSource.js:51:28)

I'm using the plugin as follows:

// karma.conf.js
new HardSourceWebpackPlugin({
  cacheDirectory: path.join(__dirname, 'test-cache/[confighash]'),
  recordsPath: path.join(__dirname, 'test-cache/[confighash]/records.json'),
  configHash: function(webpackConfig) {
    return require('node-object-hash')().hash(webpackConfig)
  },

  environmentFiles: [
    'package.json',
    'npm-shrinkwrap.json',
    'webpack.config.test.js',
    'webpack.config.js',
  ],
})

// webpack.config.hot-dev-server.js
new HardSourceWebpackPlugin({
  cacheDirectory: path.join(__dirname, 'test-cache/[confighash]'),
  recordsPath: path.join(__dirname, 'test-cache/[confighash]/records.json'),
  configHash: function(webpackConfig) {
    return require('node-object-hash')().hash(webpackConfig)
  },

  environmentFiles: [
    'package.json',
    'npm-shrinkwrap.json',
    'webpack.config.hot-dev-server.js',
    'webpack.config.js',
  ],
})

The config files for the 2 builds are consistently different, so the generated configHash should be consistently different.

Thanks for your work on this project!

Cannot read property '...\node_modules\extract-text-webpack-plugin' of null

Just found a small bug that I think can be easily fixed. I noticed that I was getting weird errors relating to extract-text-webpack-plugin that would seemingly come and go at random. I figured out that some types like

class Store {
    name =
    constructor(){
    }
}

or

const a = 1;
const a = 2;

would trip the extract-text-webpack-plugin error. Instead of letting webpack show the correct error, the app just crashes. I tracked down the error to index.js and found that module.meta was undefined, for some reason:

        if (module[extractTextNS] || module.meta[extractTextNS]) {
          moduleCache[identifier] = null;
          return;
        }

Checking for module.meta seems to fix the problem:

        if (module[extractTextNS] || module.meta && module.meta[extractTextNS]) {
          moduleCache[identifier] = null;
          return;
        }

Reading from cache does not work with Webpack v2.2 final

This is the error I get in advanced boilerplate right now:

Unhandled rejection TypeError: Class constructor RawModule cannot be invoked without 'new'
    at new HardModule (/Users/sebastian/Workspace/sebastian-software/advanced-boilerplate/node_modules/hard-source-webpack-plugin/lib/hard-module.js:18:13)

Doesn't work with UglifyJSPlugin

Continue discussion from MoOx/phenomic#442

I pushed my work here https://github.com/MoOx/phenomic/tree/hard-source-wp

Step to reproduce:

git clone https://github.com/MoOx/phenomic/tree/hard-source-wp
npm install && npm run docs

Trigger a build

cd docs && npm run build
The second time you trigger the build, you'll receive some error like this:

Error: phenomic.browser.bff0132abecff214e404.js from UglifyJs
TypeError: Cannot read property 'sections' of undefined
    at new SourceMapConsumer (/home/khoa/web/phenomic/node_modules/source-map/lib/source-map-consumer.js:20:19)
    at RawSource.CacheModule.source.node (/home/khoa/web/phenomic/node_modules/hard-source-webpack-plugin/index.js:79:7)
    at PrefixSource.node (/home/khoa/web/phenomic/node_modules/webpack-core/lib/PrefixSource.js:27:26)
    at /home/khoa/web/phenomic/node_modules/webpack-core/lib/ConcatSource.js:38:49
    at Array.map (native)
    at ConcatSource.node (/home/khoa/web/phenomic/node_modules/webpack-core/lib/ConcatSource.js:37:60)
    at /home/khoa/web/phenomic/node_modules/webpack-core/lib/ConcatSource.js:38:49
    at Array.map (native)
    at ConcatSource.node (/home/khoa/web/phenomic/node_modules/webpack-core/lib/ConcatSource.js:37:60)
    at /home/khoa/web/phenomic/node_modules/webpack-core/lib/ConcatSource.js:38:49
    at /home/khoa/web/phenomic/node_modules/webpack/lib/optimize/UglifyJsPlugin.js:135:31
    at Array.forEach (native)
    at Compilation.<anonymous> (/home/khoa/web/phenomic/node_modules/webpack/lib/optimize/UglifyJsPlugin.js:44:10)
    at Compilation.applyPluginsAsync (/home/khoa/web/phenomic/node_modules/tapable/lib/Tapable.js:71:13)
    at Compilation.<anonymous> (/home/khoa/web/phenomic/node_modules/webpack/lib/Compilation.js:567:9)
    at Compilation.next (/home/khoa/web/phenomic/node_modules/tapable/lib/Tapable.js:67:11)
    at ExtractTextPlugin.<anonymous> (/home/khoa/web/phenomic/docs/node_modules/extract-text-webpack-plugin/index.js:303:4)
    at Compilation.applyPluginsAsync (/home/khoa/web/phenomic/node_modules/tapable/lib/Tapable.js:71:13)
    at Compilation.<anonymous> (/home/khoa/web/phenomic/node_modules/webpack/lib/Compilation.js:563:8)
    at Compilation.next (/home/khoa/web/phenomic/node_modules/tapable/lib/Tapable.js:67:11) +2s

If you remove UglifyJSPlugin from the plugin stack (by removing the last commit), the error goes away, everything works at expected (as for as I know) and THE BUILD TIME SPEEDS UP 200% (20sec vs 40sec)

Webpack breaks on cached run

versions:

  • hard-source-webpack-plugin v0.3.4
  • webpack 2.2.0-rc.0 as well as 2.1.0-beta.25

When I run the project the first time, everything is fine, but on subsequent runs I get this crash until I clear the hard-source cache or disable the plugin:

/pathToProject/node_modules/webpack/lib/dependencies/HarmonyImportSpecifierDependency.js:31
	var importedModule = this.importDependency.module;
	                                          ^

TypeError: Cannot read property 'module' of undefined
    at HardHarmonyImportSpecifierDependency.HarmonyImportSpecifierDependency.getWarnings (/pathToProject/node_modules/webpack/lib/dependencies/HarmonyImportSpecifierDependency.js:31:44)
    at /pathToProject/node_modules/webpack/lib/Compilation.js:643:21
    at Array.forEach (native)
    at /pathToProject/node_modules/webpack/lib/Compilation.js:642:22
    at Array.forEach (native)
    at Compilation.reportDependencyWarnings (/pathToProject/node_modules/webpack/lib/Compilation.js:641:9)
    at Compilation.<anonymous> (/pathToProject/node_modules/webpack/lib/Compilation.js:507:8)
    at Array.forEach (native)
    at Compilation.finish (/pathToProject/node_modules/webpack/lib/Compilation.js:506:15)
    at /pathToProject/node_modules/webpack/lib/Compiler.js:471:16
    at /pathToProject/node_modules/tapable/lib/Tapable.js:189:11
    at Compilation.<anonymous> (/pathToProject/node_modules/webpack/lib/Compilation.js:454:10)
    at /pathToProject/node_modules/webpack/lib/Compilation.js:425:12
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickCallback (internal/process/next_tick.js:98:9)

Note that I run webpack with babel-register for the config files, not sure if that is related.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.