Git Product home page Git Product logo

thread-loader's Introduction

npm node tests coverage discussion size

thread-loader

Runs the following loaders in a worker pool.

Getting Started

npm install --save-dev thread-loader

or

yarn add -D thread-loader

or

pnpm add -D thread-loader

Put this loader in front of other loaders. The following loaders run in a worker pool.

Loaders running in a worker pool are limited. Examples:

  • Loaders cannot emit files.
  • Loaders cannot use custom loader API (i. e. by plugins).
  • Loaders cannot access the webpack options.

Each worker is a separate node.js process, which has an overhead of ~600ms. There is also an overhead of inter-process communication.

Use this loader only for expensive operations!

Examples

webpack.config.js

module.exports = {
  module: {
    rules: [
      {
        test: /\.js$/,
        include: path.resolve('src'),
        use: [
          'thread-loader',
          // your expensive loader (e.g babel-loader)
        ],
      },
    ],
  },
};

with options

use: [
  {
    loader: 'thread-loader',
    // loaders with equal options will share worker pools
    options: {
      // the number of spawned workers, defaults to (number of cpus - 1) or
      // fallback to 1 when require('os').cpus() is undefined
      workers: 2,

      // number of jobs a worker processes in parallel
      // defaults to 20
      workerParallelJobs: 50,

      // additional node.js arguments
      workerNodeArgs: ['--max-old-space-size=1024'],

      // Allow to respawn a dead worker pool
      // respawning slows down the entire compilation
      // and should be set to false for development
      poolRespawn: false,

      // timeout for killing the worker processes when idle
      // defaults to 500 (ms)
      // can be set to Infinity for watching builds to keep workers alive
      poolTimeout: 2000,

      // number of jobs the poll distributes to the workers
      // defaults to 200
      // decrease of less efficient but more fair distribution
      poolParallelJobs: 50,

      // name of the pool
      // can be used to create different pools with elsewise identical options
      name: 'my-pool',
    },
  },
  // your expensive loader (e.g babel-loader)
];

prewarming

To prevent the high delay when booting workers it possible to warmup the worker pool.

This boots the max number of workers in the pool and loads specified modules into the node.js module cache.

const threadLoader = require('thread-loader');

threadLoader.warmup(
  {
    // pool options, like passed to loader options
    // must match loader options to boot the correct pool
  },
  [
    // modules to load
    // can be any module, i. e.
    'babel-loader',
    'babel-preset-es2015',
    'sass-loader',
  ]
);

Contributing

Please take a moment to read our contributing guidelines if you haven't yet done so.

CONTRIBUTING

License

MIT

thread-loader's People

Contributors

alexander-akait avatar cap-bernardito avatar clarkdo avatar dependabot[bot] avatar dfederm avatar ersachin3112 avatar evilebottnawi avatar fa93hws avatar iansu avatar janlent1 avatar jantimon avatar johnnyreilly avatar joshwiens avatar jpetitcolas avatar mesoptier avatar michael-ciniawsky avatar mistic avatar pycka avatar qnighy avatar sendilkumarn avatar sokra avatar wearymonkey avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

thread-loader's Issues

poolRespawn seems to have no (positive) effect

It seems that poolRespawn does not have the statet effect of improving the build time when running an incremental build. It seems that there even is a significant performance issue (equal to restarting a new thread-pool) when using it instead of a warmup and setting the poolTimeout to Infinity.

What would be the right configuration to get performance improvements on incremental build?

SIGINT causes hang, callbacks never hit

Sourcegraph is using thread-loader to build webpack packages. Sending Ctrl+C (SIGINT) to gulp while webpack is running causes the process to hang.

I think this is what is happening:

  1. thread-loader starts worker subprocesses and starts sending work to them.
  2. thread-loader receives work via pitch() from webpack, calls run(), sends it to subprocess, tries to read from the subprocess pipe.
  3. user sends SIGINT. because all of the processes are in the same process group, they all get the signal. the subprocesses quit.
  4. an 'end' event is sent on the pipe. we don't listen for that event in readBuffer.js. as a result, we never hit the workerPool.run() callback in index.js, or the callback from readBuffer(), and we're hung.

It is possible there is a race - I can reproduce this issue about 85% of the time. The other 15% I get a stack trace like this:

Error: This socket has been ended by the other party
    at Socket.writeAfterFIN [as write] (net.js:402:12)
    at PoolWorker.writeJson (/Users/kevin/src/github.com/sourcegraph/sourcegraph/node_modules/thread-loader/dist/WorkerPool.js:94:22)
    at PoolWorker.run (/Users/kevin/src/github.com/sourcegraph/sourcegraph/node_modules/thread-loader/dist/WorkerPool.js:74:12)
    at WorkerPool.distributeJob (/Users/kevin/src/github.com/sourcegraph/sourcegraph/node_modules/thread-loader/dist/WorkerPool.js:366:20)
    at /Users/kevin/src/github.com/sourcegraph/sourcegraph/node_modules/async/queue.js:10:5
    at Object.process (/Users/kevin/src/github.com/sourcegraph/sourcegraph/node_modules/async/internal/queue.js:175:17)
    at /Users/kevin/src/github.com/sourcegraph/sourcegraph/node_modules/async/internal/queue.js:82:19
    at Immediate.<anonymous> (/Users/kevin/src/github.com/sourcegraph/sourcegraph/node_modules/async/internal/setImmediate.js:27:16)
    at runCallback (timers.js:694:18)
    at tryOnImmediate (timers.js:665:5)
    at processImmediate (timers.js:647:5)
    at process.topLevelDomainCallback (domain.js:121:23)

Oddly, I cannot reproduce this issue when I load why-is-node-running - I get the stack trace and a quit 100% of the time instead of ~15% of the time. Perhaps this has to do with the overhead of the instrumentation causing the parent process to lose the race every time.

I have a patch which essentially involves sending an end event to onWorkerMessage when readBuffer fails. This still involves some ugly logging but at least the process terminates when you send a signal to it.

See also mafintosh/why-is-node-running#41.
See also sourcegraph/sourcegraph#186.

File loader fails to emit files when using thread-loader on node 9

We've upgraded our app to node v9 and we've a new issue that prevents us to use this loader. We can reproduce that error on webpack-dev-server and webpack. Hereunder is a log from webpack that builds our app.

[... the webpack compilation looks fine so far]
 [393] ./src/rootRouter.js 1.71 kB {79} [built]
 [394] ./src/cdppResources.js 963 bytes {79} [built]
 [399] ./resources/logo_regards_blue_black.png 77 bytes {79} [built]
 [400] ./resources/logo_regards_blue_white.png 77 bytes {79} [built]
 [401] ./resources/logo_regards_grey_black.png 77 bytes {79} [built]
 [402] ./resources/logo_regards_grey_white.png 77 bytes {79} [built]
 [403] ./webpack-config-front/src/conf/staticConfiguration.js 206 bytes {79} [built] [failed] [1 error]
    + 1621 hidden modules

ERROR in ./webpack-config-front/src/conf/staticConfiguration.js
Module build failed: Thread Loader (Worker 3)
File Loader
emitFile is required from module system
    at Object.loader (/app_to_build/node_modules/file-loader/dist/index.js:28:29)
 @ ./src/main.jsx 59:2-74
npm ERR! code ELIFECYCLE
npm ERR! errno 2

https://github.com/webpack-contrib/file-loader/blob/e73131fe1986cf930058e8f0868872a4a91d5f31/src/index.js#L10

If I evaluate in the console this.emitFile, it prints the real function several times but one time I get an undefined.

Our webpack rules (nothing special so far):

     {
          test: /\.jsx?$/, 
          // Exclude the DLL folder build from the transpilation 
          exclude: [/node_modules/, /dist/], 
          use: [ 
            'thread-loader', 
            'babel-loader?cacheDirectory', 
          ]
     },
     {
          test: /staticConfiguration(\.dev)?\.js$/,
          loader: 'file-loader',
          options: {
            name: 'staticConfiguration.js',
            outputPath: 'conf/',
          },
      },

When we stop using thread-loader the error disappear. And with node v6 or v7 there is no issue.

$ npm -v
5.8.0-next.0
$ node -v
v9.8.0
$ cat package.json
    "webpack": "3.11.0",
    "webpack-dev-server": "~2.11.0",
    "file-loader": "1.1.10",
    "thread-loader": "^1.1.5",

The loader does not apply config after search up the directory tree for configuration

  • Operating System: Windows 10
  • Node Version: v14.9.0
  • NPM Version: 6.14.8
  • webpack Version: 4.44.1
  • postcss-loader Version: 4.0.0

Expected Behavior

I have postcss.config.js file in the root of the project, my webpack.config.js is imported from ./webpack/app folder. I expect my postcss configuration to be used/applied correctly.

Actual Behavior

Running webpack ends up with console messages:

You did not set any plugins, parser, or stringifier. Right now, PostCSS does nothing. Pick plugins for your case on https://www.postcss.parts/ and use them in postcss.config.js.

And the postcss.config.js file is actually loaded - I've tried to add simple console.log('Im here') to it, and this message appears in my console, as expected. But, apparently, options exported from that file are not correctly sent to postcss.

Code

// webpack.config.js
{
...
module: {
		rules: [
			{
				test: /\.css$/,
				use: [
					MiniCssExtractPlugin.loader,
					{
						loader: 'css-loader',
						options: {
							importLoaders: 1
						}
					},
					{
						loader: 'postcss-loader'
					}
				]
			},
			{
				test: /\.less$/,
				include: APPROOT,
				use: [
					MiniCssExtractPlugin.loader,
					{
						loader: 'css-loader',
						options: {
							importLoaders: 2
						}
					},
					{
						loader: 'postcss-loader'
					},
					{
						loader: 'less-loader'
					}
				]
			}
		]
	}
...
}
// postcss.config.js
module.exports = {
	plugins: [
		'autoprefixer'
	]
};

Issues with Less-Loader

Are there any known issues with less loader and the thread loader?

I am getting the following error:

Module build failed: TypeError: Cannot read property 'bind' of undefined
at createWebpackLessPlugin (/Users/sresan.thevarajah/code/main/node_modules/less-loader/dist/createWebpackLessPlugin.js:37:49)
at getOptions (/Users/sresan.thevarajah/code/main/node_modules/less-loader/dist/getOptions.js:25:26)
at Object.lessLoader (/Users/sresan.thevarajah/code/main/node_modules/less-loader/dist/index.js:32:42)

What is undefined is the loadModule property.

My usage looks like
ExtractTextPlugin.extract({fallback: "style-loader", use: [{loader: "thread-loader" option: { name: "less"}}, "css-loader", "less-loader"]})

I am using two instances of extract text plugin as well.

process not closing with thread-loader >=2.0.1

I have an instance of webpack dev server running in a forked process from my main process, and I noticed the fork started sticking around after the main process is closed, when it should be getting closed also. My project uses the thread-loader for babel-loader. The behavior started when I upgraded from thread-loader 2.0.0, and I've narrowed it down to being introduced in 2.0.1, and the only commit in it is #51.

I'm still working on an example I can link to, but basically my code looks like this

//main process
  let child = fork(path.resolve(__dirname, './run-dev-server.js'), params);

  process.on('exit', function(){
    console.log('EXITING')
    child.kill();
  });

//run-dev-server.js

const compiler = webpack(config);
const server = new webpackDevServer(compiler, serverOptions);
server.listen(port, host);

//webpack rule in webpack.config.js
{
  test: /\.js$/,
  exclude: /node_modules|babel-helpers/,
  use: [
    'thread-loader',
    {
       loader: 'babel-loader',
       options: {
         cacheDirectory: true,
         extends: path.join(__dirname, '../.babelrc')
       }
    }
  ]
}

when i try to exit the main process, I see that EXITING gets printed, but now the forked process sticks around in the background still running webpack-dev-server. I might be able to change the signal I send to something that can't be ignored, but that seems like something I shouldn't have to do.

The behavior is not present in v2.0.0 (things work there), but 2.0.1 can reliably reproduce it. I'll try to get something I can link to though.

process.env.NODE_ENV can not pass to child process

when i use jest.run to run my tests, thread-loader can not pass process.env.NODE_ENV to child process. i print process.env.NODE_ENV in main process and child process, such as

image

explicit pass env,child process will get the correct process.env.NODE_ENV

this.worker = childProcess.spawn(process.execPath, [].concat(sanitizedNodeArgs).concat(workerPath, options.parallelJobs), {
    detached: true,
    stdio: ['ignore', 'pipe', 'pipe', 'pipe', 'pipe'],
    env: process.env
 });

does not find loader-runner when using Yarn PnP

Error: Cannot find module 'loader-runner'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:581:15)
    at Function.Module._load (internal/modules/cjs/loader.js:507:25)
    at Module.require (internal/modules/cjs/loader.js:637:17)
    at require (internal/modules/cjs/helpers.js:20:18)
    at Object.<anonymous> (/Users/druf/Library/Caches/Yarn/v3/npm-thread-loader-1.2.0-35dedb23cf294afbbce6c45c1339b950ed17e7a4/node_modules/thread-loader/dist/worker.js:11:21)
    at Module._compile (internal/modules/cjs/loader.js:689:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
    at Module.load (internal/modules/cjs/loader.js:599:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
    at Function.Module._load (internal/modules/cjs/loader.js:530:3)
internal/modules/cjs/loader.js:583
    throw err;
    ^

process not closing with thread-loader 2.1.3

the question is same as #56, but my version is 2.1.3, and my webpack configuration is like this:

...
const threadLoader = require('thread-loader');
...
const jsWorkerPool = {
   poolTimeout: 2000,
   workerParallelJobs: 200,
};
threadLoader.warmup({ jsWorkerPool }, ['vue-loader', 'babel-loader'])
...
 rules: [
      {
        test: /\.vue$/,
        use: [
          {
            loader: 'thread-loader',
            options: jsWorkerPool,
          },
          'vue-loader',
        ],
      },
      {
        test: /\.js$/,
        use: [
          {
            loader: 'thread-loader',
            options: jsWorkerPool,
          },
          'babel-loader',
        ],
        exclude: /node_modules/,
      },
      {
        test: /\.css$/,
        use: ['vue-style-loader', MiniCssExtractPlugin.loader, ...cssOptions],
      },
      {
        test: /\.scss$/,
        use: ['vue-style-loader', MiniCssExtractPlugin.loader, ...sassOptions],
      },
     ...
]

my vue version is 2.6.10, vue-loader version is 15.9.0, i do not know if the version of vue-loader and vue-style-loader affects it

Error using val-loader

Using [email protected] which is included through @vue/[email protected].

When using [email protected] in the project, the following error occurs:

Error trace
ERROR in ./app/javascript/evolve/IconAliases.js
Module build failed (from ./node_modules/@vue/cli-service/node_modules/thread-loader/dist/cjs.js):
Thread Loader (Worker 0)
Cannot read property 'context' of undefined
    at PoolWorker.fromErrorObj (node_modules/@vue/cli-service/node_modules/thread-loader/dist/WorkerPool.js:258:12)
    at Object.exec (node_modules/@vue/cli-service/node_modules/thread-loader/dist/worker.js:155:70)
    at Object.valLoader (node_modules/val-loader/dist/index.js:53:22)

After looking at the source code, I found that the distribution of 2.1.2 has the following code:

exec: (code, filename) => {
  const module = new _module2.default(filename, undefined);
  module.paths = _module2.default._nodeModulePaths(undefined.context); // eslint-disable-line no-underscore-dangle

which has undefined.context instead of this.context.

Thoughts on how to fix it? Also, any workaround to avoid hitting this line?

handle undefined `this.worker.stdio` correctly

Not sure how exactly to reproduce, but I get this half way through a compilation:

uncaughtException TypeError: Invalid attempt to destructure non-iterable instance
    at /Users/bazyli.brzoska/Projects/p/node_modules/thread-loader/dist/WorkerPool.js:7:585
    at new PoolWorker (/Users/bazyli.brzoska/Projects/p/node_modules/thread-loader/dist/WorkerPool.js:49:25)
    at WorkerPool.distributeJob (/Users/bazyli.brzoska/Projects/p/node_modules/thread-loader/dist/WorkerPool.js:326:23)
    at /Users/bazyli.brzoska/Projects/p/node_modules/async/queue.js:10:5
    at process (/Users/bazyli.brzoska/Projects/p/node_modules/async/internal/queue.js:165:17)
    at Immediate._onImmediate (/Users/bazyli.brzoska/Projects/p/node_modules/async/internal/setImmediate.js:27:16)
    at tryOnImmediate (timers.js:762:5)
    at processImmediate [as _immediateCallback] (timers.js:733:5)

Using Node 8.2.0-rc.1 and latest versions of both webpack (3.1) and thread-loader.

Add silent mode for production builds

I'm analyzing my webpack bundle size actually, and the profiler always adds 2 lines at the beginning of stats.json which are coming from the thread-loader.

Is there a way to make it silent, or could you add an option?

webpack 4 Support

Official Migration Guide for Loaders/Plugins

TODO

- [x] update usage of fileDependencies && contextDependencies to reflect new data types. These are {Set}'s now instead of an {Array} (https://github.com/webpack-contrib/thread-loader/blob/master/src/index.js#L24-L25)

First Contributors

💁 If you're stuck or in doubt about something always feel free to ask question in the issue thread and/or by pinning one of the @webpack-contrib/org-maintainers

webpack 1 complains thread-loader doesn't return a function

I'm not sure if you intend to support webpack 1. When using this loader with webpack 1, I see a warning that cjs.js doesn't return a function. This makes sense; it returns an object with the pitch method.

It should be possible for cjs.js to return something like this to keep webpack 1 happy:

module.exports = function() { throw 'should never be called' }
Object.assign(module.exports, require('./index'));

I've been able to implement this workaround myself and everything appears to be working.

The emitWarning method should retain an attribute that is a reference to the native WARNING

The emitWarning method should retain an attribute that is a reference to the native WARNING, just like WEBPACK
thread-loader link

// ...
emitWarning: (warning) => {
  writeJson({
    type: 'emitWarning',
    id,
    data: toErrorObj(warning),
  });
},

function toErrorObj(err) {
  return {
    message: err.message,
    details: err.details,
    stack: err.stack,
    hideStack: err.hideStack,
  };
}

webpack link

constructor(warning, { from = null } = {}) {
  let message = "Module Warning";

  if (from) {
    message += ` (from ${from}):\n`;
  } else {
    message += ": ";
  }

  if (warning && typeof warning === "object" && warning.message) {
    message += warning.message;
  } else if (warning) {
    message += String(warning);
  }

  super(message);

  this.name = "ModuleWarning";
  this.warning = warning;
  this.details =
    warning && typeof warning === "object" && warning.stack
      ? cleanUp(warning.stack, this.message)
      : undefined;

  Error.captureStackTrace(this, this.constructor);
}

EmitError method is best like this

Webpack watcher stops after compilation error when using thread loader

When using webpack in watch mode with thread loader, compilation errors seem to stop the watch. For an example, if I make a syntax error and save the file, even if I fix it later and save the file again, webpack won't recompile. This scenario works fines when I remove thread-loader from webpack config.

Here is a sample project that reproduces the scenario. Steps to reproduce:

  1. yarn && yarn watch
  2. after the watcher has started, introduce a syntax error in either index.js or Header.js and save the file
  3. fix the syntax error and save the file again

I've encountered this issue on two different projects, on two different machines (windows and linux) and on both node v6.11.2 and node v8.4.0.

The versions of the packages I'm using can be found below:

{
  "name": "curly-potato",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "scripts": {
    "watch": "webpack --watch"
  },
  "devDependencies": {
    "babel-core": "^6.26.0",
    "babel-loader": "^7.1.2",
    "thread-loader": "^1.1.0",
    "webpack": "^3.5.5"
  }
}

Minimal webpack config:

const path = require('path');

module.exports = {
    entry: path.resolve(__dirname, 'index.js'),
    output: {
        path: path.resolve(__dirname, 'build'),
        filename: 'index.js'
    },
    module: {
        rules: [
            {
                test: /\.js$/,
                use: [
                    'thread-loader',
                    'babel-loader'
                ]
            }
        ]
    }
};

Got " Cannot destructure property 'outputOptions' of 'loaderContext._compilation' as it is undefined." after upgrade to the latest version

  • Operating System: Any
  • Node Version: v16.0.0
  • NPM Version: 7.10.0
  • webpack Version:^5.45.1
  • css-loader Version:6.0.0

Expected Behavior

compile styles

Actual Behavior

got the error

Code

client config

            {
                test: /\.css$/,
                exclude: /node_modules/,
                include: includePaths,
                use: [
                    {
                        loader: MiniCssExtractPlugin.loader,
                        options: {
                            esModule: true,
                        },
                    },
                    getThreadLoader('client-css'),
                    {
                        loader: '@teamsupercell/typings-for-css-modules-loader',
                        options: {
                            disableLocalsExport: true,
                            verifyOnly: process.env.NODE_ENV === 'production',
                        },
                    },
                    {
                        loader: 'css-loader',
                        options: {
                            modules: {
                                localIdentName: '[hash:8]',
                            },
                            sourceMap: false,
                        },
                    },
                    {
                        loader: 'postcss-loader',
                        options: {
                            sourceMap: false,
                        },
                    },
                ],
            },

server config

            {
                test: /\.css$/,
                exclude: /node_modules/,
                include: includePaths,
                use: [
                    getThreadLoader('server-css'),
                    {
                        loader: 'css-loader',
                        options: {
                            modules: {
                                mode: 'local',
                                exportOnlyLocals: true,
                                localIdentName: '[hash:8]',
                            },
                        },
                    },
                    {
                        loader: 'postcss-loader',
                    },
                ],
            },

How Do We Reproduce?

upgrade the loader from 5.2.6 to the latest version

.sourceMap is an alias for .sourceMaps, cannot use both from thread-loader

Hi I am wondering if the below is from thread-loader, do please let me know what I can do to fix this

I have upgraded babel to below versions in reactjs application.
"babel-core": "7.0.0-bridge.0",
"@babel/core": "^7.2.2",
"@babel/polyfill": "^7.2.5",
"@babel/preset-env": "^7.3.1",
"@babel/preset-react": "^7.0.0",

After this my webpack build is failing with below error

ERROR in ./src/App.tsx Module build failed (from ./node_modules/thread-loader/dist/cjs.js): Thread Loader (Worker 0) .sourceMap is an alias for .sourceMaps, cannot use both

I am not sure where to remove sourceMap/s, i am not using any sourcemap in babel.config.js and I have removed devtool from webpack.config.js Can you please help?

Exception when tslint-loader reports error

When using tslint-loader and webpack is in watch mode, there's an exception when the tslint-loader reports an error, which crashes the process and stops the watch.

Exception:

 40% building modules 1/2 modules 1 active ...ientSuggestions\ancientSuggestions.tsC:\Users\David\Code\ClickerHeroesTracker\WebClient\node_modules\thread-loader\dist\WorkerError.js:14
  var originError = err.stack.split('\n').filter(function (line) {
                             ^

TypeError: Cannot read property 'split' of undefined
    at stack (C:\Users\David\Code\ClickerHeroesTracker\WebClient\node_modules\thread-loader\dist\WorkerError.js:14:30)
    at new WorkerError (C:\Users\David\Code\ClickerHeroesTracker\WebClient\node_modules\thread-loader\dist\WorkerError.js:44:19)
    at PoolWorker.fromErrorObj (C:\Users\David\Code\ClickerHeroesTracker\WebClient\node_modules\thread-loader\dist\WorkerPool.js:251:14)
    at PoolWorker.onWorkerMessage (C:\Users\David\Code\ClickerHeroesTracker\WebClient\node_modules\thread-loader\dist\WorkerPool.js:230:37)
    at C:\Users\David\Code\ClickerHeroesTracker\WebClient\node_modules\thread-loader\dist\WorkerPool.js:122:17
    at Socket.onChunk (C:\Users\David\Code\ClickerHeroesTracker\WebClient\node_modules\thread-loader\dist\readBuffer.js:32:9)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at Socket.Readable.read (_stream_readable.js:381:10)
    at flow (_stream_readable.js:761:34)
    at resume_ (_stream_readable.js:743:3)
    at _combinedTickCallback (internal/process/next_tick.js:80:11)
    at process._tickCallback (internal/process/next_tick.js:104:9)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] watch: `webpack --config ./webpack.config.js --progress --profile --watch`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] watch script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

Relevant package versions:

    "thread-loader": "^1.1.4",
    "tslint-loader": "^3.5.3",
    "webpack": "^3.11.0",

Subset of webpack.config.js

    config.module.rules = [
      {
        test: /\.ts$/,
        enforce: 'pre',
        loader: 'tslint-loader',
        exclude: path.resolve("./node_modules"),
        options: {
          configFile: path.resolve('./tslint.json'),
          tsConfigFile: path.resolve('./tsconfig.json'),
          emitErrors: true,
          failOnHint: true,
          typeCheck: true,
          fix: true,
        }
      },

// ...

    config.module.rules.push({
      test: /\.ts$/,
      loaders: [
        'cache-loader',
        {
          loader: 'thread-loader',
          options: {
            // there should be 1 cpu for the fork-ts-checker-webpack-plugin
            workers: require('os').cpus().length - 1
          }
        },
        {
          loader: 'ts-loader',
          options: {
            // IMPORTANT! use happyPackMode mode to speed-up compilation and reduce errors reported to webpack
            happyPackMode: true,
          }
        },
        'angular2-template-loader'
      ]
    });


Does it work with Webpack-dev-server?

It appears that thread-loader does not start when running Webpack-Dev-Server builds. Is this by design? When using webpack build, i get a notification for each thread running like this

ts-loader: Using [email protected] and C:\Source\
ts-loader: Using [email protected] and C:\Source\
ts-loader: Using [email protected] and C:\Source\
ts-loader: Using [email protected] and C:\Source\

But in webpack-dev-server build, no notification happens, and changing the number of workers property doesn't affect the build speed.

Versions:
webpack 3.5.6
node: v6.10.0
npm: 3.10.0

build currently failing

The build in Travis CI is currently failing with this error message:

(+) Error: Client request error: getaddrinfo ENOTFOUND api.nodesecurity.io api.nodesecurity.io:443

EPIPE: broken pipe and sometimes ERR_STREAM_DESTROYED when Sending Ctrl+C (SIGINT)

Hello guys,

Sometimes when starting a webpack compilation from the webpack node api, stopping it in the middle with a Ctrl+C (SIGINT) generates the following error output for every spawned worker:

{ [Error: EPIPE: broken pipe, write] errno: -32, code: 'EPIPE', syscall: 'write' }
Error [ERR_STREAM_DESTROYED]: Cannot call write after a stream was destroyed
    at doWrite (_stream_writable.js:406:19)
    at clearBuffer (_stream_writable.js:517:5)
    at WriteStream.Writable.uncork (_stream_writable.js:314:7)
    at /Users/tiago/test_project/node_modules/thread-loader/dist/worker.js:58:22
    at process._tickCallback (internal/process/next_tick.js:61:11)

Then the process just hangs until I send a new Ctrl+C. Any plans/idea to solve this @evilebottnawi @sokra ? It seems related to #33 and #36

babel-loader. exclude[0] must be a string/Function/RegExp

Hi, I'm trying to use thread-loader with babel-loader but I have this error:

".exclude[0] must be a string/Function/RegExp."
"Error: .exclude[0] must be a string/Function/RegExp.
at c:\Code\Development\Applications\node_modules@babel\core\lib\config\validation\option-assertions.js:180:15
at Array.forEach ()
at assertConfigApplicableTest (c:\Code\Development\Applications\node_modules@babel\core\lib\config\validation\option-assertions.js:178:11)
at c:\Code\Development\Applications\node_modules@babel\core\lib\config\validation\options.js:107:5
at Array.forEach ()
at validateNested (c:\Code\Development\Applications\node_modules@babel\core\lib\config\validation\options.js:83:21)
at validate (c:\Code\Development\Applications\node_modules@babel\core\lib\config\validation\options.js:74:10)
at loadPrivatePartialConfig (c:\Code\Development\Applications\node_modules@babel\core\lib\config\partial.js:66:50)
at Object.loadPartialConfig (c:\Code\Development\Applications\node_modules@babel\core\lib\config\partial.js:110:18)
at Object. (c:\Code\Development\Applications\node_modules\babel-loader\lib\index.js:144:26)"

I have an exclude option set on bable-loader, but even if i comment it out I have the error.

threadLoader.warmup(
    {
        // pool options, like passed to loader options
        // must match loader options to boot the correct pool
        name: 'babel-pool',
        poolRespawn: !isDev
    },
    [
        'babel-loader'
        // '@babel/preset-env',
        // '@babel/preset-typescript',
        // '@babel/plugin-proposal-object-rest-spread',
        // '@babel/plugin-transform-spread',
        // '@babel/plugin-syntax-dynamic-import',
        // '@babel/plugin-proposal-decorators',
        // '@babel/plugin-proposal-class-properties'
    ]
);
{
                test: /\.(j|t)s$/,
                loaders: [
                    {
                        loader: 'thread-loader',
                        options: {
                            name: 'babel-pool',
                            poolRespawn: !isDev
                        }
                    },
                    {
                        loader: 'babel-loader',
                        options: {
                            compact: !isDev,
                            inputSourceMap: true,
                            sourceMap: true,
                            presets: [
                                '@babel/preset-env',
                                '@babel/preset-typescript'
                            ],
                            plugins: [
                                '@babel/plugin-proposal-object-rest-spread',
                                '@babel/plugin-transform-spread',
                                '@babel/plugin-syntax-dynamic-import',
                                [
                                    '@babel/plugin-proposal-decorators',
                                    {
                                        legacy: true
                                    }
                                ],
                                '@babel/plugin-proposal-class-properties'
                            ],
                            exclude: [/src[\\\/]assets[\\\/]js[\\\/].*/]
                        }
                    }
                ]
            },

Emphasise the need to override poolTimeout when watching

Hi!

As mentioned here, when watching is enabled (ie the development config with webpack-dev-server) the poolTimeout option should be changed from its default of 500 ms to Infinity to avoid paying the overhead of worker startup on every rebuild.

This is hinted at in a code comment of the examples section of the docs, however it seems that people aren't noticing this. eg:
facebook/create-react-app#4346
jaredpalmer/razzle#587

As such I think it would make sense to call this out explicitly in the usage section of the docs.

Use compiler.inputFileSystem to make thread-loader work with memfs

I couldn't make thread-loader open an entry file from memory system as it yielding the following error: ENOENT: no such file or directory, open. I assume it has to do with the fact worker.js uses fs https://github.com/webpack-contrib/thread-loader/blob/master/src/worker.js#L3.

Would using _compiler.inputFileSystem or allowing an option to specify a in-memory file system in the loader allow that?

Repro below (create the package.json/index.js and run npm test):

package.json

{
  "name": "webpack-thread-loader-memfs",
  "version": "1.0.0",
  "main": "index.js",
  "scripts": {
    "test": "node index.js"
  },
  "dependencies": {
    "memfs": "^3.2.0",
    "thread-loader": "^2.1.3",
    "unionfs": "^4.4.0",
    "webpack": "^4.43.0",
    "webpack-cli": "^3.3.12"
  }
}

index.js

const fs = require("fs");
const path = require("path");
const webpack = require("webpack");
const { Union } = require("unionfs");
const { createFsFromVolume, Volume } = require("memfs");

const entry = path.join(__dirname, "entry.js");

const memoryFs = createFsFromVolume(new Volume());
memoryFs.join = path.join.bind(path);
const ufs = new Union();
ufs.use(fs).use(memoryFs);

memoryFs.mkdirpSync(__dirname);
memoryFs.writeFileSync(entry, `console.log('entry')`, "utf8");

const config = {
  entry,
  output: {
    filename: "bundle.js",
    path: __dirname,
  },
  mode: "production",
  module: {
    rules: [
      {
        test: /\.(js|jsx)$/,
        exclude: /node_modules/,
        use: [{ loader: require.resolve("thread-loader") }],
      },
    ],
  },
};

const compiler = webpack(config);
compiler.inputFileSystem = ufs;

compiler.run((error, stats) => {
  console.log(
    stats.toString({
      chunks: false,
      entrypoints: false,
      hash: false,
      version: false,
      modules: false,
      colors: true,
    })
  );
});

Output

ERROR in ./entry.js
Module build failed (from ./node_modules/thread-loader/dist/cjs.js):
Thread Loader (Worker 0)
ENOENT: no such file or directory, open '/Users/path-to-project/webpack-thread-loader-memfs/entry.js'
    at PoolWorker.fromErrorObj (/Users/path-to-project/webpack-thread-loader-memfs/node_modules/thread-loader/dist/WorkerPool.js:262:12)
    at /Users/path-to-project/webpack-thread-loader-memfs/node_modules/thread-loader/dist/WorkerPool.js:204:29
    at mapSeries (/Users/path-to-project/webpack-thread-loader-memfs/node_modules/neo-async/async.js:3625:14)
    at PoolWorker.onWorkerMessage (/Users/path-to-project/webpack-thread-loader-memfs/node_modules/thread-loader/dist/WorkerPool.js:170:35)
    at /Users/path-to-project/webpack-thread-loader-memfs/node_modules/thread-loader/dist/WorkerPool.js:152:14
    at Socket.onChunk (/Users/path-to-project/webpack-thread-loader-memfs/node_modules/thread-loader/dist/readBuffer.js:36:9)
    at Socket.emit (events.js:315:20)
    at Socket.Readable.read (_stream_readable.js:513:10)
    at Socket.read (net.js:623:39)
    at flow (_stream_readable.js:986:34)
    at emitReadable_ (_stream_readable.js:577:3)
    at processTicksAndRejections (internal/process/task_queues.js:83:21)

Worker process creating forks of itself

Given my understanding of how this loader works, if I pass it a workers: 3 option to limit the number of processes to 3, I would expect the process tree to look something like this (inspecting via pstree):

\_ node --max-old-space-size=8192 ./node_modules/.bin/webpack --color --config webpack/webpack.config.js --watch
|   \_ /usr/comp/nodenv/versions/10.16.0/bin/node /code/frontend/node_modules/thread-loader/dist/worker.js 50
|   \_ /usr/comp/nodenv/versions/10.16.0/bin/node /code/frontend/node_modules/thread-loader/dist/worker.js 50
|   \_ /usr/comp/nodenv/versions/10.16.0/bin/node /code/frontend/node_modules/thread-loader/dist/worker.js 50

However, what I see in reality is this:

\_ node --max-old-space-size=8192 ./node_modules/.bin/webpack --color --config webpack/webpack.config.js --watch
|   \_ /usr/comp/nodenv/versions/10.16.0/bin/node /code/frontend/node_modules/thread-loader/dist/worker.js 50
|   |   \_ /usr/comp/nodenv/versions/10.16.0/bin/node /code/frontend/node_modules/thread-loader/dist/worker.js 50
|   |   \_ /usr/comp/nodenv/versions/10.16.0/bin/node /code/frontend/node_modules/thread-loader/dist/worker.js 50
|   |   \_ /usr/comp/nodenv/versions/10.16.0/bin/node /code/frontend/node_modules/thread-loader/dist/worker.js 50
|   \_ /usr/comp/nodenv/versions/10.16.0/bin/node /code/frontend/node_modules/thread-loader/dist/worker.js 50
|   |   \_ /usr/comp/nodenv/versions/10.16.0/bin/node /code/frontend/node_modules/thread-loader/dist/worker.js 50
|   |   \_ /usr/comp/nodenv/versions/10.16.0/bin/node /code/frontend/node_modules/thread-loader/dist/worker.js 50
|   |   \_ /usr/comp/nodenv/versions/10.16.0/bin/node /code/frontend/node_modules/thread-loader/dist/worker.js 50
|   \_ /usr/comp/nodenv/versions/10.16.0/bin/node /code/frontend/node_modules/thread-loader/dist/worker.js 50
|       \_ /usr/comp/nodenv/versions/10.16.0/bin/node /code/frontend/node_modules/thread-loader/dist/worker.js 50
|       \_ /usr/comp/nodenv/versions/10.16.0/bin/node /code/frontend/node_modules/thread-loader/dist/worker.js 50
|       \_ /usr/comp/nodenv/versions/10.16.0/bin/node /code/frontend/node_modules/thread-loader/dist/worker.js 50

Is this expected? It looks like the worker process is being forked, but I've looked everywhere and don't see any code that should be forked it. Why don't I see only 3 processes? This is important to understand to adjust the settings to maximize build performance.

Update: the extra child process seem to go away pretty quickly. Maybe there's something going on inside node that I don't understand.

Issues with Postcss-Loader

Are there any known issues with postcss loader and the thread loader?

I am getting the following error:

ERROR in ./view/less/_export.less (./node_modules/thread-loader/dist/cjs.js??ref--6-2!./node_modules/css-loader/dist/cjs.js??ref--6-3!./node_modules/postcss-loader/src??ref--6-4!./node_modules/less-loader/dist/cjs.js??ref--6-5!./node_modules/style-resources-loader/lib??ref--6-6!./view/less/_export.less)
Module build failed (from ./node_modules/thread-loader/dist/cjs.js):
Thread Loader (Worker 11)
Cannot read property 'postcss' of null
    at PoolWorker.fromErrorObj (/Users/chenguangxiao/Workspace/DMP/dmp-web/node_modules/thread-loader/dist/WorkerPool.js:262:12)
    at /Users/chenguangxiao/Workspace/DMP/dmp-web/node_modules/thread-loader/dist/WorkerPool.js:204:29
    at mapSeries (/Users/chenguangxiao/Workspace/DMP/dmp-web/node_modules/neo-async/async.js:3625:14)
    at PoolWorker.onWorkerMessage (/Users/chenguangxiao/Workspace/DMP/dmp-web/node_modules/thread-loader/dist/WorkerPool.js:170:35)
    at readBuffer (/Users/chenguangxiao/Workspace/DMP/dmp-web/node_modules/thread-loader/dist/WorkerPool.js:152:14)
    at Socket.onChunk (/Users/chenguangxiao/Workspace/DMP/dmp-web/node_modules/thread-loader/dist/readBuffer.js:36:9)
    at Socket.emit (events.js:198:13)
    at Processor.normalize (/Users/chenguangxiao/Workspace/DMP/dmp-web/node_modules/postcss/lib/processor.js:157:13)
    at new Processor (/Users/chenguangxiao/Workspace/DMP/dmp-web/node_modules/postcss/lib/processor.js:52:25)
    at postcss (/Users/chenguangxiao/Workspace/DMP/dmp-web/node_modules/postcss/lib/postcss.js:55:10)
    at Promise.resolve.then.then (/Users/chenguangxiao/Workspace/DMP/dmp-web/node_modules/postcss-loader/src/index.js:140:12)
 @ ./view/less/_export.less 2:26-335 43:4-64:5 46:18-327
 @ multi ./view/less/_export.less ./src/index

My usage looks like


const threadLoader = require('thread-loader')
const cssWorkerPool = {
  workerParallelJobs: 2,
  poolTimeout: 2000,
}
threadLoader.warmup(cssWorkerPool, [
  'cache-loader',
  'style-loader',
  'css-loader',
  'postcss-loader',
  'less-loader',
  'style-resources-loader',
])


    {
        test: /\.(le|c)ss$/,
        use: [
          {
            loader: 'cache-loader',
          },
          {
            loader: 'style-loader',
          },
          {
            loader: 'thread-loader',
            options: cssWorkerPool,
          },
          {
            loader: 'css-loader',
            options: {
              url: true,
            },
          },
          {
            loader: 'postcss-loader',
            options: {
              plugins: [require('postcss-cssnext')],
            },
          },
          {
            loader: 'less-loader',
            options: { paths: [pathUtil.resolve('/', 'node_modules')] },
            // options: {
            //   javascriptEnabled: true
            // }
          },
          {
            loader: 'style-resources-loader',
            options: {
              patterns: pathUtil.resolve('view/less/pre', '_export.less'),
            },
          },
        ],
      },

serialization?

Stumbled on it by noticing that my regex's were failing, but raises a larger issue:

Screen Shot 2019-08-15 at 12 18 36 PM

There are plenty of non-JSON values used in loader options that currently break when using thread-loader (regex, dates, functions). It seems like an easy solution would be to use a more comprehensive serialization method, or allow users to plug their own in. This will allow thread-loader to work with even more loaders and seems like a quick win.

I think this is the line in question:

const messageBuffer = Buffer.from(JSON.stringify(data), 'utf-8');

Quick glance shows a few libraries that handle this, I'm sure you could either do this internally or find a better one as well:

Issue with css-loader when passing options with function prop

Environment

OS: Mac OS 10.14
css-loader: 2.1.0
thread-loader: 2.1.2
webpack: 4.28.4

Description
The getLocalIdent which is a custom function in the option of css-loader. It appear in the thread-loader option but not in the css-loader option. As below image shows:
image

loader configuration:

    'cache-loader',
    {
      loader: 'thread-loader',
      options: workerPool
    },
    {
      loader: 'css-loader',
      options: {
        context: path.resolve(__dirname, '..', '..', 'src'),
        modules: true,
        exportOnlyLocals,
        localIdentName: '[path]__[name]__[local]',
        getLocalIdent: (context, localIdentName, localName) => genLocalIdent(localName, context.resourcePath),
        importLoaders: 2
      }
    },
    'postcss-loader',
    {
      loader: 'sass-loader',
      options: {
        outputStyle: 'expanded',
        includePaths: [
          path.resolve(__dirname, '..', '..', 'src/component-library/_styles'),
          path.resolve(__dirname, '..', '..', 'src/css')
        ]
      }
    }

It works fine without thread-loader, so I think this should be thread-loader issue.

Please checkout sample repo and test it

Cannot use latest thread-loader with [email protected]

  • Operating System: Windows 10
  • Node Version: 12.13
  • NPM Version: yarn 1.12.3
  • webpack Version: 4.44.2
  • thread-loader Version: 3.0.0

Expected Behavior

No errors.

Actual Behavior

Error: Cannot read property 'hooks' of undefined occures when building project with exposed ts modules.

Code

// index.ts
function foo() {};

export default foo;

How Do We Reproduce?

Repo with example: https://github.com/MhMadHamster/webpack-expose-ts-modules

Initially opened issue in the expose-loader repository: webpack-contrib/expose-loader#115

the option of vue-loader will be missing when use thread-loader

when i use both of the vue-loader and thread-loader, for some reasons, the vue-loader is not directly in ./node_modules, so i have to manually set the one of the option values of the vue-loader which is called compile,like:
{ test: /\.vue$/, use: [ 'thread-loader', 'cache-loader', { loader: 'vue-loader', options: { compiler: require('./node_modules/@weiyi/hammer/vue-template-compiler') } } ] },
then there are some mistakes
Thread Loader (Worker 0) compiler.parseComponent is not a function at PoolWorker.fromErrorObj (D:\dingnan\vue-thread-loader\node_modules\thread-loader\dist\WorkerPool.js:262:12) at D:\dingnan\vue-thread-loader\node_modules\thread-loader\dist\WorkerPool.js:204:29 at mapSeries (D:\dingnan\vue-thread-loader\node_modules\neo-async\async.js:3625:14) at parse (D:\dingnan\vue-thread-loader\node_modules\@vue\component-compiler-utils\dist\parse.js:14:23) at Object.module.exports (D:\dingnan\vue-thread-loader\node_modules\vue-loader\lib\index.js:69:22) @ ./src/main.js 2:0-28 7:13-16
these mistakes above will not be appear when not using thread-loader,and i find the compile will be an empty object when using thread-loader, so i suppose it's a bug,i don't know how to do, can someone who can help me? thanks a lot! @ericclemmons @sapegin @MoOx

Support extensible loaderContext

There have been some issues related to the lack of loaderContext provided:

#10 (loadModule)
#25 (target, minimize, and resourceQuery)

We have a suite of custom webpack plugins and loaders which expose and consume custom APIs. Rather than forking the thread-loader to support our use cases, what do you think about making the IPC bridge extensible so that new capabilities can be added without bloating the "core" context?

Webpack 5's "getOptions" is missing

Expected Behavior

Loaders that use webpack 5's this.getOptions() API should work with thread-loader.

See webpack/webpack#10017

Actual Behavior

this.getOptions is undefined.

How Do We Reproduce?

Pick any loader that uses getOptions from loader-utils and modify it to use this.getOptions instead. That works when using webpack 5 without thread-loader, but fails when used behind thread-loader.

webpack build hangs with warmup and new cache API

  • Operating System: Alpine 3.12
  • Node Version: 14.16.1
  • NPM Version: yarn 1.22.5
  • webpack Version: 5.38.1
  • thread-loader Version: 3.0.4

Expected Behavior

The build completes.

Actual Behavior

The build hangs, it never terminates, it stops with webpack 5.38.1 compiled successfully in XXX ms but never displays the Done in XXXs. neither quits.

How Do We Reproduce?

I isolated that it comes from the warmup statement. I think the loader never exits because of the new cache API (the filesystem one from the v5) which surely does fulfill its caching purpose but as a consequence no job is being launched so the warmup workers are pending/zombies.
You can reproduce with that branch from your example: https://github.com/eytienne/thread-loader/tree/repro_build_hangs

add loaderContext._compiler, is important

awesome-typescript-loader need access loaderContext._compiler

TypeError: Cannot read property 'parentCompilation' of undefined
at Object.getRootCompiler (/home/czy/桌面/react-demo/node_modules/awesome-typescript-loader/src/instance.ts:67:15)
at compiler (/home/czy/桌面/react-demo/node_modules/awesome-typescript-loader/src/index.ts:42:23)
at Object.loader (/home/czy/桌面/react-demo/node_modules/awesome-typescript-loader/src/index.ts:16:12)
at LOADER_EXECUTION (/home/czy/桌面/react-demo/node_modules/loader-runner/lib/LoaderRunner.js:119:14)
at runSyncOrAsync (/home/czy/桌面/react-demo/node_modules/loader-runner/lib/LoaderRunner.js:120:4)
at iterateNormalLoaders (/home/czy/桌面/react-demo/node_modules/loader-runner/lib/LoaderRunner.js:229:2)
at /home/czy/桌面/react-demo/node_modules/loader-runner/lib/LoaderRunner.js:202:4

doc

Documentation Is:

  • Missing
  • Needed
  • Confusing
  • Not Sure?

Please Explain in Detail...

Your Proposal for Changes

Cannot read property 'createChildCompiler' of undefined

Original Issue at: vuejs/vue-cli#3192

Vue Cli Version

3.2.1

Environment info

Environment Info:

  System:
    OS: Windows 10
    CPU: (8) x64 Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz
  Binaries:
    Node: Not Found
    Yarn: Not Found
    npm: 6.5.0 - C:\Program Files\nodejs\npm.CMD
  Browsers:
    Edge: 42.17134.1.0
  npmPackages:
    @vue/babel-preset-app:  3.2.0
    @vue/cli-overlay:  3.2.0
    @vue/cli-plugin-babel: ^3.2.0 => 3.2.0
    @vue/cli-plugin-e2e-cypress: ^3.2.0 => 3.2.0
    @vue/cli-plugin-eslint: ^3.2.0 => 3.2.1
    @vue/cli-plugin-pwa: ^3.2.0 => 3.2.0
    @vue/cli-plugin-unit-jest: ^3.2.0 => 3.2.0
    @vue/cli-service: ^3.2.0 => 3.2.0
    @vue/cli-shared-utils:  3.2.0
    @vue/component-compiler-utils:  2.3.1
    @vue/eslint-config-airbnb: ^4.0.0 => 4.0.0
    @vue/preload-webpack-plugin:  1.1.0
    @vue/test-utils: ^1.0.0-beta.20 => 1.0.0-beta.27
    @vue/web-component-wrapper:  1.2.0
    babel-helper-vue-jsx-merge-props:  2.0.3
    babel-plugin-transform-vue-jsx:  4.0.1
    eslint-plugin-vue: ^5.0.0-0 => 5.0.0
    jest-serializer-vue:  2.0.2
    vue: ^2.5.17 => 2.5.21
    vue-eslint-parser:  2.0.3
    vue-hot-reload-api:  2.3.1
    vue-jest:  3.0.2
    vue-loader:  15.4.2
    vue-router: ^3.0.1 => 3.0.2
    vue-style-loader:  4.1.2
    vue-template-compiler: ^2.5.17 => 2.5.21
    vue-template-es2015-compiler:  1.6.0
    vuex: ^3.0.1 => 3.0.1
  npmGlobalPackages:
    @vue/cli: Not Found

Steps to reproduce

Step 1: Create vue.config.js with the following

module.exports = {
  parallel: true,
  chainWebpack: (config) => {
    config.module.rule('worker')
      .test(/\.worker\.js$/i)
      .use('worker-loader')
      .loader('worker-loader');
  },
};

Step 2: npm run serve

What is expected?

Should successfully build the project

What is actually happening?

ERROR Failed to compile with 1 errors 4:46:14 PM

error in ./src/workers/sample.worker.js

Module build failed (from ./node_modules/thread-loader/dist/cjs.js):
Thread Loader (Worker 0)
Cannot read property 'createChildCompiler' of undefined

at Object.pitch (xxxxx\node_modules\worker-loader\dist\index.js:83:39)

@ ./node_modules/cache-loader/dist/cjs.js??ref--12-0!./node_modules/thread-loader/dist/cjs.js!./node_modules/babel-loader/lib!./node_modules/cache-loader/dist/cjs.js??ref--0-0!./node_modules/vue-loader/
lib??vue-loader-options!./src/components/HelloWorld.vue?vue&type=script&lang=js& 45:0-51 71:33-45
@ ./src/components/HelloWorld.vue?vue&type=script&lang=js&
@ ./src/components/HelloWorld.vue
@ ./node_modules/cache-loader/dist/cjs.js??ref--12-0!./node_modules/thread-loader/dist/cjs.js!./node_modules/babel-loader/lib!./node_modules/cache-loader/dist/cjs.js??ref--0-0!./node_modules/vue-loader/
lib??vue-loader-options!./src/views/Home.vue?vue&type=script&lang=js&
@ ./src/views/Home.vue?vue&type=script&lang=js&
@ ./src/views/Home.vue
@ ./src/router.js
@ ./src/main.js
@ multi ./src/main.js

ERROR Build failed with errors.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] build: vue-cli-service build --modern
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.


Turning off parallel flag in vue.config.js successfully builds the project

module.exports = {
  parallel: false,
  chainWebpack: (config) => {
    config.module.rule('worker')
      .test(/\.worker\.js$/i)
      .use('worker-loader')
      .loader('worker-loader');
  },
};

webpack build hangs on Windows 10

Whenever I add thread-loader webpack hangs when building.

 {
    test: /\.jsx?$/,
    use: [
      { loader: "thread-loader" }, 
      { loader: "babel-loader" }
    ],
    exclude: /node_modules/
  }

I left it overnight just to test and webpack never finished, it should usually come back in less than 1 minute. I've also tried adjusting the number of workers and the poolTimeout to no avail. Removing thread-loader does fix the issue.

I'd like to add that my incremental build (aka webpack --watch) is faster with thread-loader and can keep building without issues.

I've also seen that HappyPack has some bugs either open or closed that show the same behavior. And my project also hangs if I replace thread-loader with happyPack. Apparently they solved their issues by fixing some fork() code. Though, it seems it didn't work for me. I'm wondering if there something on Windows that is preventing the process from returning.

Issue with `sass-loader` (this.getResolve is not a function)

Hey folks,

I've facing the following issue with sass-loader:

Error Details
ERROR in ./app.scss (./node_modules/thread-loader/dist/cjs.js!./node_modules/css-loader/dist/cjs.js!./node_modules/sass-loader/dist/cjs.js!./app.scss)
Module build failed (from ./node_modules/thread-loader/dist/cjs.js):
Thread Loader (Worker 0)
this.getResolve is not a function
    at PoolWorker.fromErrorObj (/Users/ngryman/Projects/Open Source/Code/thread-sass-loaders-bug/node_modules/thread-loader/dist/WorkerPool.js:262:12)
    at /Users/ngryman/Projects/Open Source/Code/thread-sass-loaders-bug/node_modules/thread-loader/dist/WorkerPool.js:204:29
    at mapSeries (/Users/ngryman/Projects/Open Source/Code/thread-sass-loaders-bug/node_modules/neo-async/async.js:3625:14)
    at PoolWorker.onWorkerMessage (/Users/ngryman/Projects/Open Source/Code/thread-sass-loaders-bug/node_modules/thread-loader/dist/WorkerPool.js:170:35)
    at readBuffer (/Users/ngryman/Projects/Open Source/Code/thread-sass-loaders-bug/node_modules/thread-loader/dist/WorkerPool.js:152:14)
    at Object.loader (/Users/ngryman/Projects/Open Source/Code/thread-sass-loaders-bug/node_modules/sass-loader/dist/index.js:52:26)
 @ ./app.scss 1:14-155

When I remove thread-loader from the loaders chain, it works correctly. Here is a very simple repo to reproduce that issue: https://github.com/ngryman/thread-sass-loaders-bug.

I've also already surfaced that issue in the sass-loader repo as I'm not sure which package is buggy.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.