[Bug] virtual module trigger extra compilation in webpack 5 about webpack-virtual-modules HOT 5 CLOSED

Took several days to bisect the problem. The problem lies in the fact that WatchFileSystem will scan the filesystem after a compilation has finished, and if it finds any differences, it triggers a rebuild. This scan obviously does not go through the cached input file system, so it reports that the virtual modules "missing in initial scan" (sample). It boils down to the following steps:

1. Watchpack goes through all "watchers" it keeps (1, 2), in our case, this eventually calls _onRemove method on virtual files.
2. Collected changes are buffered and emitted with an "aggregated" event.
3. The key is that watchFileSystem always calls the callback upon receiving the event, effectively triggers another watch cycle (with virtual modules purged from inputFileSystem so the virtual module plugin adds it back).

My idea to deal with this is to steal the watchFileSystem and roll a new one that filters out virtual files before proceeding. If none of the removals come from actual files, then we can safely ignore this event.

This involves precisely mimicing what Watchpack does (which slightly varies across versions of Webpack). The drawback is that we definitely need separate implementations for Webpack 3, 4, and 5, and the way is not future-proof.

Forgive me that writing clean TS is time-consuming for me. Moreover, under Webpack 5.64.1, when logging things down, I see some nonsense removals (like node_modules/babel-loader.js/package.json) that still trigger the rebuilds 2~3 times, possibly related to buggy cache, but I am running out of time budgets on studying this further. I will be grateful if someone can turn this idea into a PR.

Update: Did some quick tests, my fork works for me 5.60~5.64.0, and 5.64.1 with --no-cache.

from webpack-virtual-modules.

Update: I have worked out a more robust, Proxy-based solution that treats the original WatchFileSystem as an abstract object. Should be more resilient to breaking changes in Webpack and Watchpack.

Code: https://github.com/andy0130tw/webpack-virtual-modules/blob/24d823f3133da9f397224cdfedd39be341546b21/src/wfs.ts

from webpack-virtual-modules.

@larixer Sure. I have opened a PR.

from webpack-virtual-modules.

@andy0130tw Awesome, do you still have no bandwidth for the pull request?

from webpack-virtual-modules.

Closing the issue as stale, please retest on latest webpack-virtual-modules and reopen with reproduction steps if the issue still exists.

from webpack-virtual-modules.