[Feature]: Support resolve the modules with hard-links about rspack HOT 17 CLOSED

@Boshen Check this out! https://github.com/colinaaa/rspack-hardlink-repro

from rspack.

Done in #5924

from rspack.

Relate pnpm hardlinks

Pnpm use symlinks from package node_modules to workspace node_modules, and use hardlinks from workspace node_modules files to global files.

Currently resolver can only deal with symlinks and get the node_modules of workspace. So this issue should occur when
there are multiple workspaces.

cc @Boshen

from rspack.

This isn't a bundler problem. Your codebase is no longer a monorepo, it's just different packages put inside the same codebase, and then you enabled compilation from source.

I don't have an exact solution, but you can try:

• resolve alias that module
• hoist the dependency to the root

from rspack.

there's no source or target role around paths sharing the same inode, thus i believe it's impossible to determine the resolved path without a rule.

in your plugin, the resolved path is determined by the first request. i think it is unstable.

from rspack.

This isn't a bundler problem. Your codebase is no longer a monorepo, it's just different packages put inside the same codebase, and then you enabled compilation from source.

I don't have an exact solution, but you can try:

• resolve alias that module
• hoist the dependency to the root

I agree that having such a huge monorepo with a bunch of pnpm workspaces inside is not a good way. But this helps us to collaborate between different teams efficiently. And our build tools based on webpack and esbuild works just fine with the plugin.

I don't think resolve.alias is a good solution since it will break the version constraint of the packages, it is likely to use a different version of a package with resolve.alias and cause bugs. It also requires all developers to be familiar with resolve and the dependencies(and the dependencies of dependencies) and write them down to the webpack.config.js, which is just impossible. This is not how you manage dependencies in such a huge project.

Not to mention hoist the dependency to the root. We would not use such a monorepo if we could merge the workspaces into one.

from rspack.

there's no source or target role around paths sharing the same inode, thus i believe it's impossible to determine the resolved path without a rule.

I'm not sure what do you mean by "there's no source or target role around paths sharing the same inode". Could you please explain that to me in detail? Thanks!

in your plugin, the resolved path is determined by the first request. i think it is unstable.

I agree that it is not 100% stable, but it is correct(at least from our experience).

from rspack.

I'm not sure what do you mean by "there's no source or target role around paths sharing the same inode". Could you please explain that to me in detail? Thanks!

as you know, a symbolic link represents a link file linking to another file, when a resolver resolves a request that points to a symbolic link with option like preferRealPath it gives the linked file's path.

while a "hard link" just represents a path to an inode node. in your case, there's two "hard links" linked to a same inode node. the two paths are of equal status thus a resolver cannot say "who it prefer".

so i do think, other than just "resolving paths with same inode number to one path", we need an explicit option to tell "which path the resolver should choose when resolving paths with same inode number to one path"

from rspack.

"use the first coming result" might not be a good idea as it will change.

it might cause problem when there're different loaders/plugins logic against the two paths sharing the same inode number. though it's rarely common in node_modules scene.

from rspack.

Another concern I have is that the term "inode" is closely associated with filesystems, so I'm not sure if it will function properly in filesystems without inode feature, such NTFS (though it has a similiar feature).

just to address some concerns, I know nothing about filesystem(_;).

also not intended to reject any possible feature

from rspack.

@colinaaa Can you provide a github test repository that replicates this project structure?

from rspack.

"use the first coming result" might not be a good idea as it will change.

it might cause problem when there're different loaders/plugins logic against the two paths sharing the same inode number. though it's rarely common in node_modules scene.

I agree that using the first coming result is not an ideal way. But we have to return results for each request, and there is no way to know if there are requests to come in the future. So the only way to make it work is to return the first coming result.

from rspack.

In my opinion, "use the first coming result" or inode check of hardlink are both business-specific logic that can be ensured by the business itself. They are not suitable to implement in OXC.

But in Rspack, the before_resolve hook cannot get the resolved path, and the after_resolve hook would generate full request, which leading to a substantial increase in transmission costs. Rspack does indeed need a low-cost way to modify the resolve result.

from rspack.

@Boshen Check this out! https://github.com/colinaaa/rspack-hardlink-repro

rspack-hardlink-repro  main ❯ ls -lh ./packages/a/node_modules/lodash
lrwxr-xr-x  1 bytedance  staff    62B Mar 12 16:09 ./packages/a/node_modules/lodash -> ../../../node_modules/.pnpm/[email protected]/node_modules/lodash

rspack-hardlink-repro  main ❯ ls -lh ./packages/b/node_modules/lodash
lrwxr-xr-x  1 bytedance  staff    62B Mar 12 16:09 ./packages/b/node_modules/lodash -> ../../../node_modules/.pnpm/[email protected]/node_modules/lodash

rspack-hardlink-repro  main ❯ ls -i ./packages/a/node_modules/lodash
162517384 ./packages/a/node_modules/lodash

rspack-hardlink-repro  main ❯  ls -i ./packages/b/node_modules/lodash
162517383 ./packages/b/node_modules/lodash

Is this replicated correctly?

from rspack.

rspack-hardlink-repro on  main is 📦 v1.0.0 via  v18.18.0
❯ /bin/ls -lh ./packages/a/node_modules/lodash
lrwxr-xr-x@ 1 bytedance  staff    62B Mar 12 14:43 ./packages/a/node_modules/lodash -> ../../../node_modules/.pnpm/[email protected]/node_modules/lodash

rspack-hardlink-repro on  main is 📦 v1.0.0 via  v18.18.0
❯ /bin/ls -lh ./packages/b/node_modules/lodash
lrwxr-xr-x  1 bytedance  staff    40B Mar 12 15:04 ./packages/b/node_modules/lodash -> .pnpm/[email protected]/node_modules/lodash

rspack-hardlink-repro on  main is 📦 v1.0.0 via  v18.18.0
❯ /bin/ls -i ./packages/b/node_modules/lodash/lodash.js
797938 ./packages/b/node_modules/lodash/lodash.js

rspack-hardlink-repro on  main is 📦 v1.0.0 via  v18.18.0
❮ /bin/ls -i ./packages/a/node_modules/lodash/lodash.js
797938 ./packages/a/node_modules/lodash/lodash.js

Did you run the pnpm --prefix packages/b install?

from rspack.

I can't get the same inode 🤔

from rspack.

@Boshen Sorry! Should run pnpm --prefix packages/b install before pnpm install :) README.md updated

from rspack.