Node.js Loaders Team Meeting 2022-02-15 about loaders HOT 23 CLOSED

I’ve added whatwg/html#5572 to the agenda. I put it first in case @guybedford doesn’t want to stay for the ambient loaders discussion. (@guybedford I assume you can make the meeting?)

from loaders.

Hello Node Loaders Team and community,

FYI, I've recently used the current Loaders API to build a new method of testing web code under Node.js, for the bleeding-edge SvelteKit web framework.

The create-esm-loader and node-esm-loader wrappers helpers help bring stability and sanity.

Details:
sveltejs/kit#19 (comment)

Loader sources:
https://github.com/brev/esm-loaders

So far, I have been able to achieve everything I need. Some of it is not ideal, but I think your planned work covers most of it already.

In case it's helpful. Thanks.

from loaders.

Perhaps we can add the import.meta.resolve discussion to this meeting? Let me know if that might be suitable.

from loaders.

from loaders.

@JakobJingleheimer this was one of the use cases for moving things off thread and Modules WG have talked about it for years, some edge cases do exist with doing so though but they would be a follow on not a blocker.

from loaders.

@GeoffreyBooth sure thing, done here: #68. thanks.

from loaders.

I see a lot of discussion in this thread is about whether resolve could be made synchronous without losing functionality.

Specifically to answer:

So it would seem that we could make resolve sync and aside from the churn of the breaking change, there would be no loss of functionality and no abandoned use cases. Does that hold true for the future, as well? Are there any potential use cases for an async resolve, that we want to preserve the capability to support in the future? If so, what are they?

I'd like to share a few places where I have found resolve being asynchronous to be useful (I've noted other possible approaches for all of these, along with potential downsides);

File system operations (checking existence, permissions)

It's possible to use the sync versions of these operations (accessSync / statSync / etc.) but of course these block the thread (which is currently shared with the main NodeJS process, so blocking it is not desirable when a dynamic import is triggered while the program is already running). Also using promise-based versions is better for reusability (e.g. my own approach shares code between a NodeJS loader and a webserver which performs pre-processing when serving resources for browser-based testing)

Loading js-based config

A common pattern is for tools to use a *.js file to get their configuration (see e.g. Rollup, WebPack). This config is loaded by simply loading the file using import or require. For import, this necessarily requires promises.

It would be possible to use top-level await to load config like this in advance, but that assumes that we always know where to look for them. In reality, we often want to look for config files once we know the path of the file we are loading from.

Other dynamic importing

For example a loader which decides to load typescript only when it finds a .ts file or Babel only when it finds a .jsx file. This could be achieved by loading it in the load method then assuming its presence in subsequent resolve calls, but that could also get quite complicated. (this is not an actual use-case for me but is something I considered)

Network requests

Again not a use-case for me, but I could imagine a loader which fetches resources over the network. In this case, an import like ./foo would need to perform a network request to find out what the file extension should be (ignoring this and just resolving to plain foo then handling everything in load would be problematic as an import for ./foo in one location and ./foo.js in another would not be recognised as being the same module)

from loaders.

from loaders.

Per current discussion in whatwg/html#5572 it seems like having a clear position for Node.js on this topic is important. (please don't comment in that thread now unless you absolutely have to - the point being to try to get some internal consensus on the matter)

from loaders.

I'm not sure we should add WHATWG issues to our meeting if they are not interested in Node's feedback from what I read in the comments.

from loaders.

it may help to have a unified Nodejs position as to whether we are able to make the resolver synchronous or not.

I think the question we need to answer is this: what do we lose, if anything, if we make the resolve hook synchronous? Currently Node’s internal version, defaultResolve, is async but doesn’t do anything asynchronous, and so it could be made async without consequence. Likewise, all of the examples of loaders in this repo and in the Node docs show sample resolve hooks that are defined as async functions but don’t do anything async, and could be changed into sync functions. I looked at the source for ts-node and the same seems to apply there too.

So it would seem that we could make resolve sync and aside from the churn of the breaking change, there would be no loss of functionality and no abandoned use cases. Does that hold true for the future, as well? Are there any potential use cases for an async resolve, that we want to preserve the capability to support in the future? If so, what are they?

from loaders.

Are there any potential use cases for an async resolve, that we want to preserve the capability to support in the future? If so, what are they?

Softwares that want to run a bunch of resolutions in parallel (TypeScript, Webpack, etc) would want resolvers to be async, that's at least part of why they reimplement the resolution themselves at the moment.

Although such softwares also tend to have many other reasons to implement their own resolution pipelines as they add features like types etc that would require to extend require.resolve in ways that aren't possible today, it would be nice if someday this wasn't necessary...

from loaders.

Currently Node’s internal version, defaultResolve, is async but doesn’t do anything asynchronous

That's not the case as of HTTP(S) Imports: defaultResolve() calls defaultGetFormatWithoutErrors(), and in the case of a network protocol, calls getHttpProtocolModuleFormat() (which is async, although it may settle immediately).

from loaders.

That’s not the case as of HTTP(S) Imports: defaultResolve() calls defaultGetFormatWithoutErrors(), and in the case of a network protocol, calls getHttpProtocolModuleFormat() (which is async, although it may settle immediately).

Okay, but it doesn’t have to, right? I thought load was responsible for definitively determining format, and resolve only provided it if it happened to already know what it was; but determining the format wasn’t something that resolve needed to do.

from loaders.

Actually, yes. Why is it doing that at all… 🤔

Ah, I know why: it was part of nodejs/node#40980. I think as it currently is, when the import is a network protocol, defaultResolve()'s format will be an unsettled promise.

Maybe when getHttpProtocolModuleFormat() is called with ignoreErrors set to true, getHttpProtocolModuleFormat() should just abort (returning null)?

from loaders.

Is this related to the possibility of making resolve sync from the main thread while keeping the implementation possibly async in the loader thread? I'm not intimately familiar with the topic, but I think there's a desire to expose all loader hooks as sync in the main thread to support loading of CJS files.

from loaders.

Is this related to the possibility of making resolve sync from the main thread while keeping the implementation possibly async in the loader thread?

Yes

I'm not intimately familiar with the topic, but I think there's a desire to expose all loader hooks as sync in the main thread to support loading of CJS files.

I don't understand—it already can load CJS file.

from loaders.

I thought someone on the loaders team was talking about allowing loaders to implement both resolve() and load() behind a require() call. So the main thread can make a blocking require() call, and it can be dispatched to the loader's thread, where resolve() and load() are allowed to asynchronously determine the module's source text, running a compiler in the loader thread, for example.

from loaders.

I don't recall that specifically, but if using the atomic wizardry mentioned recently, that sounds possible (not sure if it's a good idea or something that might be considered specifically supporting if it doesn't JustWork after loaders are moved off-thread).

from loaders.

I thought someone on the loaders team was talking about allowing loaders to implement both resolve() and load() behind a require() call.

This sounds like the effort a few years back of trying to make require of an ESM file possible. A PR was opened to achieve that via low-level C++ trickery, not atomics wizardry, so maybe this atomics approach might be a new way to potentially solve that problem? (The C++ version didn’t pass code review.)

from loaders.

For the record, this is the kind of implementation for the "syncify" part of the implementation (here, for babel-register):

https://github.com/babel/babel/blob/e77e3de402693a216088cb75cdf6a245cfd7d0f8/packages/babel-register/src/worker-client.js#L69-L85

from loaders.

to make require of an ESM file possible.

That seems like a different thing: blocking the main thread while simultaneously spinning the main thread's event loop to evaluate an ESM module.

Rather, this would be blocking the main thread waiting on the loader thread to return a path and source text to be executed. Execution would still happen synchronously on the main thread, and an error would be thrown if load() returned format: esm.

from loaders.

Do you mind submitting a PR to https://github.com/nodejs/loaders/blob/main/doc/resources.md to add these links?

from loaders.