un-ts / synckit Goto Github PK

Usage

const dohMathSync = api.doMathAsync.bind(api)

main.mjs

import { Worker } from 'worker_threads'
import * as Comlink from 'comlink'
import { Endpoint } from 'synckit'

const worker = new Worker(new URL('./worker.mjs', import.meta.url))
const api = Comlink.wrap(Endpoint(worker))

const dohMathSync = api.doMathAsync.bind(api)

worker.mjs

import { parentPort } from 'worker_threads'
import * as Comlink from 'comlink'
import { Endpoint } from 'synckit'

const api = {
  async doMathAsync() {
    return 4
  }
}

Comlink.expose(api, Endpoint(parentPort))

Related

GoogleChromeLabs/comlink#642

Hi, this came up when using eslint with eslint-import-resolver-typescript on multiple Windows systems:

Error [ERR_UNSUPPORTED_ESM_URL_SCHEME]: Only URLs with a scheme in: file, data are supported by the default ESM loader. On Windows, absolute paths must be valid file:// URLs. Received protocol 'c:'
    at new NodeError (node:internal/errors:400:5)
    at throwIfUnsupportedURLScheme (node:internal/modules/esm/resolve:1055:11)
    at defaultResolve (node:internal/modules/esm/resolve:1135:3)
    at nextResolve (node:internal/modules/esm/loader:163:28)
    at ESMLoader.resolve (node:internal/modules/esm/loader:842:30)
    at ESMLoader.getModuleJob (node:internal/modules/esm/loader:424:18)
    at ESMLoader.import (node:internal/modules/esm/loader:525:22)
    at initializeLoader (node:internal/process/esm_loader:75:58)
    at loadESM (node:internal/process/esm_loader:90:11)
    at runMainESM (node:internal/modules/run_main:55:21)

Did not matter if using VSCode-bundled Node 16 or Node 18.13.0. In the UI it just makes the save progress hang forever:

Same eslint config is working on Linux. We spent 2 days debugging this on Windows and finally found the issue:

diff --git a/lib/index.cjs b/lib/index.cjs
index 5df1cca065bc0ea42cec7f0ce15c65731f8635b0..e59be1661422dc768648ac166beaadb66e7f4d7b 100644
--- a/lib/index.cjs
+++ b/lib/index.cjs
@@ -176,7 +176,7 @@ const setupTsRunner = (workerPath, { execArgv, tsRunner }) => {
       execArgv = ["-r", pnpApiPath, ...execArgv];
       const pnpLoaderPath = path__default["default"].resolve(pnpApiPath, "../.pnp.loader.mjs");
       if (isFile(pnpLoaderPath)) {
-        execArgv = ["--experimental-loader", pnpLoaderPath, ...execArgv];
+        execArgv = ["--experimental-loader", node_url.pathToFileURL(pnpLoaderPath), ...execArgv];
       }
     }
   }

This makes it work. I can confirm this on the command line, if I call node with --experimental-loader with an absolute Windows path, the ERR_UNSUPPORTED_ESM_URL_SCHEME error is thrown, while it works when supplying it as a file url.

Is that something that can be fixed in the code or is there any other way to make it work?

The sample in the readme involves two files. Does synckit offer a simple function to return a synchronous version of an async function? Something like what desync offers with const readFile = deasync(myAsyncFn)

Hello, thank you for this wonderful library! I'm currently using this in a Babel plugin and when testing with Jest it will hang and never complete the test if there's any sort of syntax error. Is there a way to surface these errors and kill the process if that happens? For example, you can just comment random code in runAsWorker and it should get stuck:

runAsWorker(async () => {
 abcd
})

https://github.com/fisker/make-synchronized

Idea from @fisker

https://github.com/swc-project/swc-node

https://npmjs.com/web-worker

Is there any way or example to provide JS function to runAsWorker function instead of file path? Here is what I am trying to do:

// this.__latest is an async function
    latest = () => {
        if (this._data) {
            // return this._latest[0];
            return createSyncFn(runAsWorker(this.__latest.bind(this)))();
        }
        return null;
    }

https://www.npmjs.com/package/synckit?activeTab=dependents

https://npm-stat.com/charts.html?package=synckit&from=2023-01-01

The performance is unacceptable

@JounQin

// main.ts
import { createSyncFn, TsRunner } from 'synckit'

const syncFn = createSyncFn(require.resolve('./worker.ts'), {
    tsRunner: TsRunner.TSX,
})

syncFn(1)

// worker.ts
import { runAsWorker } from 'synckit'

console.log('worker started')

runAsWorker(async function main(...args) {
    console.log(args)
    return 'hello'
})

After starting with yarn tsx main.ts, work is not executed.

prepare

$ npm remove ts-node
$ npm exec tsc
$ ls worker.ts worker.js
worker.ts worker.js
$ grep worker runner.ts
const fn = createSyncFn(require.resolve("./worker.js"))

actual

$ node runner.js
(hangs forever)
^C

expected

$ npm i ts-node
$ node runner.js
OK

$ npm remove ts-node
$ mv worker.ts worker.ts.bak
$ node runner.js
OK

seems like

Wrong extension priority, calling ts without tsrunner leads to syntax error.

Having no reply, so hangs forever.

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Location: package.json
Error type: The renovate configuration file contains some invalid settings
Message: Configuration option 'packageRules[0].node' should be a json object, Invalid configuration option: author, Invalid configuration option: commitlint, Invalid configuration option: es2015, Invalid configuration option: eslintConfig, Invalid configuration option: eslintIgnore, Invalid configuration option: fesm5, Invalid configuration option: files, Invalid configuration option: jest, Invalid configuration option: keywords, Invalid configuration option: license, Invalid configuration option: main, Invalid configuration option: module, Invalid configuration option: name, Invalid configuration option: packageRules[1].tslib, Invalid configuration option: packageRules[1].uuid, Invalid configuration option: packageRules[2].@1stg/lib-config, Invalid configuration option: packageRules[2].@changesets/changelog-github, Invalid configuration option: packageRules[2].@changesets/cli, Invalid configuration option: packageRules[2].@types/jest, Invalid configuration option: packageRules[2].@types/node, Invalid configuration option: packageRules[2].@types/uuid, Invalid configuration option: packageRules[2].clean-publish, Invalid configuration option: packageRules[2].npm-run-all, Invalid configuration option: packageRules[2].ts-expect, Invalid configuration option: packageRules[2].ts-jest, Invalid configuration option: packageRules[2].ts-node, Invalid configuration option: packageRules[2].type-coverage, Invalid configuration option: packageRules[2].typescript, Invalid configuration option: packageRules[2].yarn-deduplicate, Invalid configuration option: prettier, Invalid configuration option: remarkConfig, Invalid configuration option: renovate, Invalid configuration option: scripts, Invalid configuration option: typeCoverage, Invalid configuration option: types, Invalid configuration option: version, The "node" object can only be configured at the top level of a config but was found inside "packageRules[0]"

https://github.com/rx-ts/synckit/blob/d25b60da23d8e2fd3b65627ad4560e935132555c/src/index.ts#L87

For if I want to pass in custom flags like --conditions, --experimental-loader, etc.

When i want to use worker_thread to handle buffer, i would like to use transferList to move the buffer

port.postMessage(value[, transferList])

I can not do this in this lib

So that it can be changed across levels of packages.

Integrate with worker_threads inspired by https://github.com/lambci/sync-threads.

According to the research based on the issue at jimmywarting/await-sync#1 (comment)

Worker threads in Node.js send their standard IO to the main thread for processing. When synchronous operations are implemented, the blocking of these operations can cause the logs of these worker threads to be synchronized and output only after the computation results are completed.

Similarly, following the implementation of whatwg-workers, by rewriting the write methods of stdout and stderr streams within worker threads, it is possible to allow the standard input and standard error within the worker threads to be directly written into the streams.

// caused by https://github.com/nodejs/node/blob/0b3fcfcf351fba9f29234976eeec4afb09ae2cc0/lib/internal/worker/io.js#L360
// override process write to make stdout and stderr work in worker thread
function overrideProcess4WorkThread() {
  for (const [i, stream] of [process.stdout, process.stderr].entries()) {
    // process.stdout.fd === 1 & process.stderr.fd === 2 . In Worker threads, this field does not exist.
    const fd = i + 1
    const streamOriginWritev = stream._writev?.bind(stream)
    stream._writev = (chunks, cb) => {
      if (chunks.length === 0) {
        return
      }
      const chunk = chunks.pop()!
      // type-coverage:ignore-next-line -- we can't control
      fs.write(fd, chunk.chunk as string, null, chunk.encoding, err => {
        if (err) cb(err)
        else if (chunks.length === 0) cb()
        else streamOriginWritev?.(chunks, cb)
      })
    }
  }
}

and test demo output

If allowed, I'm glad to raise a pr

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Repository problems

These problems occurred while renovating this repository. View logs.

• WARN: Using npm packages for Renovate presets is now deprecated. Please migrate to repository-based presets instead.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

• chore(deps): update dependency deasync to ^0.1.30
• chore(deps): update dependency lint-staged to ^15.2.5
• fix(deps): update dependency tslib to ^2.6.3
• chore(deps): update dependency eslint to ^8.57.0
• chore(deps): update dependency node to v18.20.3
• chore(deps): update dependency node-gyp to ^10.1.0
• chore(deps): update dependency simple-git-hooks to ^2.11.1
• chore(deps): update dependency typescript to ^5.4.5
• chore(deps): update yarn to v4.2.2
• chore(deps): update codecov/codecov-action action to v4
• chore(deps): update dependency @commitlint/cli to v19
• chore(deps): update dependency @pkgr/rollup to v6
• chore(deps): update dependency eslint to v9
• chore(deps): update dependency execa to v9
• chore(deps): update dependency node to v20 (node, @types/node)
• 🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update dependency @changesets/cli to ^2.27.5
• chore(deps): update dependency @types/jest to ^29.5.12
• chore(deps): update dependency ts-jest to ^29.1.4
• fix(deps): update dependency @pkgr/core to ^0.1.1
• chore(deps): update dependency @commitlint/cli to ^18.6.1
• chore(deps): update dependency @swc-node/register to ^1.9.1
• chore(deps): update dependency make-synchronized to ^0.2.9
• chore(deps): update dependency prettier to ^3.3.1
• chore(deps): update dependency tsx to ^4.12.1
• chore(deps): update dependency type-coverage to ^2.29.0
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

https://github.com/egoist/esbuild-register
https://github.com/esbuild-kit/tsx

GoogleChromeLabs/comlink#642 (comment)

https://github.com/pyodide/synclink

follow evanw/esbuild@ecae25d

more info: ota-meshi/eslint-plugin-json-schema-validator#220