Git Product home page Git Product logo

synckit's Introduction

synckit

GitHub Actions Codecov type-coverage npm GitHub Release

Conventional Commits Renovate enabled JavaScript Style Guide Code Style: Prettier

Perform async work synchronously in Node.js using worker_threads with first-class TypeScript and Yarn P'n'P support.

TOC

Usage

Install

# yarn
yarn add synckit

# npm
npm i synckit

API

// runner.js
import { createSyncFn } from 'synckit'

// the worker path must be absolute
const syncFn = createSyncFn(require.resolve('./worker'), {
  tsRunner: 'tsx', // optional, can be `'ts-node' | 'esbuild-register' | 'esbuild-runner' | 'tsx'`
})

// do whatever you want, you will get the result synchronously!
const result = syncFn(...args)
// worker.js
import { runAsWorker } from 'synckit'

runAsWorker(async (...args) => {
  // do expensive work
  return result
})

You must make sure, the result is serializable by Structured Clone Algorithm

Types

export interface GlobalShim {
  moduleName: string
  /**
   * `undefined` means side effect only
   */
  globalName?: string
  /**
   * 1. `undefined` or empty string means `default`, for example:
   * ```js
   * import globalName from 'module-name'
   * ```
   *
   * 2. `null` means namespaced, for example:
   * ```js
   * import * as globalName from 'module-name'
   * ```
   *
   */
  named?: string | null
  /**
   * If not `false`, the shim will only be applied when the original `globalName` unavailable,
   * for example you may only want polyfill `globalThis.fetch` when it's unavailable natively:
   * ```js
   * import fetch from 'node-fetch'
   *
   * if (!globalThis.fetch) {
   *   globalThis.fetch = fetch
   * }
   * ```
   */
  conditional?: boolean
}

Options

  1. execArgv same as env SYNCKIT_EXEC_ARGV
  2. globalShims: Similar like env SYNCKIT_GLOBAL_SHIMS but much more flexible which can be a GlobalShim Array, see GlobalShim's definition for more details
  3. timeout same as env SYNCKIT_TIMEOUT
  4. transferList: Please refer Node.js worker_threads documentation
  5. tsRunner same as env SYNCKIT_TS_RUNNER

Envs

  1. SYNCKIT_EXEC_ARGV: List of node CLI options passed to the worker, split with comma ,. (default as []), see also node docs
  2. SYNCKIT_GLOBAL_SHIMS: Whether to enable the default DEFAULT_GLOBAL_SHIMS_PRESET as globalShims
  3. SYNCKIT_TIMEOUT: timeout for performing the async job (no default)
  4. SYNCKIT_TS_RUNNER: Which TypeScript runner to be used, it could be very useful for development, could be 'ts-node' | 'esbuild-register' | 'esbuild-runner' | 'swc' | 'tsx', 'ts-node' is used by default, make sure you have installed them already

TypeScript

ts-node

If you want to use ts-node for worker file (a .ts file), it is supported out of box!

If you want to use a custom tsconfig as project instead of default tsconfig.json, use TS_NODE_PROJECT env. Please view ts-node for more details.

If you want to integrate with tsconfig-paths, please view ts-node for more details.

esbuild-register

Please view esbuild-register for its document

esbuild-runner

Please view esbuild-runner for its document

swc

Please view @swc-node/register for its document

tsx

Please view tsx for its document

Benchmark

It is about 50x faster than sync-threads but 10x slower than native for reading the file content itself 1000 times during runtime, and 40x faster than sync-threads but 10x slower than native for total time on my personal MacBook Pro with 64G M1 Max.

And it's almost 5x faster than deasync but requires no native bindings or node-gyp.

See benchmark.cjs and benchmark.esm for more details.

You can try it with running yarn benchmark by yourself. Here is the benchmark source code.

Sponsors

1stG RxTS UnTS
1stG Open Collective backers and sponsors RxTS Open Collective backers and sponsors UnTS Open Collective backers and sponsors

Backers

Backers

1stG RxTS UnTS
1stG Open Collective backers and sponsors RxTS Open Collective backers and sponsors UnTS Open Collective backers and sponsors

Changelog

Detailed changes for each release are documented in CHANGELOG.md.

License

MIT Β© JounQin@1stG.me

synckit's People

Contributors

coderaiser avatar fisker avatar github-actions[bot] avatar jounqin avatar krossekrabbe avatar lgtm-com[bot] avatar noahnu avatar onigoetz avatar renovate[bot] avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

synckit's Issues

ERR_UNSUPPORTED_ESM_URL_SCHEME

Hi, this came up when using eslint with eslint-import-resolver-typescript on multiple Windows systems:

Error [ERR_UNSUPPORTED_ESM_URL_SCHEME]: Only URLs with a scheme in: file, data are supported by the default ESM loader. On Windows, absolute paths must be valid file:// URLs. Received protocol 'c:'
    at new NodeError (node:internal/errors:400:5)
    at throwIfUnsupportedURLScheme (node:internal/modules/esm/resolve:1055:11)
    at defaultResolve (node:internal/modules/esm/resolve:1135:3)
    at nextResolve (node:internal/modules/esm/loader:163:28)
    at ESMLoader.resolve (node:internal/modules/esm/loader:842:30)
    at ESMLoader.getModuleJob (node:internal/modules/esm/loader:424:18)
    at ESMLoader.import (node:internal/modules/esm/loader:525:22)
    at initializeLoader (node:internal/process/esm_loader:75:58)
    at loadESM (node:internal/process/esm_loader:90:11)
    at runMainESM (node:internal/modules/run_main:55:21)

Did not matter if using VSCode-bundled Node 16 or Node 18.13.0. In the UI it just makes the save progress hang forever:

Screenshot (3)

Same eslint config is working on Linux. We spent 2 days debugging this on Windows and finally found the issue:

diff --git a/lib/index.cjs b/lib/index.cjs
index 5df1cca065bc0ea42cec7f0ce15c65731f8635b0..e59be1661422dc768648ac166beaadb66e7f4d7b 100644
--- a/lib/index.cjs
+++ b/lib/index.cjs
@@ -176,7 +176,7 @@ const setupTsRunner = (workerPath, { execArgv, tsRunner }) => {
       execArgv = ["-r", pnpApiPath, ...execArgv];
       const pnpLoaderPath = path__default["default"].resolve(pnpApiPath, "../.pnp.loader.mjs");
       if (isFile(pnpLoaderPath)) {
-        execArgv = ["--experimental-loader", pnpLoaderPath, ...execArgv];
+        execArgv = ["--experimental-loader", node_url.pathToFileURL(pnpLoaderPath), ...execArgv];
       }
     }
   }

This makes it work. I can confirm this on the command line, if I call node with --experimental-loader with an absolute Windows path, the ERR_UNSUPPORTED_ESM_URL_SCHEME error is thrown, while it works when supplying it as a file url.

Is that something that can be fixed in the code or is there any other way to make it work?

syntax errors hang server

Hello, thank you for this wonderful library! I'm currently using this in a Babel plugin and when testing with Jest it will hang and never complete the test if there's any sort of syntax error. Is there a way to surface these errors and kill the process if that happens? For example, you can just comment random code in runAsWorker and it should get stuck:

runAsWorker(async () => {
 abcd
})

[feat] override stdout & stderr in worker_thread for debug worker

According to the research based on the issue at jimmywarting/await-sync#1 (comment)

Worker threads in Node.js send their standard IO to the main thread for processing. When synchronous operations are implemented, the blocking of these operations can cause the logs of these worker threads to be synchronized and output only after the computation results are completed.

Similarly, following the implementation of whatwg-workers, by rewriting the write methods of stdout and stderr streams within worker threads, it is possible to allow the standard input and standard error within the worker threads to be directly written into the streams.

// caused by https://github.com/nodejs/node/blob/0b3fcfcf351fba9f29234976eeec4afb09ae2cc0/lib/internal/worker/io.js#L360
// override process write to make stdout and stderr work in worker thread
function overrideProcess4WorkThread() {
  for (const [i, stream] of [process.stdout, process.stderr].entries()) {
    // process.stdout.fd === 1 & process.stderr.fd === 2 . In Worker threads, this field does not exist.
    const fd = i + 1
    const streamOriginWritev = stream._writev?.bind(stream)
    stream._writev = (chunks, cb) => {
      if (chunks.length === 0) {
        return
      }
      const chunk = chunks.pop()!
      // type-coverage:ignore-next-line -- we can't control
      fs.write(fd, chunk.chunk as string, null, chunk.encoding, err => {
        if (err) cb(err)
        else if (chunks.length === 0) cb()
        else streamOriginWritev?.(chunks, cb)
      })
    }
  }
}

and test demo output
image

If allowed, I'm glad to raise a pr

Starting with tsx does not execute correctly.

@JounQin

// main.ts
import { createSyncFn, TsRunner } from 'synckit'

const syncFn = createSyncFn(require.resolve('./worker.ts'), {
    tsRunner: TsRunner.TSX,
})

syncFn(1)
// worker.ts
import { runAsWorker } from 'synckit'

console.log('worker started')

runAsWorker(async function main(...args) {
    console.log(args)
    return 'hello'
})

After starting with yarn tsx main.ts, work is not executed.

Can i use transferList in this lib?

When i want to use worker_thread to handle buffer, i would like to use transferList to move the buffer

port.postMessage(value[, transferList])

I can not do this in this lib

[bug] hangs forever

prepare

$ npm remove ts-node
$ npm exec tsc
$ ls worker.ts worker.js
worker.ts worker.js
$ grep worker runner.ts
const fn = createSyncFn(require.resolve("./worker.js"))

actual

$ node runner.js
(hangs forever)
^C

expected

$ npm i ts-node
$ node runner.js
OK
$ npm remove ts-node
$ mv worker.ts worker.ts.bak
$ node runner.js
OK

seems like

Wrong extension priority, calling ts without tsrunner leads to syntax error.

Having no reply, so hangs forever.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Repository problems

These problems occurred while renovating this repository. View logs.

  • WARN: Using npm packages for Renovate presets is now deprecated. Please migrate to repository-based presets instead.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

  • chore(deps): update dependency node to v18.20.2
  • chore(deps): update dependency node-gyp to ^10.1.0
  • chore(deps): update dependency simple-git-hooks to ^2.11.1
  • chore(deps): update dependency typescript to ^5.4.5
  • chore(deps): update yarn to v4.2.1
  • chore(deps): update codecov/codecov-action action to v4
  • chore(deps): update dependency @commitlint/cli to v19
  • chore(deps): update dependency @pkgr/rollup to v6
  • chore(deps): update dependency eslint to v9
  • chore(deps): update dependency node to v20 (node, @types/node)
  • πŸ” Create all rate-limited PRs at once πŸ”

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

Detected dependencies

github-actions
.github/workflows/ci.yml
  • actions/checkout v4
  • actions/setup-node v4
  • codecov/codecov-action v3
.github/workflows/codeql.yml
  • actions/checkout v4
  • github/codeql-action v3
  • github/codeql-action v3
  • github/codeql-action v3
.github/workflows/pkg-size.yml
  • actions/checkout v4
  • pkg-size/action v1
.github/workflows/release.yml
  • actions/checkout v4
  • actions/setup-node v4
npm
package.json
  • @pkgr/core ^0.1.0
  • tslib ^2.6.2
  • @1stg/common-config ^10.0.0
  • @changesets/changelog-github ^0.5.0
  • @changesets/cli ^2.27.1
  • @commitlint/cli ^18.4.3
  • @pkgr/rollup ^5.0.0
  • @swc-node/register ^1.6.8
  • @types/jest ^29.5.11
  • @types/node ^20.10.6
  • clean-pkg-json ^1.2.0
  • concurrently ^8.2.2
  • cross-env ^7.0.3
  • deasync ^0.1.29
  • esbuild-register ^3.5.0
  • esbuild-runner ^2.2.2
  • eslint ^8.56.0
  • execa ^8.0.1
  • jest ^29.7.0
  • lint-staged ^15.2.0
  • make-synchronized ^0.0.3
  • node-gyp ^10.0.1
  • patch-package ^8.0.0
  • prettier ^3.1.1
  • simple-git-hooks ^2.9.0
  • sync-threads ^1.0.1
  • ts-expect ^1.3.0
  • ts-jest ^29.1.1
  • ts-node ^10.9.2
  • tsx ^4.7.0
  • type-coverage ^2.27.1
  • typescript ^5.3.3
  • node ^14.18.0 || >=16.0.0
  • prettier ^3.1.1
  • yarn 4.0.2
nvm
.nvmrc
  • node 18.18

  • Check this box to trigger a request for Renovate to run again on this repository

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Location: package.json
Error type: The renovate configuration file contains some invalid settings
Message: Configuration option 'packageRules[0].node' should be a json object, Invalid configuration option: author, Invalid configuration option: commitlint, Invalid configuration option: es2015, Invalid configuration option: eslintConfig, Invalid configuration option: eslintIgnore, Invalid configuration option: fesm5, Invalid configuration option: files, Invalid configuration option: jest, Invalid configuration option: keywords, Invalid configuration option: license, Invalid configuration option: main, Invalid configuration option: module, Invalid configuration option: name, Invalid configuration option: packageRules[1].tslib, Invalid configuration option: packageRules[1].uuid, Invalid configuration option: packageRules[2].@1stg/lib-config, Invalid configuration option: packageRules[2].@changesets/changelog-github, Invalid configuration option: packageRules[2].@changesets/cli, Invalid configuration option: packageRules[2].@types/jest, Invalid configuration option: packageRules[2].@types/node, Invalid configuration option: packageRules[2].@types/uuid, Invalid configuration option: packageRules[2].clean-publish, Invalid configuration option: packageRules[2].npm-run-all, Invalid configuration option: packageRules[2].ts-expect, Invalid configuration option: packageRules[2].ts-jest, Invalid configuration option: packageRules[2].ts-node, Invalid configuration option: packageRules[2].type-coverage, Invalid configuration option: packageRules[2].typescript, Invalid configuration option: packageRules[2].yarn-deduplicate, Invalid configuration option: prettier, Invalid configuration option: remarkConfig, Invalid configuration option: renovate, Invalid configuration option: scripts, Invalid configuration option: typeCoverage, Invalid configuration option: types, Invalid configuration option: version, The "node" object can only be configured at the top level of a config but was found inside "packageRules[0]"

[feat] comlink endpoint adapter

Usage

const dohMathSync = api.doMathAsync.bind(api)

main.mjs

import { Worker } from 'worker_threads'
import * as Comlink from 'comlink'
import { Endpoint } from 'synckit'

const worker = new Worker(new URL('./worker.mjs', import.meta.url))
const api = Comlink.wrap(Endpoint(worker))

const dohMathSync = api.doMathAsync.bind(api)

worker.mjs

import { parentPort } from 'worker_threads'
import * as Comlink from 'comlink'
import { Endpoint } from 'synckit'

const api = {
  async doMathAsync() {
    return 4
  }
}

Comlink.expose(api, Endpoint(parentPort))

Related

GoogleChromeLabs/comlink#642

Directly accept JS function rather than worker path

Is there any way or example to provide JS function to runAsWorker function instead of file path? Here is what I am trying to do:

// this.__latest is an async function
    latest = () => {
        if (this._data) {
            // return this._latest[0];
            return createSyncFn(runAsWorker(this.__latest.bind(this)))();
        }
        return null;
    }

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.