Comments (23)
Than you for reporting this issue. I will try to figure out how to fix this problem.
from onnxruntime.
#20991 makes default ESM import to use non-dynamic-import and hope this change may fix this problem. PR is still in progress
from onnxruntime.
Hey, I also need this. I am struggling with importing this version. So far I have been importing ONNX using
import * as ort from "https://cdn.jsdelivr.net/npm/onnxruntime-web/dist/esm/ort.webgpu.min.js"
. However, when I change toimport * as ort from "https://cdn.jsdelivr.net/npm/[email protected]/dist/esm/ort.webgpu.min.js"
it seems not to have an.../esm/
folder. Do you know why that is and how to import it then?
just replace .../esm/ort.webgpu.min.js
to .../ort.webgpu.min.mjs
should work. If you are also using service worker, use ort.webgpu.bundle.min.mjs
instead of ort.webgpu.min.mjs
.
from onnxruntime.
So it turns out to be that dynamic import (ie. import()
) and top-level await
is not supported in current service worker. I was not expecting that import()
is banned in SW.
Currently, the WebAssembly factory (wasm-factory.ts) uses dynamic import to load the JS glue. This does not work in service worker. A few potential solutions are also not available:
- Modifying it to import statement: won't work, because the JS glue includes top-level
await
. - Using
importScripts
: won't work, because the JS glue is ESM - Using
eval
: won't work; same toimportScripts
I am now trying to make a JS bundle that does not use dynamic import for usage of service worker specifically. Still working on it
from onnxruntime.
Thanks, I appreciate your efforts around this. It does seem like some special-case bundle will need to be built after all; you might need iife
or umd
for the bundler output format
from onnxruntime.
Thanks, I appreciate your efforts around this. It does seem like some special-case bundle will need to be built after all; you might need
iife
orumd
for the bundler output format
I have considered this option. However, Emscripten does not offer an option to output both UMD(IIFE+CJS) & ESM for JS glue (emscripten-core/emscripten#21899). I have to choose either. I choose the ES6 format output for the JS glue, because of a couple of problems when import UMD from ESM, and import()
is a standard way to import ESM from both ESM and UMD. ( Until I know its not working in service worker by this issue)
I found a way to make ORT web working, - yes this need the build script to do some special handling. And this will only work for ESM, because the JS glue is ESM and it seems no way to import ESM from UMD in service worker.
from onnxruntime.
@ggaabe Could you please help to try import * as ort from “./ort.webgpu.bundle.min.js”
from version 1.19.0-dev.20240604-3dd6fcc089 ?
from onnxruntime.
@fs-eire my project is dependent on transformersjs, which imports onnxruntime webgpu backend like this here:
https://github.com/xenova/transformers.js/blob/v3/src/backends/onnx.js#L24
Is this the right usage? In my project I've added this to my package.json to resolve onnx-runtime to this new version though the issue is still occurring:
"overrides": {
"onnxruntime-web": "1.19.0-dev.20240604-3dd6fcc089"
}
from onnxruntime.
Maybe also important: The same error is still occurring in same spot in inference session in the onnx package and not from transformersjs. Do I need to add a resolver for onnxruntime-common as well?
from onnxruntime.
Hi @fs-eire, is the newly-merged fix in a released build I can try?
from onnxruntime.
Please try 1.19.0-dev.20240612-94aa21c3dd
from onnxruntime.
@fs-eire EDIT: Nvm the comment I just deleted, that error was because I didn't set the webpack target
to webworker
.
However, I'm getting a new error now (progress!):
Error: no available backend found. ERR: [webgpu] RuntimeError: null function or function signature mismatch
from onnxruntime.
Update: Found the error is happening in here:
onnxruntime/js/common/lib/backend-impl.ts
Lines 83 to 86 in fff68c3
For some reason the webgpu backend.init promise is rejecting due to the null function or function signature mismatch
error. This is much further along than we were before though.
from onnxruntime.
Update: Found the error is happening in here:
onnxruntime/js/common/lib/backend-impl.ts
Lines 83 to 86 in fff68c3
For some reason the webgpu backend.init promise is rejecting due to the
null function or function signature mismatch
error. This is much further along than we were before though.
Could you share me the reproduce steps?
from onnxruntime.
@fs-eire You'll need to run the webGPU setup in a chrome extension.
- You can use my code I just published here: https://github.com/ggaabe/extension
- run
npm install
- run
npm run build
- open the chrome manage extensions
![Screenshot 2024-06-14 at 9 37 14 AM](https://private-user-images.githubusercontent.com/5441185/339803607-e9ac0f03-1379-49b4-b5d4-a1bfe1842da8.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjAwMDU4NzYsIm5iZiI6MTcyMDAwNTU3NiwicGF0aCI6Ii81NDQxMTg1LzMzOTgwMzYwNy1lOWFjMGYwMy0xMzc5LTQ5YjQtYjVkNC1hMWJmZTE4NDJkYTgucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDcwMyUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA3MDNUMTExOTM2WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZmE2MGVhYTIwZWI4YTZiZGE1YTdkNGY5OTMwNWZjYzM2OWQxYTdkOWVhNjZiNjM5MzRlZTI4NzlhNTBlZTUxNiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.83M8AZXCqr6yHFSLeUI7YnjiCakJYXfeXZgTeaeEMdM)
- load unpacked
![Screenshot 2024-06-14 at 9 37 52 AM](https://private-user-images.githubusercontent.com/5441185/339803816-73da6d0a-de1c-4e4f-9eaa-24a2936b886b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjAwMDU4NzYsIm5iZiI6MTcyMDAwNTU3NiwicGF0aCI6Ii81NDQxMTg1LzMzOTgwMzgxNi03M2RhNmQwYS1kZTFjLTRlNGYtOWVhYS0yNGEyOTM2Yjg4NmIucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDcwMyUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA3MDNUMTExOTM2WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YmQ2MWNhZWM3Nzk2ZWZlMzEzMDc3YTRiY2ExZDQzNWFhMWViMWUwMjhkZjU2YWZjZmE5YTkwZTRjY2Q1NzIxZCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.yVz1Na4iKCdaBdooHQtIoRFx4JJM-EUbz2jFHGFSp8U)
- select the
build
folder from the repo. - open the
AI WebGPU Extension
extension - type some text in the text input. it will load Phi-3 mini and after finishing loading this error will occur
- if you view the extension in the extension in the extension manager and select the "Inspect views
service worker" link before opening the extension it will bring up an inspection window to view the errors as they occur. A little "errors" bubble link also shows up here after they occur.
![Screenshot 2024-06-14 at 9 40 48 AM](https://private-user-images.githubusercontent.com/5441185/339804747-8f6cd5a4-d9df-4c84-8436-51b289cfcc47.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjAwMDU4NzYsIm5iZiI6MTcyMDAwNTU3NiwicGF0aCI6Ii81NDQxMTg1LzMzOTgwNDc0Ny04ZjZjZDVhNC1kOWRmLTRjODQtODQzNi01MWIyODljZmNjNDcucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDcwMyUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA3MDNUMTExOTM2WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9Y2RiNGVkOTc2NWFkZmU1YzQwYjU0ODA0MjgxNzVhYmVkNTA5YWU2YWQ5Y2EwNjUxYWUwMDI0MmI4OGY0ZTM2NSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.14B2LwYkOGNCaQPhZBicqOTJ9L-aQgjjyVlOaQXeFGE)
- You will need to click the "Refresh" button on the extension in the extension manager to rerun the error because it does not attempt reloading the model after the first attempt until another refresh
from onnxruntime.
@ggaabe I did some debug on my box and made some fixes -
-
Changes to ONNXRuntime Web:
#21073 is created to make sure the web assembly file can be loaded correctly when
env.wasm.wasmPaths
is not specified. -
Changes to https://github.com/ggaabe/extension
ggaabe/extension#1 need to be made to the extension example, to make it load the model correctly. Please note:
- The onnxruntime-web version need to be updated to consume changes from (1) (after it get merged and published for dev channel)
- There are still errors in background.js, which looks like incorrect params passed to
tokenizer.apply_chat_template()
. However, the WebAssembly is initialized and the model loaded successfully.
-
Other issues:
- Transformerjs overrides
env.wasm.wasmPaths
to a CDN URL internally. At least for this example, we don't want this behavior so we need to reset it toundefined
to keep the default behavior. - Multi-threaded CPU EP is not supported because
Worker
is not accessible in service worker. Issue tracking: whatwg/html#8362
- Transformerjs overrides
from onnxruntime.
Awesome, thank you for your thoroughness in explaining this and tackling this head on. Is there a dev channel version I can test out?
from onnxruntime.
Not yet. Will update here once it is ready.
from onnxruntime.
sorry to bug; is there any dev build number? wasn't sure how often a release runs
from onnxruntime.
sorry to bug; is there any dev build number? wasn't sure how often a release runs
Please try 1.19.0-dev.20240621-69d522f4e9
from onnxruntime.
@fs-eire I'm getting one new error:
ort.webgpu.bundle.min.mjs:6 Uncaught (in promise) Error: The data is not on CPU. Use `getData()` to download GPU data to CPU, or use `texture` or `gpuBuffer` property to access the GPU data directly.
at get data (ort.webgpu.bundle.min.mjs:6:13062)
at get data (tensor.js:62:1)
I pushed the code changes to my repo and fixed the call to the tokenizer. To reproduce, just type 1 letter in the chrome extension’s text input and wait
from onnxruntime.
Hey, I also need this. I am struggling with importing this version. So far I have been importing ONNX using
import * as ort from "https://cdn.jsdelivr.net/npm/onnxruntime-web/dist/esm/ort.webgpu.min.js"
.
However, when I change to import * as ort from "https://cdn.jsdelivr.net/npm/[email protected]/dist/esm/ort.webgpu.min.js"
it seems not to have an .../esm/
folder. Do you know why that is and how to import it then?
from onnxruntime.
@fs-eire I'm getting one new error:
ort.webgpu.bundle.min.mjs:6 Uncaught (in promise) Error: The data is not on CPU. Use `getData()` to download GPU data to CPU, or use `texture` or `gpuBuffer` property to access the GPU data directly. at get data (ort.webgpu.bundle.min.mjs:6:13062) at get data (tensor.js:62:1)
I pushed the code changes to my repo and fixed the call to the tokenizer. To reproduce, just type 1 letter in the chrome extension’s text input and wait
This may be a problem of transformerjs. Could you try whether this problem happen in a normal page? If so, can report the issue to transformerjs. If it's only happening in service worker, I can take a closer look
from onnxruntime.
Related Issues (20)
- [Documentation] phi-3 vision tutorial lacks samples for languages that are actually used for desktop development. HOT 1
- [Build] Build python interface for Onnxruntime-qnn on aarch64 Linux HOT 11
- [Build] ‘struct onnxruntime::ProviderHostCPU’ has no member named ‘UpsampleBase__AdjustOutputSizeAsPolicy’ when CONTRIB ops are disabled. HOT 15
- Initialization crash using OnnxRuntime 17.0 (previously working on 16.3)
- [Web] `Error: using ceil() in shape computation is not yet supported for AveragePool` HOT 2
- ORT 1.18 crashes on exit after using Cuda EP to run inference on a specific model HOT 2
- [Transformers Optimizer] CLIP-ViT encoder attention not getting fused HOT 4
- [Documentation] How Configure CUDA 12.* and cuDNN for GPU with ONNX Runtime and C# on Windows 11 HOT 6
- Onnx Model run failed in a loop HOT 2
- [Mobile] QNN HTP Backend Setup on Android Device
- onnxruntime-gpu not working with my gpu / setup HOT 9
- onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasnt able to be loaded. HOT 2
- [TensorRT EP] OOM (RAM) when loading ONNX model HOT 7
- Inference result different between cuda and cpu HOT 1
- [Mobile] QNN failed to finalize QNN graph for attention layer
- SIGSEGV on CoreMLExecutionProvider when using dynamic batch
- [Mobile] Pixel 7a (Google edge tpu) NNAPI runtime exception unordered_map::at: key not found when setting USE_FP16 HOT 10
- [Feature Request] Support linux/arm64 wheels for onnxruntime-gpu HOT 1
- [Training] ImportError: cannot import name 'PropagateCastOpsStrategy' from 'onnxruntime.capi._pybind_state' HOT 3
- How to use IOBinding with torch_directml and onnxruntime in Python API? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from onnxruntime.