Whisper in web-llm with WebGPU? about web-llm HOT 4 OPEN

great suggestion, yes this is something that we can push for

from web-llm.

@tqchen my ultimate goal would be to get it run the most efficient way on android edge device.

Although there is already a solution in the onnx framework onnx framework, based on the recent merge, but i am not sure when it will be usable on android.

There were some who tried with GPU delegates, but no success yet.

Any idea how one could solve it on the edge (Android) device?

from web-llm.

There is also a demo of Whisper running via WebAssembly in that repo. https://github.com/ggerganov/whisper.cpp/tree/master/examples/talk.wasm

from web-llm.

There is also a demo of Whisper running via WebAssembly in that repo. https://github.com/ggerganov/whisper.cpp/tree/master/examples/talk.wasm

Yes, it runs on CPU. I hope, that with a GPU version one could reach real time inference.

from web-llm.