Comments (6)
Gotcha. Now it all makes sense. I'll be keeping my eye on v3. Thanks for the patience and good luck with all the work ahead.
from transformers.js.
Hi there 👋 error code 6 is usually related to out-of-memory issues. Can you provide the code you are running (as well as the model being used)?
from transformers.js.
Model I was trying to use was whisper medium.
Here's the full code for the react component:
import React, { useRef, useState, useEffect } from 'react';
import { MediaRecorder, register } from 'extendable-media-recorder';
import { connect } from 'extendable-media-recorder-wav-encoder';
import { pipeline, env, read_audio } from '@xenova/transformers';
env.allowLocalModels=false;
interface AutomaticSpeechRecognitionOutput {
text?: string;
}
const AudioInput: React.FC = () => {
const [isRecording, setIsRecording] = useState<boolean>(false);
const [audioBlob, setAudioBlob] = useState<Blob | null>(null);
const [recordTime, setRecordTime] = useState<number>(0);
const [transcription, setTranscription] = useState<string>('');
const mediaRecorderRef = useRef<MediaRecorder | null>(null);
const audioChunksRef = useRef<Blob[]>([]);
const streamRef = useRef<MediaStream | null>(null);
const recordIntervalRef = useRef<NodeJS.Timeout | null>(null);
useEffect(() => {
async function setupRecorder() {
try {
await register(await connect());
} catch (error: any) {
if (error.message.includes("already an encoder stored")) {
console.log("Encoder already registered, continuing...");
} else {
console.error('Error registering encoder:', error);
return;
}
}
try {
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
streamRef.current = stream;
} catch (error: any) {
console.error('Error accessing microphone:', error);
}
}
setupRecorder();
return () => {
if (streamRef.current) {
streamRef.current.getTracks().forEach(track => track.stop());
}
if (recordIntervalRef.current) {
clearInterval(recordIntervalRef.current);
}
};
}, []);
const transcribeAudio = async (audioBlob: Blob) => {
// const arrayBuffer = await audioBlob.arrayBuffer();
// const audioData = new Uint8Array(arrayBuffer);
// const audioData = new Float32Array(arrayBuffer);
const audioURL = URL.createObjectURL(audioBlob);
const audioData = await read_audio(audioURL, 16000);
try {
const transcriber = await pipeline('automatic-speech-recognition', 'Xenova/whisper-medium');
console.log('Transcriber initialized.'); // Confirm that the transcriber has been initialized
const output = await transcriber(audioData, { language: 'english', task: 'transcribe', chunk_length_s: 30, stride_length_s: 5 });
console.log('Transcription output:', output); // Log the full output object
if (output && !Array.isArray(output) && output.text) {
setTranscription(output.text);
} else {
setTranscription('No transcription output');
}
URL.revokeObjectURL(audioURL); // clean up the object URL after use
} catch (error: any) {
console.error('Error transcribing audio:', error);
}
};
const startRecording = () => {
if (streamRef.current) {
mediaRecorderRef.current = new MediaRecorder(streamRef.current, { mimeType: 'audio/wav' }) as any; // any; because it's from ext library
if (mediaRecorderRef.current) { // Check if mediaRecorderRef.current is not null before adding event listeners
mediaRecorderRef.current.addEventListener('dataavailable', (event: BlobEvent) => {
audioChunksRef.current.push(event.data);
});
mediaRecorderRef.current.addEventListener('stop', async () => {
if (mediaRecorderRef.current) { // Additional check before accessing mimeType
const mimeType = mediaRecorderRef.current.mimeType;
const audioBlob = new Blob(audioChunksRef.current, { type: mimeType });
setAudioBlob(audioBlob);
audioChunksRef.current = [];
await transcribeAudio(audioBlob);
}
});
audioChunksRef.current = [];
mediaRecorderRef.current.start();
setIsRecording(true);
setRecordTime(0);
recordIntervalRef.current = setInterval(() => {
setRecordTime(prevTime => prevTime + 1);
}, 1000);
} else {
console.error('Failed to initialize MediaRecorder');
}
} else {
console.error('Stream not initialized');
}
};
const stopRecording = () => {
if (mediaRecorderRef.current && mediaRecorderRef.current.state === 'recording') {
mediaRecorderRef.current.stop();
setIsRecording(false);
if (recordIntervalRef.current) {
clearInterval(recordIntervalRef.current);
}
} else {
console.error('MediaRecorder not recording or not initialized');
}
};
const playAudio = () => {
if (audioBlob) {
const audioURL = URL.createObjectURL(audioBlob);
const audio = new Audio(audioURL);
audio.play().catch((error: any) => {
console.error('Error playing the audio:', error);
URL.revokeObjectURL(audioURL);
});
}
};
return (
<div className="audio-input-container">
<button onClick={startRecording}>Start Recording</button>
<button onClick={stopRecording}>Stop Recording</button>
<button onClick={playAudio}>Play Audio</button>
<p>Recording: {isRecording ? `${recordTime} seconds` : 'No'}</p>
<p>Transcription: {transcription}</p>
</div>
);
};
export default AudioInput;
from transformers.js.
Note that every single call to
const transcriber = await pipeline('automatic-speech-recognition', 'Xenova/whisper-medium');
allocates new memory for a pipeline (and takes a lot of time to construct the model). This is most likely the reason for your out-of-memory issues, since you call this every time you transcribe audio.
I would also recommend selecting a smaller model, like https://huggingface.co/Xenova/whisper-base, https://huggingface.co/Xenova/whisper-small, https://huggingface.co/Xenova/whisper-tiny, https://huggingface.co/distil-whisper/distil-medium.en, or https://huggingface.co/distil-whisper/distil-small.en.
Hope that helps!
from transformers.js.
That's good to know. I'll give those other models a shot. However this happens on the first call.
The dev machine i'm trying to run it in has a RTX 4090. I'm surprised that's the issue, since i've never run into memory problems when running whisper in python. Does WebGPU have a memory ceiling?
Thanks for your help.
from transformers.js.
The dev machine i'm trying to run it in has a RTX 4090. I'm surprised that's the issue, since i've never run into memory problems when running whisper in python. Does WebGPU have a memory ceiling?
Assuming you are running Transformers.js v2, everything still runs with WASM/CPU. You can follow along with the development of v3 here, which will add WebGPU support.
from transformers.js.
Related Issues (20)
- TypeError: Cannot read properties of undefined (reading 'create') HOT 6
- Could not read from file: D:\vite-project\node_modules\stream-browserify\web HOT 7
- Any plans to add moondream and build a demo? Xenova/moondream2 HOT 2
- TypeError: Failed to fetch dynamically imported module HOT 4
- Error using Xenova/nanoLLaVA in pipeline HOT 4
- Excessive Memory consumption. HOT 4
- could not find model_q4.onnx_data (v3 PR) HOT 1
- Uncaught (in promise) Error: no available backend found. ERR: [webgpu] TypeError: Failed to fetch dynamically imported module: HOT 11
- Options for the "translation" pipeline when using Xenova/t5-small
- Have considered using wasm technology to implement this library? HOT 1
- Example not working on Chrome/Arc v.124(M1 Mac) HOT 5
- Can you use all transformers models with transformers.js?
- Request to support suno-ai/bark model HOT 2
- Something went wrong during model construction (most likely a missing operation). Using `wasm` as a fallback. HOT 4
- v3: Issue loading T5-Small with webgpu backend HOT 1
- WebAssembly Compilation Error Due to CSP Restrictions HOT 5
- [v3] webgpu-chat demo fails with `no available backend found`
- How to point to a specific model path in order to use compressed models? (brotli) HOT 1
- Method to send `HF_TOKEN` from the client side
- Pipeline execution time with 'image-classification' pipeline HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from transformers.js.