Comments (5)
We talked about this during today's call.
This is hard to do, because AudioContext
have methods and nodes that are not present in OfflineAudioContext
. What happens if you're receiving data from a PeerConnection
using MediaStreamTrackAudioSourceNode
, and then you turn it into an OfflineAudioContext
, where this AudioNode
doesn't exist?
How are AudioParam
changes scheduled? I assume one would have to re-do all the AudioParam
automation calls? AudioContext.currentTime
would reset to 0?
Additionally, changing the inheritance is a breaking change, this wouldn't evaluate to true:
Object.getPrototypeOf(AudioContext).name == "BaseAudioContext"
from web-audio-api-v2.
For
export a song as a wav
as long as the Float32Array
representation of the audio is accessible the WAV file header can be set at any time. To that end if the song is live AudioWorkletProcessor
can be used as a bypass for any connected audio node https://github.com/web-platform-tests/wpt/blob/d5be80a86d4f938250c075ac12414ad47516969c/webaudio/js/worklet-recorder.js then WAV headers can be set once the start and end times of the audio to be exported are reached using, for example, https://github.com/mattdiamond/Recorderjs/blob/master/src/recorder.js.
Alternatively, at Chromium, Chrome MediaRecorder
with mimeType
set to "video/x-matroska;codecs=pcm"
or "audio/webm;codecs=pcm"
can be used to record all audio as PCM, which can then be extracted from the Matroska file (e.g., using mkv2xml
or ts-ebml
) and converted to Float32Array
then further processed to converted to a WAV file proper, see #63.
Another alternative is using Native Messaging to convert any file or data input type to WAV. E.g., opus-tools to convert Opus to WAV with opusdec
, e.g., https://github.com/guest271314/native-messaging-espeak-ng downloads opus-tools to convert WAV output by espeak-ng
to OGG.
from web-audio-api-v2.
thanks for discussing the feature request in the group. i kind of expected that i failed to see the big picture :)
suspend()
was indeed a bad idea. instead of suspending could a new AudioContext state maybe work better? For example, going offline with context.offline()
- which would fail should the context contain MediaElement/Stream nodes or something else unavailable in the BaseAudioContext - and calling context.resume()
to switch the state back to realtime.
re-scheduling AudioParams sounds right at least for the present use case. no idea how to resolve the breaking change in the inheritance pattern though, or how to respond to OfflineAudioContext methods when the state is realtime.
but yeah, there might well be further issues that are outside my limited perspective, so pls feel free to close the issue as you see best.
from web-audio-api-v2.
We discussed this again in the call.
There are too many differences and problems to solve for this to be solved at the Web Audio API level, and I think in general that a DAW always will have its own transport layer and automation scheduling on this transport, different from the AudioContext.currentTime
, so suspend wouldn't work as you say.
You can alleviate the memory requirements by rendering track by track and performing a mix down, or creating nodes at the right time using suspend()
, or destroying your real-time Web Audio API graph to create an Offline one.
from web-audio-api-v2.
we have two online daw use cases where the user needs to A) export a song as a wav,
Minimally modifying existing code from https://github.com/web-platform-tests/wpt/blob/d5be80a86d4f938250c075ac12414ad47516969c/webaudio/js/worklet-recorder.js to use AudioWorklet
for processing "real-time" playback and https://github.com/chromium/chromium/blob/77578ccb4082ae20a9326d9e673225f1189ebb63/third_party/blink/web_tests/webaudio/resources/audio-file-utils.js to process either real-time or non-real-time input audio data to wav output as a file
<!DOCTYPE html>
<html>
<head>
<title>Audio input to wav file</title>
</head>
<body>
<input type="file" />
<audio controls></audio>
<script type="worklet">
// https://github.com/web-platform-tests/wpt/blob/d5be80a86d4f938250c075ac12414ad47516969c/webaudio/js/worklet-recorder.js
/**
* @class RecorderProcessor
* @extends AudioWorkletProcessor
*
* A simple recorder AudioWorkletProcessor. Returns the recorded buffer to the
* node when recording is finished.
*/
class RecorderProcessor extends AudioWorkletProcessor {
/**
* @param {*} options
* @param {number} options.processorOption.duration A duration to record in seconds.
* @param {number} options.processorOptions.channelCount A channel count to record.
*/
constructor(options) {
super();
console.log(options);
this._createdAt = currentTime;
this._elapsed = 0;
this._recordDuration =
(options.processorOption && options.processorOptions.duration) || 1;
this._recordChannelCount =
(options.processorOptions && options.processorOptions.channelCount) || 1;
this._recordBufferLength = sampleRate * this._recordDuration;
this._recordBuffer = [];
for (let i = 0; i < this._recordChannelCount; ++i) {
this._recordBuffer[i] = [];
// new Float32Array(this._recordBufferLength);
}
this.stop = false;
this.port.onmessage = e => {
this.stop = true;
this.port.onmessage = null;
}
}
process(inputs, outputs) {
if (this.stop) {
this.port.postMessage({
currentFrame,
sampleRate,
type: 'recordfinished',
recordBuffer: this._recordBuffer.map(floats => new Float32Array(floats))
});
this.port.close();
return false;
}
// Records the incoming data from |inputs| and also bypasses the data to
// |outputs|.
const input = inputs[0];
const output = outputs[0];
if (!input.length) {
console.log(input);
return true;
}
for (let channel = 0; channel < input.length; channel++) {
let inputChannel, outputChannel;
try {
inputChannel = input[channel];
outputChannel = output[channel];
outputChannel.set(inputChannel);
// console.log(this._recordBuffer, channel);
const buffer = this._recordBuffer[channel];
// const capacity = buffer.length - currentFrame;
buffer.push(...inputChannel.slice(0));
} catch (e) {
console.error(e, {channel, inputs, outputs, input, output});
// throw e;
}
}
return true;
}
}
registerProcessor('recorder-processor', RecorderProcessor);
</script>
<script>
// https://github.com/chromium/chromium/blob/77578ccb4082ae20a9326d9e673225f1189ebb63/third_party/blink/web_tests/webaudio/resources/audio-file-utils.js
// Utilities for creating a 16-bit PCM WAV file from an AudioBuffer
// when using Chrome testRunner, and for downloading an AudioBuffer as
// a float WAV file when running in a browser.
function writeString(s, a, offset) {
for (let i = 0; i < s.length; ++i) {
a[offset + i] = s.charCodeAt(i);
}
}
function writeInt16(n, a, offset) {
n = Math.floor(n);
let b1 = n & 255;
let b2 = (n >> 8) & 255;
a[offset + 0] = b1;
a[offset + 1] = b2;
}
function writeInt32(n, a, offset) {
n = Math.floor(n);
let b1 = n & 255;
let b2 = (n >> 8) & 255;
let b3 = (n >> 16) & 255;
let b4 = (n >> 24) & 255;
a[offset + 0] = b1;
a[offset + 1] = b2;
a[offset + 2] = b3;
a[offset + 3] = b4;
}
// Return the bits of the float as a 32-bit integer value. This
// produces the raw bits; no intepretation of the value is done.
function floatBits(f) {
let buf = new ArrayBuffer(4);
new Float32Array(buf)[0] = f;
let bits = new Uint32Array(buf)[0];
// Return as a signed integer.
return bits | 0;
}
function writeAudioBuffer(audioBuffer, a, offset, asFloat) {
let n = audioBuffer.length;
// let n = audioBuffer.reduce((a, b) => a + b.length, 0);
let channels = audioBuffer.numberOfChannels;
// let channels = audioBuffer.length;
for (let i = 0; i < n; ++i) {
for (let k = 0; k < channels; ++k) {
let buffer = audioBuffer.getChannelData(k);
// let buffer = audioBuffer[k];
if (asFloat) {
let sample = floatBits(buffer[i]);
writeInt32(sample, a, offset);
offset += 4;
} else {
let sample = buffer[i] * 32768.0;
// Clip samples to the limitations of 16-bit.
// If we don't do this then we'll get nasty wrap-around distortion.
if (sample < -32768) sample = -32768;
if (sample > 32767) sample = 32767;
writeInt16(sample, a, offset);
offset += 2;
}
}
}
}
// See http://soundfile.sapp.org/doc/WaveFormat/ and
// http://www-mmsp.ece.mcgill.ca/Documents/AudioFormats/WAVE/WAVE.html
// for a quick introduction to the WAVE PCM format.
function createWaveFileData(audioBuffer, asFloat) {
let bytesPerSample = asFloat ? 4 : 2;
let frameLength = audioBuffer.length; // audioBuffer[0].length
let numberOfChannels = audioBuffer.numberOfChannels; // audioBuffer.length
let sampleRate = audioBuffer.sampleRate; // ac.sampleRate; sampleRate
let bitsPerSample = 8 * bytesPerSample;
let byteRate = (sampleRate * numberOfChannels * bitsPerSample) / 8;
let blockAlign = (numberOfChannels * bitsPerSample) / 8;
let wavDataByteLength = frameLength * numberOfChannels * bytesPerSample;
let headerByteLength = 44;
let totalLength = headerByteLength + wavDataByteLength;
let waveFileData = new Uint8Array(totalLength);
let subChunk1Size = 16; // for linear PCM
let subChunk2Size = wavDataByteLength;
let chunkSize = 4 + (8 + subChunk1Size) + (8 + subChunk2Size);
writeString('RIFF', waveFileData, 0);
writeInt32(chunkSize, waveFileData, 4);
writeString('WAVE', waveFileData, 8);
writeString('fmt ', waveFileData, 12);
writeInt32(subChunk1Size, waveFileData, 16); // SubChunk1Size (4)
// The format tag value is 1 for integer PCM data and 3 for IEEE
// float data.
writeInt16(asFloat ? 3 : 1, waveFileData, 20); // AudioFormat (2)
writeInt16(numberOfChannels, waveFileData, 22); // NumChannels (2)
writeInt32(sampleRate, waveFileData, 24); // SampleRate (4)
writeInt32(byteRate, waveFileData, 28); // ByteRate (4)
writeInt16(blockAlign, waveFileData, 32); // BlockAlign (2)
writeInt32(bitsPerSample, waveFileData, 34); // BitsPerSample (4)
writeString('data', waveFileData, 36);
writeInt32(subChunk2Size, waveFileData, 40); // SubChunk2Size (4)
// Write actual audio data starting at offset 44.
writeAudioBuffer(audioBuffer, waveFileData, 44, asFloat);
return waveFileData;
}
function createAudioData(audioBuffer, asFloat) {
return createWaveFileData(audioBuffer, asFloat);
}
function finishAudioTest(event) {
let audioData = createAudioData(event.renderedBuffer);
testRunner.setAudioData(audioData);
testRunner.notifyDone();
}
// Save the given |audioBuffer| to a WAV file using the name given by
// |filename|. This is intended to be run from a browser. The
// developer is expected to use the console to run downloadAudioBuffer
// when necessary to create a new reference file for a test. If
// |asFloat| is given and is true, the WAV file produced uses 32-bit
// float format (full WebAudio resolution). Otherwise a 16-bit PCM
// WAV file is produced.
function downloadAudioBuffer(
audioBuffer,
/* filename, */ asFloat = false
) {
console.log(asFloat);
// Don't download if testRunner is defined; we're running a layout
// test where this won't be useful in general.
// if (window.testRunner) return false;
// Convert the audio buffer to an array containing the WAV file
// contents. Then convert it to a blob that can be saved as a WAV
// file.
let wavData = createAudioData(audioBuffer, /* filename, */ asFloat);
let blob = new Blob([wavData], { type: 'audio/wav' });
// Manually create html tags for downloading, and simulate a click
// to download the file to the given file name.
return blob;
// audio.src = URL.createObjectURL(blob);
/*
let a = document.createElement('a');
a.style.display = 'none';
a.download = filename;
let audioURL = window.URL.createObjectURL(blob);
let audio = new Audio();
audio.src = audioURL;
a.href = audioURL;
document.body.appendChild(a);
a.click();
*/
return true;
}
function audioInputToWav(audioInput, asFloat = false) {
// TODO handle possible audio inputs
let mediaElement,
mediaStream,
mediaStreamTrack,
blob,
file,
blobURL,
mediaFragmentURI,
url,
audioBuffer,
array,
typedArray;
switch (Object.getPrototypeOf(audioInput).constructor.name) {
case 'AudioBuffer':
console.log(audioInput);
case 'HTMLAudioElement':
case 'HTMLVideoElement':
console.log(audioInput);
break;
case 'MediaStream':
console.log(audioInput);
break;
case 'MediaStreamTrack':
console.log(audioInput);
break;
case 'Blob':
case 'File':
console.log(audioInput);
break;
case 'String':
console.log(audioInput);
break;
case 'Array':
console.log(audioInput);
break;
case 'Float32Array':
console.log(audioInput);
break;
case 'ArrayBuffer':
console.log(audioInput);
break;
case 'Uint8Array':
console.log(audioInput);
break;
case 'Uint16Array':
console.log(audioInput);
break;
default:
break;
}
}
const input = document.querySelector('input[type=file]');
const mediaElement = document.querySelector('audio');
input.onchange = async e => {
const [file] = e.target.files;
const ac = new AudioContext();
// const ab = await ac.decodeAudioData(await file.arrayBuffer());
ac.onstatechange = e => console.log(e);
await ac.suspend();
const audioWorklet = URL.createObjectURL(
new Blob(
[document.querySelector('script[type=worklet]').textContent],
{ type: 'text/javascript' }
)
);
await ac.audioWorklet.addModule(audioWorklet);
const source = new MediaElementAudioSourceNode(ac, { mediaElement });
// const msd = new MediaStreamAudioDestinationNode(ac);
// const { stream: mediaStream } = msd;
// const ms = new MediaStreamAudioSourceNode(ac, { mediaStream });
// const [track] = mediaStream.getAudioTracks();
// track.onmute = track.onunmute = track.onended = e => {
// console.log(e);
// };
// audioInputToWav(mediaStream);
audioInputToWav(mediaElement);
audioInputToWav(file);
// audioInputToWav(track);
const aw = new AudioWorkletNode(ac, 'recorder-processor', {
processorOptions: { channelCount: 2 },
});
aw.onprocessorerror = console.trace;
aw.port.onmessage = async e => {
const { recordBuffer, sampleRate, currentFrame } = e.data;
// an Array could be substituted for AudioBuffer, here,
// with changes made where numberOfChannels,
// length, sampleRate are used in original code
const audioBuffer = new AudioBuffer({
length: currentFrame,
sampleRate,
numberOfChannels: recordBuffer.length,
});
recordBuffer.forEach((floats, index) => {
audioBuffer.getChannelData(index).set(floats);
});
console.log(audioBuffer);
const blob = downloadAudioBuffer(audioBuffer, true);
audioInputToWav(blob);
const audio = new Audio(URL.createObjectURL(blob));
audio.controls = true;
audio.onloadedmetadata = e => {
console.log(e.target.duration, mediaElement.currentTime);
audio.play();
};
audio.onpause = e => {
console.log(e.target.duration, mediaElement.currentTime);
};
document.body.appendChild(audio);
};
source.connect(aw);
// ms.connect(aw);
aw.connect(ac.destination);
const blobURL = URL.createObjectURL(file);
audioInputToWav(blobURL);
mediaElement.src = blobURL + '#t=0,5';
mediaElement.oncanplaythrough = async _ => {
console.log(mediaElement.src);
mediaElement.oncanplaythrough = null;
await ac.resume();
await mediaElement.play();
if ('mozCaptureStream' in mediaElement)
setTimeout(_ => mediaElement.pause(), 5000);
};
mediaElement.onplay = async _ => {};
mediaElement.onpause = e => {
aw.port.postMessage('stop');
// track.stop();
mediaElement.onpause = null;
console.log(e, source);
};
};
</script>
</body>
</html>
https://github.com/guest271314/audioInputToWav/blob/master/index.html
and B) freeze a (midi) track into an audio clip. currently we can reconstruct the realtime graph in an OfflineAudioContext on demand
Am not sure gather if there is any difference between a midi track and a MediaStreamTrack
streaming midi data or what the exact expected output is. MediaRecorder.pause()
and resume()
could be used to record real-time media stream, omitting audio input not intended to be captured for further processing. Alternatively, an oscillator node could be created to stream silence continuously, and RTCRtpSender.replaceTrack()
could be used to substitute stream of silence for midi track then replace the track back to midi or other track.
from web-audio-api-v2.
Related Issues (20)
- V2 documentation logistics HOT 2
- interaction between audio elements and the Web Audio API HOT 2
- Add ability to pause/resume AudioBufferSourceNode HOT 11
- AudioOutputContext HOT 21
- AnalyserNode: provide access to complex FFT result HOT 15
- AnalyserNode: efficiency improvements to FFT post-processing HOT 3
- Alternatives for module loading of AudioWorklet HOT 5
- Should 'Atomics.wait' be available in AudioWorklets associated with an OfflineAudioContext? HOT 8
- Request to Expose AudioBuffer to DedicatedWorker HOT 29
- Rename master branch to main HOT 2
- Set up to display the spec. HOT 10
- AudioWorklet should have a 'oversample' and a 'bufferSize' property HOT 5
- cancelScheduledValues and cancelAndHoldAtTime throws RangeError for non-finite
- Switching between 2 Microphones HOT 1
- No way to convert data from WebCodecs AudioData to AudioBuffer HOT 7
- Add frequency of one oscillator to be based on another HOT 1
- Define TextEncoderStream(), and TextDecoderStream() in AudioWorkletGlobalScope HOT 1
- Add Travis CI HOT 1
- RawPCMStreamNode (StreamNode, or InputStreamNode) HOT 8
- Set up PR review HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from web-audio-api-v2.