webaudio / web-audio-api-v2 Goto Github PK
View Code? Open in Web Editor NEWThe Web Audio API v2.0, developed by the W3C Audio WG
License: Other
The Web Audio API v2.0, developed by the W3C Audio WG
License: Other
In https://webaudio.github.io/web-audio-api/#constructors-4, there is no description for the options parameter for the AudioBuffer constructor. Maybe we want to add a short sentence for completeness.
AnalyserNode already performs a windowed FFT. However, it provides only access to a smoothed power spectrum (log-magnitudes of the complex FFT result vector) through getFloatFrequencyData() resp. getByteFrequencyData(). There is no access to raw magnitude nor any phase information, even though the FFT must compute it. This needlessly limits the applicability of AnalyserNode.
Please add an accessor getComplexFrequencyData() that returns the unsmoothed complex-valued FFT result vector (called X[k] here directly. Whatever format is used internally is fine - it could be a float[2][fftsize], float[fftsize][2], or float[2*fftsize] with real and imaginary components interleaved or not. The idea here is to provide direct access to the FFT result without incurring any performance penalties.
A getPolarFrequencyData() that converts the FFT result into polar (magnitude & phase) coordinates would also be useful, as this is a frequent use case. Alternatively, since the existing getFloatFequencyData() infrastructure (with smoothing, log, and scaling) already provides the magnitudes, a getPhaseData() that computes just the phase vector would suffice. Here is a fast atan2() approximation that would provide added value. Cheers!
https://webaudio.github.io/web-audio-api/#fft-windowing-and-smoothing-over-time specifies the post-processing of the complex FFT result vector to obtain smoothed log-magnitudes for getFloatFrequencyData() in AnalyserNode. I want to suggest an efficiency improvement here.
Leaving out smoothing for the moment, the post-processing computes each element of the result vector as Y[k] = 20*log10(sqrt(r[k]*r[k] + i[k]*i[k]))
, where r[k] and i[k] are the real and imaginary components of the complex FFT result X[k]. However, the sqrt() operation here can be eliminated, since Y[k] = 10*log10(r[k]*r[k] + i[k]*i[k])
is mathematically identical.
Smoothing complicates this picture, since the two forms (with and without square root) are now no longer identical. Taking a step back, however, we can ask why there is smoothing before taking logs - what does that achieve? I believe that the unstated purpose here is to create a peak detector: exponential smoothing before taking logs creates an impulse response with exponentially fast attack but slow linear decay to zero, both at a rate governed by the smoothing constant tau.
The proposed sqrt-free version would make this asymmetry more pronounced - essentially doubling the attack and halving the decay rate. We could improve backwards compatibility by internally substituting tau := 2*tau - 1
to re-double the decay rate, at the cost of now having quadruple the previous attack rate.
If this sounds bad, there are three mitigating factors at work here:
Finally, I can provide a fast approximate log if there's interest. Cheers!
WebCodecs uses AudioBuffer as a container of unencoded audio.
https://wicg.github.io/web-codecs/#dom-audioframe-buffer
The AudioFrame, which wraps AudioBuffer, is provided as an input to AudioEncoder and as an output from AudioDecoder. We've exposed the encoder and decoder interfaces to DedicatedWorker to facilitate offloaded codec IO.
I think the change is as simple as changing the Exposed= parameter for just AudioBuffer. I'll send a PR to demonstrate.
i'm trying to figure out how to have a lower latency when playing wav samples.
i'm digging into audio worklet to find some informations but no solutions (if it exist)
i was thinking about loading my wav samples in an internal buffer when loading page
and playing it back with a lower latency but is it even possible for now ?
we have two online daw use cases where the user needs to A) export a song as a wav, and B) freeze a (midi) track into an audio clip. currently we can reconstruct the realtime graph in an OfflineAudioContext on demand, but this doubles the memory requirements, and may even take some time with complex graphs.
feature request: would be cool to just call AudioContext.suspend() to turn the realtime context into an offline one, call startRendering() and then happily resume() back to realtime in oncomplete. i guess this would also comply with autoplaypolicy.
would inheritance chain AudioContext extends OfflineAudioContext extends BaseAudioContext
make any sense?
Describe the feature
Add ability to pause/resume AudioBufferSourceNode
Is there a prototype?
Describe the feature in more detail
I think people would love to have that. Not sure why is it not possible and explicitly stated as not being provided.
Describe the Issue
Need to start a new doc for the new version
Probably want to have issue templates for bugs and feature requests like we have for https://github.com/WebAudio/web-audio-api/issues/new/choose
Could just steal the templates from there; don't think we need to make any changes for v2 for that.
Describe the feature
Take two oscillators. Say I have one oscillatorA that is ramping linearly, I'd like there to be a way such that you can make the other oscillatorB scale as a function of the initial one.
For example:
oscNodeA.frequency.setValueAtTime(1, context.currentTime); oscNodeA.frequency.linearRampToValueAtTime( 3000, context.currentTime + 10 ); oscNodeB.frequency.value = oscNodeA.frequency.value * 2;
Which should operate like if I pass oscNodeA.frequency.value in an object, it will constantly update the frequency as oscNodeA changes frequency. It's useful for shifting frequencies of whole chords made by Oscillators
First, https://webaudio.github.io/web-audio-api/#channel-limitations describes some limitations. But this doesn't really add anything that can't be determined from the (fixed!) channelCountMode
and channelCount
values specified for the node. With these values, all inputs are mixed to stereo on input so the output is always stereo.
Second, https://webaudio.github.io/web-audio-api/#panner-channel-limitations references https://webaudio.github.io/web-audio-api/#channel-limitations, so we don't need this section either (for the same reasons).
Describe the feature
WebCodecs defines AudioData
. In the WebCodecs specification this note appears:
NOTE: The Web Audio API currently uses f32-planar exclusively.
However, the format
of AudioData
from AudioDecoder
is 'f32'
not 'f32-planar'
.
Even though sampleRate
set at AudioDecoder
configuration sampleRate
is other than 48000
(and opusenc
supports --raw-rate
option to specifically set sample rate for Opus encoded audio) the resulting WebCodecs AudioData
instance always has sampleRate
set to 48000
.
The effective result is that there is no way that I am aware of to convert the data from AudioData.copyTo(ArrayBuffer, {planeIndex: 0})
to an AudioBuffer
instance that can be played with AudioBufferSourceNode
or resampled to a different sampleRate
, for example, 22050
.
Since MediaStreamTrackGenerator
suffers from "overflow" and no algorithm exists in the WebCodecs specification to handle the overflow outside of one defined by the user it is necessary for the user to write the algorithm. After testing a user might find a magic number to delay the next call to MediaStreamTrackGenerator.writable.WritableStreamDefaultWriter.write()
https://plnkr.co/edit/clbdVbhaRhCKWmPS that approach does not achieve the same result when attempting to use a Web Audio API AudioBuffer
and AudioSourceNode
async function main() {
const oac = new AudioContext({
sampleRate: 48000,
});
let channelData = [];
const decoder = new AudioDecoder({
error(e) {
console.error(e);
},
async output(frame) {
const { duration: d } = frame;
const size = frame.allocationSize({ planeIndex: 0 });
const data = new ArrayBuffer(size);
frame.copyTo(data, { planeIndex: 0 });
const view = new Float32Array(data);
let i = 0;
for (let i = 0; i < view.length; i++) {
if (channelData.length === 220) {
const floats = new Float32Array(220);
floats.set(channelData.splice(0, 220));
const ab = new AudioBuffer({
sampleRate: 48000,
length: floats.length,
numberOfChannels: 1,
});
ab.getChannelData(0).set(floats);
const source = new AudioBufferSourceNode(oac, { buffer: ab });
source.connect(oac.destination);
console.log(ab.duration, ab.sampleRate);
source.start();
await new Promise((r) => {
console.log(ab);
source.onended = r;
});
}
channelData.push(view[i]);
}
if (decoder.decodeQueueSize === 0) {
if (channelData.length) {
const floats = new Float32Array(220);
floats.set(channelData.splice(0, 220));
const ab = new AudioBuffer({
sampleRate: 48000,
length: floats.length,
numberOfChannels: 1,
});
ab.getChannelData(0).set(floats);
console.log(ab.duration, ab.sampleRate);
const source = new AudioBufferSourceNode(oac, { buffer: ab });
source.connect(oac.destination);
source.start();
await new Promise((r) => (source.onended = r));
await decoder.flush();
return;
}
}
},
});
const encoded = await (await fetch('./encoded.json')).json();
let base_time = encoded[encoded.length - 1].timestamp;
console.assert(encoded.length > 0, encoded.length);
console.log(JSON.stringify(encoded, null, 2));
const metadata = encoded.shift();
console.log(encoded[encoded.length - 1].timestamp, base_time);
metadata.decoderConfig.description = new Uint8Array(
base64ToBytesArr(metadata.decoderConfig.description)
).buffer;
console.log(await AudioEncoder.isConfigSupported(metadata.decoderConfig));
decoder.configure(metadata.decoderConfig);
while (encoded.length) {
const chunk = encoded.shift();
chunk.data = new Uint8Array(base64ToBytesArr(chunk.data)).buffer;
const eac = new EncodedAudioChunk(chunk);
decoder.decode(eac);
}
}
verifying the AudioData
data and AudioBuffer
channel data are incompatible.
Is there a prototype?
No.
Describe the feature in more detail
Web Audio API AudioBuffer
<=> WebCodecs AudioData
Provide an algorithm and method to convert WebCodecs AudioData
to Web Audio API AudioBuffer
with option to set sample rate of the resulting object.
Please forgive my ignorance here. I'm new to audio synthesis...
WebAudio supports using custom waveforms with oscillator nodes, but the API (createPeriodicWave
) only accept arguments defined in the frequency domain (real
and imag
).
I have periodic waves in the time domain (defined as functions), but without an FFT, there seems to be no way to pass them into an oscillator node. I understand there are a couple of ways to feed arbitrary signals into the routing graph, but this request is about the signature of createPeriodicWave
.
There are a few JavaScript libraries online with FFT implementations, but none have been updated for years. I could probably figure it out for myself, or compile some C to WebAssembly or something, but it would obviously be much nicer if createPeriodicWave
also accepted time-domain arguments.
More generally, the API could provide FFT and IFFT helpers (especially if they are already implicitly required by the implementation), but even then, the createPeriodicWave
method should still be able to understand time-domain arguments.
Again, sorry if I've misunderstood something.
Following up on issue WebAudio/web-audio-api#1848, which briefly discusses why a blocking wait like Atomics.wait
is detrimental to real-time audio processing, in particular this comment made by @hoch:
For real-time audio use case, synchronously blocking the thread is generally not a good idea.
... and this comment by @padenot:
one should never ever ever wait in a real-time audio callback, so we disabled Atomic.wait [...] if they are trying to wait on anything, they are doing something wrong
To briefly reflect on these thoughts, I think it should be pointed out that, strictly speaking, none of these apply to offline audio processing, and consequently, I think there is probably no reason whatsoever to disable Atomics.wait
in AudioWorklets associated with an OfflineAudioContext (no reason other than the consistency of available APIs, that is).
In fact, for offline audio rendering that involves cross-thread audio processing, a blocking wait is essentially necessary at one point to guarantee cross-thread processing completion, especially if low or zero latency is necessary, due to how an offline rendering thread schedules its process calls (i.e. as fast as computationally possible, without considering cross-thread audio processing load).
This is currently possible and feasible with a busy wait (and there are plenty of high-level workarounds as well).
Any thoughts on making Atomics.wait
available in offline rendering, to replace busy waiting in these cases?
The WG will flesh out some plans for V2 document migration. There are a couple of issues, but the general consensus is to separate out the V2 spec.
Describe the feature
Decode hexadecimal input to Float32Array
capable of being copied to an existing AudioBuffer
or processing with AudioWorkletProcessor
. Encode Float32Array
to hexadecimal.
From V1 repository
If you need help on how to use WebAudio, ask on your favorite forum or consider visiting
the WebAudio Slack Channel (register here) or
StackOverflow.
Visited web-audio.slack.com first, yet apparently an "invitation" is required
Don't have an account on this workspace yet?
Contact the workspace administrator for an invitation
If this is issue is deemed off-topic for this repository kindly post the "invitation" to the slack page so can ask the question there.
Banned from SO for 4+ more years.
Is there a prototype?
No completely. Though found this example https://babbage.cs.qc.cuny.edu/IEEE-754.old/32bit.html today.
A sampling of research efforts thus far:
Describe the feature in more detail
Provide more information how it does and how it works.
Decoding PCM from hexadecimal to Float32Array
and encoding Float32Array
to hexadecimal PCM (or other audio codec).
The use case is creating N discrete files, for example using a WebM container template where the values can be read from or written to the template for the purpose of "streaming" (encoding and decoding, reading, writing) those N files for "seamless" playback using AudioWorklet
without using decodeAudioData()
. Since Chromium is capable of playing Matroska(TM) container with PCM codec (MediaRecorder.isTypeSupported("audio/webm;codecs=pcm")
) and Firefox is currently moving towards supporting WebM with FLAC encoded audio (https://bugzilla.mozilla.org/show_bug.cgi?id=1606822) this will provide a means to encode and decode Matroska and WebM files "on the fly" for various use cases.
Sample input
Header (in an XML file output by mkv2xml
)
<mkv2xml>
<EBML>
<EBMLVersion>1</EBMLVersion>
<EBMLReadVersion>1</EBMLReadVersion>
<EBMLMaxIDLength>4</EBMLMaxIDLength>
<EBMLMaxSizeLength>8</EBMLMaxSizeLength>
<DocType>matroska</DocType>
<DocTypeVersion>4</DocTypeVersion>
<DocTypeReadVersion>2</DocTypeReadVersion>
</EBML>
<Segment>
<Info>
<TimecodeScale>1000000</TimecodeScale>
<MuxingApp>Chrome</MuxingApp>
<WritingApp>Chrome</WritingApp>
</Info>
<Tracks>
<TrackEntry>
<TrackNumber>1</TrackNumber>
<TrackUID>66451940004784068</TrackUID>
<TrackType>2</TrackType>
<CodecID>A_PCM/FLOAT/IEEE</CodecID>
<Audio>
<SamplingFrequency>22050.0</SamplingFrequency>
<Channels>1</Channels>
<BitDepth>32</BitDepth>
</Audio>
</TrackEntry>
</Tracks>
Hexadecimal values within a <data>
tag
0000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000
9201c93b4201a13b4201a13b9201c93be001703be001703b00000000000072bb
004017bd009080bd1a018d3c7f813f3d8981443de201f13b00c0ddbce001703b
0601833ccf81e73c9201c93b000000006a01353ce381713c1a018d3c4001a03a
4201a13be201f13b6a01353c4201213ce201f13b19810c3c9201c93b6a01353c
e381713cba015d3c19810c3c4201213c9201c93be201f13b9201c93be201f13b
6a01353ce201f13b19810c3c4201a13b9201c93b9201c93be201f13b9201c93b
4001203b9201c93b4001203b9201c93be001703b4001203b000000004001203b
6a01353c0080c9bc00602bbd9201c93b2e21173d6041303d00805dbc004021bd
e001703ba7a1533d0181803d7e41bf3c00401cbd00208dbd2e01973c4c41263d
ba015d3c4201a13b6a41b53c6b61353d56412b3d9201c93b00c0e7bc0601833c
cf81e73c6041303df801fc3c000000001a018d3c6a01353c1a018d3c0000fcbc
00a049bd00200dbd008049bc0601833ce201f13b4001a03a19810c3c9241c93c
ba015d3c4201213ca781d33c0000f2bb008035bc4201a13b000072bb0601833c
19810c3c000022bbe381713c9201c93b0000cabb0000a4ba1a018d3ce4c1f13c
0601033d9201c93b0080bfbc4201a13ce001703b9201493c4201a13b00000dbc
0080c9bc4201213ccf81e73c00805dbccf81e73c008049bc008035bc5641ab3c
7e41bf3c1a010d3d4001a03a008035bc4001203be4c1f13cf801fc3c4001a03a
0000a4ba008035bc0000a1bb0000a4ba5641ab3c00408dbc0601833c00000000
2e01973c19810c3c0000000000000000000000009381493d4201213ccf81e73c
9201c93b008049bc75613a3d0000000038211c3d6a01353c1a018d3ca781d33c
ba015d3c2e01973c000000002e01973c008049bc00c062bd0000f2bbbb81dd3c
e4c1f13ce201f13bba015d3c4201a13be381713c00000000000000002e01973c
bb81dd3c008035bc7e41bf3c9201c93b008035bce4c1f13c0000a4bae001703b
e201f13b0040a1bc19810c3c19810c3c0000f2bb4201213c4201a13bba015d3c
ba015d3c008035bc9201c93b4001a03a5641ab3c9201c93b0000a4ba000072bb
0000a1bb1a018d3c6a01353c000000004201a13be001703be381713ce001703b
004097bc1a010d3d00000000000021bc0601833c00805dbc4201a13c0000a4ba
4201a13bba015d3c9201c93b4201a13c0080c9bcbb81dd3c1a010d3d0080b5bc
6a41b53c00c0d3bce4c1f13c9381493d00409cbdeee1763d9201c93b0000cabb
0b81853d0080c9bc4201213ce001703b4001203bcf81e73c004097bc0000a1bb
e4c1f13c00805dbc4c41263de201f13b0080c9bc75613a3d00c0e7bc1a018d3c
e201f13b0080b5bc9201493ce4c1f13c000072bc0040a1bc0601833c2e01973c
008035bce001703bc5a1623d00e06cbd0000000038211c3dba015d3c0080b5bc
0080abbcc5a1623d000022bb000000000080abbc19810c3ccf81e73c004097bc
e201f13b4201a13ba781d33c0000f2bb19810c3c00c0ddbc000072bbc5a1623d
000021bc0080b5bca781d33c4001a03a0601833c0601833c004083bc000021bc
19810c3c1a018d3c000000004201a13c000022bb5641ab3c0000a4ba0040a1bc
9241c93c4201213c4001a03a0000a4ba9241c93c0040a1bcbb81dd3c8981443d
004097bda7a1533d4001a03a0080c9bc1a118d3d0080bfbc00805dbc4001a03a
00a03fbd4c41a63d008035bd000072bbcf81e73c00401cbd0601033d002003bd
7e41bf3c00c0d3bc2e01973c9201493c002012bdf801fc3c9201c93b4201213c
4c41263d9201493c00000dbc000072bb56412b3d4001a03ae381713c0601033d
f801fc3c00000dbc7e41bf3c002012bdf8017c3d4001a03a2e01973c9241c93c
008035bd8981443d7e41bf3ce201f13b2421123d00602bbd0080abbc1011883d
19810c3c008049bcba015d3c000022bbe201f13b0601833cf801fc3c00200dbd
e4c1f13c0601033d004097bd0601833d56412b3d00e0a8bde201f13b2e21173d
4221213d4201a13b00805dbc00c0d3bc29a1943d00408dbcba015d3c2e01973c
004021bd1a118d3d4001203b00e067bd2e31973d004017bd000021bc0181803d
00a044bd1a118d3d4001a03a00c0d8bdfd81fe3d0000fcbc1a018d3c7e41bf3c
0080bfbcdac16c3d9201c93b0080bfbc2e01973cba015d3c5641ab3c4001203b
00e06cbdc5a1623d9201493c4001203be001703b00408dbc7f813f3d4001a03a
00408dbc5641ab3c19810c3c4001203b0080b5bc4201a13c56412b3d4001203b
6a01353c4201a13c009085bdb631db3d00e06cbd1a018d3cbba15d3d00f0b2bd
e4e1713d2421923d00a0c9bd2421123da7a1533d0060b0bde4f1f13d00b094bd
e4e1713dba015d3c00409cbdca41e53d000072bde4c1f13c66e1b23d00a0cebd
00602bbd4001203e00401cbd000072bb0000000000c0d3bc1a010d3d6041303d
e001703b00200dbd00000000002003bd4201a13c0601833c5641ab3c2e21173d
that we can retrieve using DOMParser()
fetch("data.xml")
.then(response => response.text())
.then(xml => {
const res = [];
const parser = new DOMParser();
const doc = parser.parseFromString(xml, "application/xml");
const mkv = doc.documentElement;
const str = [...mkv.querySelectorAll("data")].map(({
textContent
}) => textContent.match(/\w+/g)).flat().join("");
console.log(str);
})
though am not certain about the canonical procedure to convert the above values to Float32Array
(s).
Thanks to the author of https://github.com/vi/mkvparse we know that the input is playable using ffplay
$ cat mediarecorder_pcm.xml | xml2 | grep '/mkv2xml/Segment/Cluster/SimpleBlock/data=' | cut -f2-2 -d= | xxd -r -p | ffplay -f f32le -ar 22050 -ac 1 -
.
Expected output is a Float32Array
1/4 the length of input hexadecimal string input that is suitable for playback using Web Audio API (i.e., within AudioWorkletProcessor
).
Is there a simpler algorithm to convert the hexadecimal values than the approach at https://babbage.cs.qc.cuny.edu/IEEE-754.old/32bit.html?
Describe the Issue
Need to have the PR review bot and Travis CI set up when we start editing the spec.
This is something that most profesionnal audio software on my system can do, we should have a look to see if we can make it work with the current equations.
Describe the Issue
Once we get spec text, we need Travis to verify the spec is correct
Expected Result
Travis should run bikeshed to generate the doc and verify that things are correct.
Actual Result
Travis CI isn't enabled yet.
Describe the issue
Is defining fetch() in AudioWorkletGlobalScope an implementer choice?
Where Is It
AFAICT not addressed.
Additional Information
fetch
is not defined in AudioWorkletGlobalScope
at Chromium. Why?
Is defining fetch
within AudioWorkletGlobalScope
at Firefox, Nightly within the parameters of the specification, or is fetch
intentionally omitted from being defined in AudioWorkletGlobalScope
?
Describe the Issue
The PR #95, the files were added to the master branch instead of the main branch
Expected Result
Everything should be on the main branch. We need to rename and/or move the master branch to the main branch.
Describe the issue
Does AudioContext.latencyHint
affect AudioWorkletProcessor::process
Callback Interval?
If so that language and effect is not clearly specified.
Where Is It
Additional Information
At DevTools open WebAudio tab. Observe the values of Callback Interval corresponding to different AudioContext.latencyHint
settings
"playback"
=> Callback Interval 23.217
"interactive"
=> Callback Interval 11.626
"balanced"
=> Callback Interval 9.9
0.5
=> Callback Interval 185.729
I was recently thinking about some ideas with web audio and I came to my head that,
could it be possible to add a oversample
and bufferSize
property to AudioWorklet?
This is could be a very useful feature for when making custom nodes.
While the oversample property does make the bufferSize property seem useless since oversampling is basically (bufferSize * n or basically 128 * n), I'm suggesting that:
the oversample
property:
WaveShaperNode.prorotype.oversample
.the bufferSize
property:
AudioWorkletOptions
and the can be only a read only property after initialization.ScriptProcessorNode
uses (a.k.a 128 - 16384). Or to a wider range base on performance and support of the worker thread and or hardware (a.k.a 128 - ?).I also realized that this could also affect the custom parameters especially with automation rates. My suggestion to work around this is that for the parameters with a k-rate
that if the oversample
is not set to 'none'
then the value is multiplied and for when the buffer size is greater than 128 frames the parameter produces values similar to a a-rate
parameter but has 1 value for each block (a.k.a per 128 frames) from the total buffer size.
For example: say the buffer size is 512. 512 divided by 128 equal 4 blocks per 128 frames. Since there a 4 blocks in 512 the k-rate
parameter should result in 4 different (or the same depending on the value prop in the main thread), values for each 128 frames per block out of 512 frames.
And for the a-rate
parameters, the value will result the same method for the oversample
property. In other words the value is multipled if the oversample
property is not set to 'none'
. Besides that buffer size shouldn't matter since an a-rate
parameter does not produce 1 value per 128 frames If you know what I mean.
Also (I'm not sure about this part since it's on a worker thread but), since oversampling and different buffer sizes can cause a little bit of latency (I think), Could it be possible to report the latency in ms as a property in both the main and worker thread?
Also this should also work with respect of the AudioContext sampleRate as well.
Similar to the step process that WaveShaperNode
does that must follow when using the oversample property according to the spec.
here's an example of what it could look like:
//main thread:
let worklet = new AudioWorkletNode(context, name, {
bufferSize: 256,
oversample: "none",
//etc...
});
worklet.oversample; //setter and getter. returns "none"
worklet.bufferSize; //read only! returns 256
worklet.latency; //read only! returns a number in ms the latency of the processor from input to output
//worker thread:
class MyProcessor extends AudioWorkletProcessor {
process(inputs, outputs, params) {
this.latency; //read only! returns a number in ms the latency of the processor from input to output
this.oversampleType; // read only! number - returns 0 for 'none' and a number for the oversample factor. i.e returns 2 if oversample = "2x"
this.bufferSize; //read only! number - returns the original selected buffer size from the AudioWorkletNodeOptions.
//this code here is used to get the result of the buffer size for when it's oversampled:
let oversampledSize = this.bufferSize * this.oversampleType;
//params:
//for k-rate parameters:
//since the buffer size is 256. 256 / 128 is 2 blocks. So there will be 2 values for each 128 frames for k-rate parameters.
//params["myParam"].length === 2
let a = params["myParam"][0]; //returns the value for the first 128 frames
let b = params["myParam"][1]; //returns the value for the next 128 frames
//for a-rate parameters:
//the buffer size doesn't matter. It's basically 'params["myParam"].length === bufferSize (256 in this example)
let c = params["myParam"][0]; //returns the first value out of 256 values
//for when either a k-rate or an a-rate parameter with oversample property is not set to "none",
//the value is multiplied otherwise it's the original value
//To achieve the original value, just divide it by the oversample amount:
let originalValue = params["myParam"][0] / this.oversampleType;
}
}
The goal of this feature is to achieve high quality processing and better performance in the worker thread with the audio worklet.
I hope i made sense and it's not too complicated. Also please correct if I'm wrong as well.
Do you think this is cool idea?
And could this be possible or not?
Thanks :)
Describe the issue
Am not sure if this is a specification issue or a Chromium bug (Chromium/Chrome is the only browser implementation am aware of at this time).
Initially ignored the issue that first observed when calling port.postMessage()
to main thread where resume()
is executed. Occasionally the processing would not commence, so refreshed the page.
When a user-defined method is created in the extended class
if values used in process()
were set in the method those values were not immediately available in process()
function.
Upon further inquiry found that process()
can be executed exactly 32 times, then appears to
not be capable of reading the value set in the method of the class
.
Where Is It
https://webaudio.github.io/web-audio-api/#AudioWorkletProcessor-methods
Additional Information
async appendBuffers({ data: { readable, processStream } }) {
processStream = new Function(`return ${processStream}`)();
let ready = false;
let next = [];
let overflow = [[], []];
let init = false;
globalThis.console.log({
currentTime,
currentFrame,
buffers: this.buffers,
});
const strategy = new ByteLengthQueuingStrategy({
highWaterMark: 32 * 1024,
});
const source = {
write: async value => {
// value (Uint8Array) length is not guaranteed to be multiple of 2 for Uint16Array
// store remainder of value in next array
if (value.length % 2 !== 0 && next.length === 0) {
next.push(...value.slice(value.length - 1));
value = value.slice(0, value.length - 1);
} else {
const prev = [...next.splice(0, next.length), ...value];
while (prev.length % 2 !== 0) {
next.push(...prev.splice(-1));
}
value = new Uint8Array(prev);
}
// we do not need length here, we process input until no more, or infinity
let data = new Uint16Array(value.buffer);
if (!init) {
init = true;
data = data.subarray(22);
console.log('init');
}
let { ch0, ch1 } = processStream(data);
// send 128 sample frames to process()
// to reduce, not entirely avoid, glitches
while (ch0.length && ch1.length) {
let __ch0, __ch1;
let _ch0 = ch0.splice(0, 128);
let _ch1 = ch1.splice(0, 128);
let [overflow0, overflow1] = overflow;
if (_ch0.length < 128 || _ch1.length < 128) {
overflow0.push(..._ch0);
overflow1.push(..._ch1);
break;
}
if (overflow0.length) {
__ch0 = overflow0.splice(0, overflow0.length);
__ch1 = overflow1.splice(0, overflow1.length);
while (__ch0.length < 128 && _ch0.length) {
let [float] = _ch0.splice(0, 1);
__ch0.push(float);
}
while (__ch1.length < 128 && _ch1.length) {
let [float] = _ch1.splice(0, 1);
__ch1.push(float);
}
}
const channel0 = new Float32Array(__ch0 || _ch0);
const channel1 = new Float32Array(__ch1 || _ch1);
this.buffers.set(this.i, {
channel0,
channel1,
});
// this is where the pause in execution of process() is observable
if (this.buffers.size < 64) {
console.log(this.buffers.size);
}
++this.i;
if (!ready) {
ready = true;
console.log(this.buffers.get(this.n));
this.port.postMessage({
start: true,
});
}
}
},
close: _ => {
console.log('writable close');
// handle overflow floats < 128 length
let [overflow0, overflow1] = overflow;
if (overflow0.length || overflow1.length) {
const channel0 = new Float32Array(overflow0.splice(0));
const channel1 = new Float32Array(overflow1.splice(0));
this.buffers.set(this.i, {
channel0,
channel1,
});
++this.i;
}
},
};
const writable = new WritableStream(source, strategy);
Object.assign(this, { readable, writable });
await readable.pipeTo(writable, {
preventCancel: true,
});
process
process(inputs, outputs) {
if (currentTime > 0.9 && this.buffers.size === 0) {
console.log(
this.i,
this.n,
this.buffers.size,
this.readable,
this.writable
);
this.endOfStream();
return false;
}
let channel0, channel1;
// this is where the pause/inability to read the value set is observable
// the pause always lasts exactly at least 32 executions of process
try {
if (this.n < 33) {
console.log(this.n, this.buffers.size);
}
// process() can be executed before appendBuffers()
// where this.buffers is set with values
// handle this.buffers.get(this.n) being undefined
// for up to 32 calls to process()
({ channel0, channel1 } = this.buffers.get(this.n));
// glitches can occur when sample frame size is less than 128
if (
(channel0 && channel0.length < 128) ||
(channel1 && channel1.length < 128)
) {
console.log(
channel0.length,
channel1.length,
currentTime,
currentFrame,
this.buffers.size
);
}
} catch (e) {
console.error(e, this.buffers.get(this.n), this.i, this.n);
// handle this.buffers.size === 0 while this.readable.locked (reading)
// where entry in this.buffers (Map) deleted and this.writable (writing)
// not having set new entry this.buffers,
// resulting in no data to set at currentTime
// example of JavaScript being single threaded?
return true;
}
const [[outputChannel0], [outputChannel1]] = outputs;
outputChannel0.set(channel0);
outputChannel1.set(channel1);
this.buffers.delete(this.n);
++this.n;
return true;
}
Logs
previewer.79829115fde743502576.js:14 0.18575963718820862 // baseLatency
2previewer.79829115fde743502576.js:14 suspended
previewer.79829115fde743502576.js:14 durationchange
previewer.79829115fde743502576.js:14 loadedmetadata
previewer.79829115fde743502576.js:14 play
previewer.79829115fde743502576.js:14 playing
audioWorklet.js:17 {currentTime: 0, currentFrame: 0, buffers: Map(0)}
audioWorklet.js:46 init
audioWorklet.js:80 1
audioWorklet.js:85 {channel0: Float32Array(128), channel1: Float32Array(128)}
audioWorklet.js:80 2
audioWorklet.js:80 3
audioWorklet.js:80 4
audioWorklet.js:80 5
audioWorklet.js:80 6
audioWorklet.js:80 7
audioWorklet.js:80 8
audioWorklet.js:80 9
audioWorklet.js:80 10
audioWorklet.js:80 11
audioWorklet.js:80 12
audioWorklet.js:80 13
audioWorklet.js:80 14
audioWorklet.js:80 15
audioWorklet.js:80 16
audioWorklet.js:80 17
audioWorklet.js:80 18
audioWorklet.js:80 19
audioWorklet.js:80 20
audioWorklet.js:80 21
audioWorklet.js:80 22
audioWorklet.js:80 23
audioWorklet.js:80 24
audioWorklet.js:80 25
audioWorklet.js:80 26
audioWorklet.js:80 27
audioWorklet.js:80 28
audioWorklet.js:80 29
audioWorklet.js:80 30
audioWorklet.js:80 31
audioWorklet.js:80 32
audioWorklet.js:80 33
audioWorklet.js:80 34
audioWorklet.js:80 35
audioWorklet.js:80 36
audioWorklet.js:80 37
audioWorklet.js:80 38
audioWorklet.js:80 39
audioWorklet.js:80 40
audioWorklet.js:80 41
audioWorklet.js:80 42
audioWorklet.js:80 43
audioWorklet.js:80 44
audioWorklet.js:80 45
audioWorklet.js:80 46
audioWorklet.js:80 47
audioWorklet.js:80 48
audioWorklet.js:80 49
audioWorklet.js:80 50
audioWorklet.js:80 51
audioWorklet.js:80 52
audioWorklet.js:80 53
audioWorklet.js:80 54
audioWorklet.js:80 55
audioWorklet.js:80 56
audioWorklet.js:80 57
audioWorklet.js:80 58
audioWorklet.js:80 59
audioWorklet.js:80 60
audioWorklet.js:80 61
audioWorklet.js:80 62
audioWorklet.js:80 63
Above, 64 (x2) Float32Array
s are created having length
set to 128
to avoid glitches due to the fact that if a Float32Array
is set at outputs
in process()
where the length
is less than 128
the remainder of the TypedArray
remains filled with 0
s.
That means there are at least 128 Float32Array
s set at this.buffers
(Map
), 64 for each channel.
audioWorklet.js:133 0 "process first call" 63 // first call to process()
audioWorklet.js:149 0 63
audioWorklet.js:149 1 62
previewer.79829115fde743502576.js:14 running
audioWorklet.js:149 2 61
audioWorklet.js:149 3 60
audioWorklet.js:149 4 59
audioWorklet.js:149 5 58
audioWorklet.js:149 6 57
audioWorklet.js:149 7 56
audioWorklet.js:149 8 55
audioWorklet.js:149 9 54
audioWorklet.js:149 10 53
audioWorklet.js:149 11 52
audioWorklet.js:149 12 51
audioWorklet.js:149 13 50
audioWorklet.js:149 14 49
audioWorklet.js:149 15 48
audioWorklet.js:149 16 47
audioWorklet.js:149 17 46
audioWorklet.js:149 18 45
audioWorklet.js:149 19 44
audioWorklet.js:149 20 43
audioWorklet.js:149 21 42
audioWorklet.js:149 22 41
audioWorklet.js:149 23 40
audioWorklet.js:149 24 39
audioWorklet.js:149 25 38
audioWorklet.js:149 26 37
audioWorklet.js:149 27 36
audioWorklet.js:149 28 35
audioWorklet.js:149 29 34
audioWorklet.js:149 30 33
audioWorklet.js:149 31 32
audioWorklet.js:149 32 31
However, after 32 calls this.buffers.size
is 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 63 63
process @ audioWorklet.js:170
audioWorklet.js:80 1
audioWorklet.js:80 2
audioWorklet.js:80 3
audioWorklet.js:80 4
audioWorklet.js:80 5
audioWorklet.js:80 6
audioWorklet.js:80 7
audioWorklet.js:80 8
audioWorklet.js:80 9
audioWorklet.js:80 10
audioWorklet.js:80 11
audioWorklet.js:80 12
audioWorklet.js:80 13
audioWorklet.js:80 14
audioWorklet.js:80 15
audioWorklet.js:80 16
audioWorklet.js:80 17
audioWorklet.js:80 18
audioWorklet.js:80 19
audioWorklet.js:80 20
audioWorklet.js:80 21
audioWorklet.js:80 22
audioWorklet.js:80 23
audioWorklet.js:80 24
audioWorklet.js:80 25
audioWorklet.js:80 26
audioWorklet.js:80 27
audioWorklet.js:80 28
audioWorklet.js:80 29
audioWorklet.js:80 30
audioWorklet.js:80 31
audioWorklet.js:80 32
audioWorklet.js:80 33
audioWorklet.js:80 34
audioWorklet.js:80 35
audioWorklet.js:80 36
audioWorklet.js:80 37
audioWorklet.js:80 38
audioWorklet.js:80 39
audioWorklet.js:80 40
audioWorklet.js:80 41
audioWorklet.js:80 42
audioWorklet.js:80 43
audioWorklet.js:80 44
audioWorklet.js:80 45
audioWorklet.js:80 46
audioWorklet.js:80 47
audioWorklet.js:80 48
audioWorklet.js:80 49
audioWorklet.js:80 50
audioWorklet.js:80 51
audioWorklet.js:80 52
audioWorklet.js:80 53
audioWorklet.js:80 54
audioWorklet.js:80 55
audioWorklet.js:80 56
audioWorklet.js:80 57
audioWorklet.js:80 58
audioWorklet.js:80 59
audioWorklet.js:80 60
audioWorklet.js:80 61
audioWorklet.js:80 62
audioWorklet.js:80 63
Another example of logs
running // AudioContext state
audioWorklet.js:133 0 "process first call" 31 // first call to process()
audioWorklet.js:149 0 31
audioWorklet.js:149 1 30
audioWorklet.js:149 2 29
audioWorklet.js:149 3 28
audioWorklet.js:149 4 27
audioWorklet.js:149 5 26
audioWorklet.js:149 6 25
audioWorklet.js:149 7 24
audioWorklet.js:149 8 23
audioWorklet.js:149 9 22
audioWorklet.js:149 10 21
audioWorklet.js:149 11 20
audioWorklet.js:149 12 19
audioWorklet.js:149 13 18
audioWorklet.js:149 14 17
audioWorklet.js:149 15 16
audioWorklet.js:149 16 15
audioWorklet.js:149 17 14
audioWorklet.js:149 18 13
audioWorklet.js:149 19 12
audioWorklet.js:149 20 11
audioWorklet.js:149 21 10
audioWorklet.js:149 22 9
audioWorklet.js:149 23 8
audioWorklet.js:149 24 7
audioWorklet.js:149 25 6
audioWorklet.js:149 26 5
audioWorklet.js:149 27 4
audioWorklet.js:149 28 3
audioWorklet.js:149 29 2
audioWorklet.js:149 30 1
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:80 1
audioWorklet.js:80 2
audioWorklet.js:80 3
audioWorklet.js:80 4
audioWorklet.js:80 5
audioWorklet.js:80 6
audioWorklet.js:80 7
audioWorklet.js:80 8
audioWorklet.js:80 9
audioWorklet.js:80 10
audioWorklet.js:80 11
audioWorklet.js:80 12
audioWorklet.js:80 13
audioWorklet.js:80 14
audioWorklet.js:80 15
audioWorklet.js:80 16
audioWorklet.js:80 17
audioWorklet.js:80 18
audioWorklet.js:80 19
audioWorklet.js:80 20
audioWorklet.js:80 21
audioWorklet.js:80 22
audioWorklet.js:80 23
audioWorklet.js:80 24
audioWorklet.js:80 25
audioWorklet.js:80 26
audioWorklet.js:80 27
audioWorklet.js:80 28
audioWorklet.js:80 29
audioWorklet.js:80 30
audioWorklet.js:80 31
audioWorklet.js:80 32
audioWorklet.js:80 33
audioWorklet.js:80 34
audioWorklet.js:80 35
audioWorklet.js:80 36
audioWorklet.js:80 37
audioWorklet.js:80 38
audioWorklet.js:80 39
audioWorklet.js:80 40
audioWorklet.js:80 41
audioWorklet.js:80 42
audioWorklet.js:80 43
audioWorklet.js:80 44
audioWorklet.js:80 45
audioWorklet.js:80 46
audioWorklet.js:80 47
audioWorklet.js:80 48
audioWorklet.js:80 49
audioWorklet.js:80 50
audioWorklet.js:80 51
audioWorklet.js:80 52
audioWorklet.js:80 53
audioWorklet.js:80 54
audioWorklet.js:80 55
audioWorklet.js:80 56
audioWorklet.js:80 57
audioWorklet.js:80 58
audioWorklet.js:80 59
audioWorklet.js:80 60
audioWorklet.js:80 61
audioWorklet.js:80 62
audioWorklet.js:80 63
audioWorklet.js:149 31 160
audioWorklet.js:149 32 159
This effectively means that when process()
is being executed no other methods defined in extended AudioWorkletProcess
are being executed.
Is that by design?
Or, the result of JavaScript being "single threaded"?
Describe the issue
A clear and concise description of what the problem is in the spec.
For some reason an instance of AudioWorklet
played back twice, after disconnect()
was called on AudioWorklet
and close()
called on AudioContext
.
After setting own requirement to
fetch()
a audio fileResponse.body
to create a ReadableStream
Float32Array
and begin playback of audio file as soon as possible, without awaiting completion of reading the entire fileachieved that requirement approximately 2 days ago using AudioWorklet
at Chromium 81 at localhost.
Today, re-ran the same code (with Web Audio tab at DevTools open) and observed 3 issues
Worklet
instances which are apparently being used by AudioWorklet
are not balanaced in the processing of the audio;AudioWorklet.disconnect()
and AudioContext.close()
and gc()
(exposed via Chromium flag --js-flags=--expose-gc
) the audio played back a second time - meaning the duration of the WAV file is 6:42 yet playback completed at 13:30If you need help on how to use WebAudio, ask on your favorite forum or consider visiting
the WebAudio Slack Channel (register here) or
StackOverflow.
Where Is It
Provide a link to where the problem is in the spec
The relevant sections of the specification which contain language that could possibly explain the implementation behaviour that have been able to locate
https://webaudio.github.io/web-audio-api/#active-source
Every AudioWorkletProcessor has an associated active source flag, initially true. This flag causes the node to be retained in memory and perform audio processing in the absence of any connected inputs.
https://webaudio.github.io/web-audio-api/#AudioWorkletProcessor-attributes
Note: Authors that register a event listener on the "message" event of this port should call close on either end of the MessageChannel (either in the AudioWorkletProcessor or the AudioWorkletNode side) to allow for resources to be collected.
Additional Information
Provide any additional information
Was trying to determine the reason for the same code playing back "choppy" and with decreased playback rate when 2 days earlier the playback was not "choppy" for the first ~30 seconds.
Opened Web Audio tab at DevTools in an attempt to debug the issue. Noticed the imbalanced printing of Render Capacity toggling between ~1% and ~98% - apparently toggling usage of the Worklet
s created by AudioWorklet
constructor.
Was not expecting the audio to be played back a second time after AudioContext.close()
being executed.
Christian Hoene:
in order to support high quality spatial audio rendering, individualized HRTF should be used.
Recently, the AES has standardized the SOFA format, which describes HRTFs.
So far, each browser just uses a default HRTF. I am wondering, whether Web Audio should support the SOFA format and thus individualized HRTFs.
Any plans?
Describe the Issue
See WebAudio/web-audio-api#2106 for a PR to add a pull request template. We're not going to do that for v1, but we will for v2, as mentioned in WebAudio/web-audio-api#2106 (comment)
This is a reminder for us to take that merge request and move it/duplicate it to this repo.
Describe the feature
Define TextEncoderStream()
, and TextDecoderStream()
(and TextEncoder
and TextDecoder
) in AudioWorkletGlobalScope
.
Is there a prototype?
TextEncoder
, TextDecoder
, TextEncoderStream()
, and TextDecoderStream()
are defined in Window
and Worker
scope.
Describe the feature in more detail
Consider a (potentially infinite) raw PCM stream source from fetch()
to set as outputs
within AudioWorletProcessor.process()
. We need to convert text to a TypedArray
which can be achieved with
const controller;
const readable = new ReadableStream({
start(_) {
return controller = _;
}
});
readable
.pipeThrough(new TextEncoderStream())
.pipeTo(new WritableStream({
write(value) {
console.log(value);
},
close() {
console.log('stream closed');
}
}));
port.onMessage.addListener(async message => {
if (message !== 'done') {
controller.enqueue(message);
}
else {
controller.close();
}
});
at Window
, however throws an error in AudioWorkletGlovbalScope
constructor(options) {
super(options);
this.byteSize = 512 * 344 * 60 * 50;
this.memory = new Uint8Array(this.byteSize);
Object.assign(this, options.processorOptions);
console.log(this);
this.port.onmessage = this.appendBuffers.bind(this);
}
async appendBuffers({ data: readable }) {
Object.assign(this, { readable });
const source = {
write: (value, c) => {
if (this.totalBytes < this.byteSize) {
this.memory.set(value, this.readOffset);
this.readOffset = this.readOffset + value.buffer.byteLength;
this.totalBytes = this.readOffset;
} else {
console.log(
value.buffer.byteLength,
this.readOffset,
this.totalBytes
);
}
if (this.totalBytes >= (344 * 512) / 2 && !this.started) {
this.started = true;
this.port.postMessage({ started: this.started });
}
},
close() {
console.log('stopped');
},
};
try {
// throws error
await this.readable
.pipeThrough(new TextEncoderStream())
.pipeTo(new WritableStream(source));
} catch (e) {
console.warn(e);
console.log(this.writeOffset, this.totalBytes);
this.endOfStream();
}
}
a9bc72ac-b7ac-43ba-96b0-9f8e58322a2b:37 ReferenceError: TextEncoderStream is not defined
at AudioWorkletProcessor.appendBuffers (a9bc72ac-b7ac-43ba-96b0-9f8e58322a2b:35)
A nice feature of this API is the ability to write custom nodes for modulation - things like parameterized envelopes or step sequencers w/ parameterized slew. This can be achieved now with a-rate outputs, but often k-rate would work just as well and it would be significantly more efficient.
This could be accomplished by adding outputRate alongside outputChannelCount to AudioWorkletNode options:
new AudioWorkletNode(context, 'ModulationProcessor', {
numberOfInputs: 0,
numberOfOutputs: 1,
outputChannelCount: [1],
outputRate: ['k-rate'],
channelCount: 1,
channelCountMode: 'explicit',
channelInterpretation: 'discrete',
});
Connecting a k-rate output to an a-rate param or to the input of another AudioNode could cause the value to be upscaled by filling a 128 length buffer.
Describe the feature
Route system or specific application audio output through a dedicated audio context intended to be utilized for this purpose.
Is there a prototype?
Yes.
inotify-tools
, pavucontrol
, and other native applications for audio capture, and exposure of captured system audio output to the browser.navigator.mediaDevices.getUserMedia({audio: true})
, using WebTransport
, pactl
, Python.Describe the feature in more detail
No specification or API, including Web Audio API, exists for the specific purpose of capturing, analyzing, or processing system or application audio output.
Media Streams and Capture goal is to capture microphone input, not capture or analyze system or application audio output.
Audio Output Devices API goal is to playback audio, not capture or analyze system or application audio output.
The goal of this proposal is singular:
Use cases for the stated singular goal:
.caf
. and .aiff
by mpv
media player.Permissions are not a necessary
When the user instructs a media player to play a sound that is an affirmative activity. When the user instructs AudioOutputContext
to capture either system audio output, which the default, or specific application audio output to speakers or headphones, that too, is an affirmative activity, requiring no further user action. There is no security concern with capturing system or application audio output that is already being routed to speakers or headphones.
This proposal is the opposite of the description of AudioDestinationNode
[1.10. The
AudioDestinationNode
Interface](1.10. The AudioDestinationNode Interface
This is an
AudioNode
representing the final audio destination and is what the user will ultimately hear. It can often be considered as an audio output device which is connected to speakers.
The language describing AudioOutputContext
can be described as
This is an
AudioNode
representing what the user hears. The default is system audio output capture. Users can set a specific application that is outputting what the user hears to be captured as a live stream.
API default behaviour
const audioOutput = await new AudoOutputContext();
// output
{
sampleSpecification: 'float32le 1ch 25Hz',
stream: <MediaStream>
application: 'system'
}
API specification application behaviour
const audioOutput = await new AudoOutputContext({
application:'mpv'
});
// output
{
application: 'mpv'
sampleSpecification: 'float32le 2ch 48000Hz',
stream: <MediaStream>
}
application
is the default 'system'
, what the user hears being output to speakers or headphones; can include multiple streams that are playing. If set at constructor and the single application is in use set at the constructor playing back audio to speakers or headphones (including line-out) the application process binary of the application, a string.
sampleSpecification
is similar to structure of destination
object of BaseAudioCotext
.
stream
is similar to MediaStreamAudioDestinationNode.stream
, a MediaStream
containing the MediaStreamTrack
of the live audio output, system or specific application.
Extensibility
Prospective
Promise
from the constructor fulfilling, when live audio output commences, or at a specific time in the future, or when the application starts playing.Solutions for other API's
This will provide solutions for other API's, for example, the ability to listen to Google Meet without having to first capture a microphone with navigator.mediaDevices.getUserMedia({audio :true})
to get deviceId
from navigator.mediaDevices.enumerateDevices()
to then use setSinkId
to route audio through the device listed as deviceId
, which can be an incorrect listing at user-agents that do not support capture of non-microphone devices.
This will provide solutions for the ability within the domain of music to listen to live jam sessions without having to request microphone permission in attempts to gain access to audio output devices, where the specification does not proffer that goal and does not specify any means to capture devices other than microphones, which is a non-goal for this API.
This will provide multiple solutions for assistive technologies, including outputting text on screen to vision impaired users, and capturing the output of speech synthesis engines for the ability to effectively transform text to captured audio output and captured audio output to text.
This will provide a means to apply computer generated audio with text to speech or human audio with voice to primary source research, to read aloud and capture original texts and treatises written as recently as two hundred years ago, back through antiquity, for the ability to accurately reproduce primary source documents in audio form.
Audio book creation, editing, publication.
Screen reading.
Capture of audio output from applications of audio codecs that user-agents do not support, for the ability to listen to any audio in the browser processed through a containerless MediaStream
.
That is, this feature will provide solutions for API's that currently do not exist, while the use cases do exist, making this API the cutting edge, both to fill gaps in existing API's that have internally or externally imposed restrictions, and to prospectively incorporate by reference the above-listed use case solutions, each a distinct domain of human activity where capture of audio output is useful in the scope of the respective case context.
Describe the issue
console.log('compileStreaming' in globalThis.WebAssembly);
console.log('instantiateStreaming' in globalThis.WebAssembly);
both return true
at Nightly 78 and Chromium 81.
However, those methods cannot be used in AudioWorkletGlobalScope
without throwing a TypeError
because Response
is not defined in AudioWorkletGlobalScope
https://webassembly.org/docs/web/#process-a-potential-webassembly-response
Process a potential WebAssembly response
The above two functions both reuse much of the same infrastructure for extracting bytes from an appropriateResponse
object, differing only in what they do with those bytes in the end. As such we define the following shared spec-level procedure:...
Given these values, to process a potential WebAssembly response:
- Let returnValue be a new promise.
- Let sourceAsPromise be a promise resolved with argument.
- Upon fulfillment of
sourceAsPromise
with valueunwrappedSource
:
i. If unwrappedSource is not aResponse
object, reject returnValue with aTypeError
exception and abort these substeps.
Specification authors object to defining fetch()
, thus Response
, in AudioWorkletGlobalScope
where the methods now are only capable of throwing a TypeError
per the WebAsssembly
specification, thus should be removed from AudioWorkletGlobalScope.WebAssembly
implementation.
Where Is It
The
AudioWorklet
object allows developers to supply scripts (such as JavaScript or WebAssembly code) to process audio on the rendering thread, supporting customAudioNode
s. This processing mechanism ensures synchronous execution of the script code with other built-inAudioNode
s in the audio graph.
https://webassembly.org/docs/web/#additional-web-embedding-api
Additional Web Embedding API
In Web embeddings, the following methods are added.
Note that it is expected that
compileStreaming
andinstantiateStreaming
be either both present or both absent.
Additional Information
Where specification authors appear to be in firm agreement to not define fetch()
, and thus Response
, in AudioWorkletGlobalScope
#73, where if that decision remains unchanged compileStreaming()
and instantiateStreaming()
will always only throw a TypeError
rendering the methods useless in AudioWorkletGlobalScope
.
The WebAssembly
specification provides the solution to the bug "both absent" must apply in this case.
If the specification authors conclude that a TypeError
being thrown for WebAssembly.compileStreaming()
and WebAssembly.instantiateStreaming()
which is the only possible outcome due to Response
not being defined in AudioWorkletGlobalScope
at a minimum a note should be included in the specification to notify users that while compileStreaming()
and instantiateStreaming()
are defined methods within AudioWorkletGlobalScope
using either will only result in a TypeError
being thrown as Response
is not defined in the AudioWorkletGlobalScope
.
I am developing a 3D environment that connects two microphones with a specific geometry in my scene , runs locally for now.
I want to be able to switch between which microphone is being used ( when I press key A use mic1 and when I press key B use mic 2 ).
The error I get is :
Firefox ----> Error:NotReadableError: Concurrent mic process limit.
Chrome -> no Error, it just doesn't switch devices
How can I fix that ? I tried to stop the stream but it doesn't work , any suggestions ?
Describe the feature
It would be nice if the OfflineAudioContext
could be configured to not allocate any memory for the rendered AudioBuffer in case it is not needed.
Is there a prototype?
No.
Describe the feature in more detail
Sometimes an OfflineAudioContext
is only used to compute a side effect and not really for rendering a buffer.
Let's imagine you have an AudioWorkletNode
which just scans the signal for its maximum value. Its usage might look like this:
const offlineAudioContext = new OfflineAudioContext({ length: 44100, sampleRate: 44100 });
const audioBufferSourceNode = new AudioBufferSourceNode(
offlineAudioContext,
{ buffer: aBufferFromSomewhere }
);
// This could be an AudioWorkletNode which scans the signal for its maximum value.
const maximumValueNode = new MaximumValueNode(offlineAudioContext);
audioBufferSourceNode.start(0);
audioBufferSourceNode
.connect(maximumValueNode)
.connect(offlineAudioContext.destination);
offlineAudioContext
.startRendering()
.then(() => {
// Now that the rendering is done ask the MaximumValueNode for the maximum value.
console.log(maximumValueNode.maximumValue);
});
In the example above the OfflineAudioContext
is only used to drive the processing of the data but the rendered buffer is not really needed. I think it would be beneficial if the buffer doesn't need to be created in any case.
Describe the feature
An alternative to module loading of AudioWorklet
.
Is there a prototype?
The closest that I am aware of is Anonymous inline modules
Describe the feature in more detail
When I run this code https://github.com/guest271314/webtransport/blob/main/webTransportAudioWorkletWebAssemblyMemoryGrow.js at console
on GitHub I get this error
Refused to load the script 'blob:https://github.com/66922005-0952-462d-abbe-a0983c78079f' because it violates the following Content Security Policy directive: "script-src github.githubassets.com". Note that 'script-src-elem' was not explicitly set, so 'script-src' is used as a fallback.
VM187:238 DOMException: The user aborted a request.
webTransportAudioWorkletMemoryGrow @ VM187:238
async function (async)
webTransportAudioWorkletMemoryGrow @ VM187:113
(anonymous) @ VM194:1
which is the Blob URL
at
const worklet = URL.createObjectURL(
new Blob(
[
registerProcessor(
'audio-worklet-quic-transport-stream',
AudioWorkletQuicTransportStream
),
],
{
type: 'text/javascript',
}
)
);
We should be able to run AudioWorklet
using inline code
class AudioWorkletProcessor {}
class AudioWorkletQuicTransportStream extends AudioWorkletProcessor {
// ...
}
to avoid making a fetch request.
Describe the feature
For example, the constructor for OfflineAudioContext
takes an unsigned long
for the length
parameter. This means the actual length is the given length modulo 2^32
, according to https://heycam.github.io/webidl/#abstract-opdef-converttoint
It might be useful if we added the [EnforceRange]
attribute to cause an error to be thrown when the parameter is outside the range.
If this makes sense, we should probably apply this everywhere we use unsigned long
.
It's certainly easier to understand if we get an error when we ask for an OfflineAudioContext
of length 4294967297 and end up with no errors and a context with length 1.
Describe the issue
The description for cancelTime
parameter for both cancelScheduledValues
and cancelAndHoldAtTime
says that non-finite values should throw a RangeError
. However, since the declared type is double
and not unrestricted double
, a TypeError
is thrown instead.
We should update the text to say that a RangeError
is thrown for negative values (deleting the part about non-finite values).
Where Is It
For cancelScheduledValues
, see https://webaudio.github.io/web-audio-api/#dom-audioparam-cancelscheduledvalues-canceltime
For cancelAndHoldAtTime
, see https://webaudio.github.io/web-audio-api/#dom-audioparam-cancelandholdattime-canceltime
Describe the issue
There seems to be no consensus so far on what an audio element should do when its sound is piped into the Web Audio API. Firefox stops playing out the sound from the audio element once it is connected to the Web Audio API. Chrome on the other hand keeps on playing the sound from audio element. Until recently it was possible to manually mute the audio element in Chrome which essentially reproduces what Firefox does automatically. The problem is that both Firefox and Chrome (since a few weeks) stop the audio element internally once it is muted which means the workaround doesn't work anymore in Chrome.
Where Is It
I'm not sure if the behavior is specified already somewhere. It could also be part of the spec which defines the media element or the one for MediaStreams ...
Additional Information
I initially filed a bug for Chrome for the same issue.
Describe the feature
Add computedValue
readonly property to AudioParam.
Is there a prototype?
I attempted to create an AudioWorkletNode which assisted with this but wasn't able to achieve anything useful since there is also no getter for the current computedValue of an AudioNode, and using the MessagePort available was either too intensive or too high latency, and timing too unpredictable to be considered a suitable workaround.
Describe the feature in more detail
It seems that previous inconsistent implementations of the getter on AudioParam.value
in Chrome (which retrieved the computed value after summing any connected inputs) has been removed, in order to follow the spec which states it should return the current interpolated value based on the current automation event (AudioParam.linearRampToValueAtTime
for example).
As stated in WebAudio/web-audio-api#1788, the AudioParam.value
getter is often used to drive visual representations of the parameter, which means incorporating any inputs' contributions is desired. While I understand the need and reasoning behind going with the decision made in WebAudio/web-audio-api#1788, access to the computed value is still very much needed. As far as I can tell there is no straightforward way to access this value anymore.
As suggested in WebAudio/web-audio-api#1788, a readonly property of AudioParam (and ideally AudioNode too) called computedValue
would solve this without introducing ambiguity into the current spec of setters and getters behaviour.
It was brought up that performance concerns might arise depending on the implementation, which is reasonable, but considering those performance issues would have existed in any previous implementations that did this, I don't see this as a major issue. Users of the API will realise if computedGetter causes performance issues and find a workaround if one exists for them, but in many cases, such a getter might be called 60 times a second or less, certainly not at audio rates - performance might not be a great concern for many users of the API, including myself.
Describe the feature
WebCodecs proposal is mentioned at V1 in several issues
This will now be handled by https://discourse.wicg.io/t/webcodecs-proposal/3662
This will now be handled by https://discourse.wicg.io/t/webcodecs-proposal/3662
This will now be handled by https://discourse.wicg.io/t/webcodecs-proposal/3662
Likely to be solved by webcodecs. No response since Oct 2018
Closing this, it is not an area of the api that we will be working on in V2. However we hope that this will be covered by https://github.com/WICG/web-codecs
https://github.com/WICG/web-codecs/blob/master/explainer.md
https://discourse.wicg.io/t/webcodecs-proposal/3662
Don't know how it will work, but having two apis that overlap a little is not a great pattern. It seems more beneficial for everyone if WebCodecs can solve this, rather than grafting some kind of partial solution to decodeAudioData but not be able to do other things that WebCodecs can do. It's up to you and us to do our best to make it happen.
This will not be considered for V2, as it is better placed in webcodecs.
https://github.com/WICG/web-codecs
This is what we're doing in Web Codecs.
- implement what is needed here in js/wasm has been done numerous times before, and works well, allowing authors that need to solve this use case to wait until WebCodec is ready.
Let me also add that we will work closely with the WebCodecs group to make sure that it can provide solutions to the problems people have had with decodeAudioData. Ideally, WebCodecs should be able to do everything that decodeAudioData does and more.
New feature requests go in the v2 repo. New feature requests about audio decoding and encoding go in the Web Codecs repo, but this is use-case has been handled from day one there.
Is there a prototype?
No.
It is not at all clear how WebCodecs will solve any of the issues where Web Audio API specification authors have suggested the same might eventually be possible.
So far there are only unsubstantiated claims without any hint of an algorithm or proof-of-concept in code where using WebCodecs would produce any result different than is currently possible using Web Audio API V1.
Since Web Audio API only processes PCM
1.4. The AudioBuffer Interface
This interface represents a memory-resident audio asset. Its format is non-interleaved 32-bit floating-point linear PCM values with a normal range of [−1,1]
, but values are not limited to this range. It can contain one or more channels. Typically, it would be expected that the length of the PCM data would be fairly short (usually somewhat less than a minute). For longer sounds, such as music soundtracks, streaming should be used with the audio element and MediaElementAudioSourceNode.5.1. Audio sample format
Linear pulse code modulation (linear PCM) describes a format where the audio values are sampled at a regular interval, and where the quantization levels between two successive values are linearly uniform.Whenever signal values are exposed to script in this specification, they are in linear 32-bit floating point pulse code modulation format (linear 32-bit float PCM), often in the form of Float32Array objects.
5.2. Rendering
The range of all audio signals at a destination node of any audio graph is nominally [-1, 1]. The audio rendition of signal values outside this range, or of the values NaN, positive infinity or negative infinity, is undefined by this specification.
without any documentation, example code, or even a concept to prove otherwise the gist of WebCodecs relevant to Web Audio API V1 or V2 would essentially be an arbitrary audio codec input to PCM output. Nothing more can be gleaned from the comments thus far, besides "hope" for some furture capability proferred by WebCodecs that will somehow be connectable to Web Audio API. For individuals that do not entertain hope, rather base conclusions on facts, there are currently no facts available which support the claims made thus far that WebCodecs will provide some means to solve the issues closed that point to WebCodecs proposal as being capable of remedying the problems described in the issues.
It is not clear how WebCodecs will change or modify at all Web Audio API processing model re "5.1. Audio sample format" or "5.2. Rendering", presently making any reference to "This will now be handled by https://discourse.wicg.io/t/webcodecs-proposal/3662" moot, null and void, particularly without any example or basic flow-chart detailing how precisely Web Audio API will connect to WebCodecs.
Describe the feature in more detail
This issue is intended to be a meta thread for WebCodecs prospective connection with the as-yet un-sepecified and un-deployed WebCodecs.
As a concrete example of the problem with simply suggesting WebCodecs will solve a current problem with the Web Audio API in the future, without evidence to support such a claim, following up on WebAudio/web-audio-api#2135 where decodeAudioData()
does not support decoding partial content, leading to memory usage that eventually crashes the tab, was able to cobble together a working example of streaming Opus audio attempting to solve this MediaSource
issue w3c/media-source#245 using Web Audio API by first splitting the file into N 15 second Opus files with ffmpeg
or mkvmerge
then sorting and merging the files again into a single file using File
s
const sorted = [...e.target.files].sort((a, b) =>
+b.name.replace(/\D/g, "") > a.name.replace(/\D/g, "") ? -1 : 0);
const offsets = sorted.map(({size}) => size);
console.log(offsets);
const blob = new Blob(sorted);
const a = document.createElement("a");
a.href = URL.createObjectURL(blob);
a.download = "house_opus_merged";
a.click();
then serving the map of sorted offsets with the merged file
<?php
if ($_GET["opus"]) {
$offsets = "[135567,127418,120207,124575,129572,120537,120918,133856,137974,135627,138015,128146,128113,127626,129090,124280,127461,133903,133995,137177,137409,148202,147645,143197,126504,127171,94884]";
header("Content-Type: application/octet-stream");
header("Content-Description: " . $offsets);
$file = file_get_contents("./house_opus_merged", FILE_USE_INCLUDE_PATH);
echo $file;
}
?>
which allows executing decodeAudioData()
on one of the included N merged parts of the single served file then commencing playback of that audio segment without decoding all of the N parts of the file first - and without discernable gaps in playback (not achieved with first attempt at splitting the file). Included is the first execution of the spittling of the file and original file in attached zip archive.
(async() => {
try {
const request = await fetch("http://localhost:8000?opus=true");
const response = await request;
const offsets = JSON.parse(response.headers.get("Content-Description"));
const inputStream = response.body.getReader();
const audioData = [];
let n = 0;
let offset;
let controller;
const ac = new AudioContext({
numberOfChannels: 2,
sampleRate: 48000
});
const msd = new MediaStreamAudioDestinationNode(ac);
const {
stream
} = msd;
const gainNode = new GainNode(ac, {
gain: 1
});
gainNode.connect(msd);
const [track] = stream.getAudioTracks();
track.enabled = false;
const audio = new Audio();
audio.autoplay = true;
audio.controls = true;
document.body.appendChild(audio);
const outputStream = new ReadableStream({
async start(c) {
return controller = c;
}
});
const audioOutputReader = outputStream.getReader();
async function processAudioOutputStream({
value: sourceBufferSegmentEnded,
done
}) {
if (done) {
await outputStream.closed;
return "processAudioOutputStream done:" + done;
}
const {
segment, trackOffset
} = await sourceBufferSegmentEnded();
if (segment === offsets.length - 1) {
track.stop();
track.disabled = true;
audioOutputReader.cancel();
}
return processAudioOutputStream(await audioOutputReader.read());
}
async function processAudioInputStream({
value, done
}) {
if (done) {
await inputStream.closed;
return "processAudioInputStream done:" + done;
}
offset = 0;
do {
audioData.push(value[offset]);
offset++;
if (audioData.length === offsets[n]) {
const uint8array = new Uint8Array(audioData.splice(0, offsets[n]));
controller.enqueue((props => () => {
return new Promise(async resolve => {
const ab = await ac.decodeAudioData(uint8array.buffer);
const source = ac.createBufferSource();
source.connect(gainNode);
source.buffer = ab;
source.start();
source.onended = _ => {
source.disconnect(gainNode);
resolve(props);
};
if (audio.srcObject !== stream) {
track.enabled = true;
audio.srcObject = stream;
};
})
})({
segment: n,
trackOffset: offsets[n]
}));
n++;
}
} while (offset < value.length);
return processAudioInputStream(await inputStream.read());
}
const input = await processAudioInputStream(await inputStream.read());
console.log({input})
const output = await processAudioOutputStream(await audioOutputReader.read());
console.log({output});
} catch (e) {
console.trace();
}
})();
The problem is twofold
ended
event of source buffer)AudioWorklet
to avoid gaps requires substantial attention to the 128 byte per process()
execution, where when the test is not correctly configured can crash the browser tab and the underlying OS.The suggested solution for this case as evidenced by the closure of similar use cases is now to use WebCodecs. However, it is not immediately clear at all how WebCodecs will overcome Web Audio API "Audio sampling format" and "Rendering" sections. This corresponding subject-matter WebCodecs issue w3c/webcodecs#28 is not resolved.
Given the code example above and cross-referencing to WebCodecs examples https://github.com/WICG/web-codecs/blob/master/explainer.md#examples the only code where Web Audio API would potentially connect to output of WebCodecs is by way of a MediaStreamTrack
, not PCM
// Finally the decoders are created and the encoded media is piped through the decoder
// and into the TrackerWriters which converts them into MediaStreamTracks.
const audioDecoder = new AudioDecoder({codec: 'opus'});
The question then becomes would the output MediaStreamTrack
be connect()
ed to a an audio node? If no post-processing or output modification is required why would Web Audio API be used at all in that case?
Given the above example code that combines N files into a single file to preserve metadata for each file for decodeAudioData()
what pattern would fulfill the suggestion that WebCodecs would solve the problems described in the linked closed issues that refer to WebCodecs?
Ideally we should be able to parse any input file having any audio codec to AudioWorklet
as Float32Array
without having to also use MediaStreamTrack
, or again, why use Web Audio API at all in that case where we can simply audio.srcObject = new MediaStream([await audioDecorder])
?
Is
audioTrackNode = new MediaStreamTrackAudioSourceNode(context, {mediaStreamTrack:await audioDecoder});
expected to be used for connect to WebCodecs, where we could stream the Opus file without splitting the file into N parts - and playback will begin without parsing the entire input, especially where the input might not have a definitive end - even though that method is only available at Firefox?
The answer to, but not limited to the above questions, relevant to the reference to WebCodecs dependence, reliance or referral should be clear.
Re
This is what we're doing in Web Codecs.
Can the specification authors who have made the above claims kindly clearly write out precisely what you are "doing" re WebCodecs being connectable in any meaningful way to Web Audio API (V1 and V2)?
Let's say you have a very simple AudioWorklet
having a single input and a single output with no AudioParams
. Let's say you made a mistake and have
process(inputs, outputs, params) {
let input = inputs[0];
let output = outputs[0];
output.fill(input[0][0]);
return true;
}
This accidentally modifies outputs[0]
to be an array of floats. But it's supposed to be a sequence<Float32Array>
.
How is the worklet supposed to handle this? I'm kind of thinking this should cause an error to be thrown. I guess that also implies that if output
isn't exactly sequence<sequence<Float32Array>>
an error is thrown. Similarly, if the length of the Float32Array
isn't 128, it's an error.
(In case it matters, I actually did this by accident.)
Describe the feature
const ac = new AudioContext();
const aw = new AudioWorkletNode(ac, 'processor');
const {body: readable} = await fetch('/path/to/raw_pcm_output');
const rawPCMStreamNode = new RawPCMStreamNode(ac, readable);
aw.connect(rawPCMStreamNode);
Is there a prototype?
Yes. Rather involved, illustrating some of the deficiencies of SharedArrayBuffer
and even dynamic SharedArrayBuffer
memory growth using WebAssembly.Memory.prototype.grow()
, thus this feature request to simplify the process.
From wasmerio/wasmer-php#121 (comment)
Put together a minimal, verifiable, complete working example of streaming from
fetch()
toAudioWorkletProcessor
via Native Messaging, including the observable restriction which lead to filing this issue, and those issues referenced above, specifically,RangeError
being thrown, and limitations on the amount of memory that can be pre-allocated at anArrayBuffer
orSharedArrayBuffer
.Posting the code here as not infrequently crash the browser and, or the OS in the midst of ongoing experiments.
Flow chart
- Create Native Messaging Chromium app (app)
- At an arbitrary web page ynamically set permissions for the app using Native File System to re-write manifest.json in app/ folder by setting the property
externally_connectable
value- Connect to the backgound script, background.js, that passes messages between the Native Messaging host (in this case we use bash and execute PHP within the shell script) and the arbitrary web page
- We do not use the Native Messaging protocol to pass STDOUT to background.js then to the web page, instead we begin PHP built-in server process, localhost, to handle the subsequent call to
fetch()
withAbortController
set within object passed as send parameter where we read STDOUT as aReadableStream
, converting to and fromJSON
andUint8Array
- Create a
MediaStreamAudioDestinationNode
for the ability to record, perform other tasks withMediaStream
, andMediaStreamTrack
of live audio stream of system audio capture- Use Transferable Streams to post
response.body
(ReadableStream
) toAudioWorkletProcessor
, where we pipe the stream toWritableStream
and writeUint8Array
s to a singleUint8Array
pre-allocated to a set amount (344 * 512 * 60 * 60
throws cannot allocate memory, here)- Store minimal data before
process()
is started by resumingAudioContext
- Pass message to background.js to kill PHP built-in server process
- Close
ReadableStreamDefaultController
Chromium Native Messaging extenstion.
app/
manifest.json
{ // Extension ID: <id> "key": "<key>", "name": "native_messaging_stream", "version": "1.0", "manifest_version": 2, "description": "Capture system audio", "icons": {}, "permissions": [ "nativeMessaging", "*://localhost/" ], "app": { "background": { "scripts": [ "background.js" ], "persistent": false } }, "externally_connectable": { "matches": [ "https://developer.mozilla.org/*", "http://localhost/*" ], "ids": [ "lmkllhfhgnmbcmhdagfhejkpicehnida" ] }, "author": "guest271314" }
background.js
const id = 'native_messaging_stream'; let externalPort, controller, signal; chrome.runtime.onConnectExternal.addListener(port => { console.log(port); externalPort = port; externalPort.onMessage.addListener(message => { if (message === 'start') { chrome.runtime.sendNativeMessage(id, {}, async _ => { console.log(_); if (chrome.runtime.lastError) { console.warn(chrome.runtime.lastError.message); } controller = new AbortController(); signal = controller.signal; // wait until bash script completes, server starts for await (const _ of (async function* stream() { while (true) { try { if ((await fetch('http://localhost:8000', { method: 'HEAD' })).ok) break; } catch (e) { console.warn(e.message); yield; } } })()); try { const response = await fetch('http://localhost:8000?start=true', { cache: 'no-store', mode: 'cors', method: 'get', signal }); console.log(...response.headers); const readable = response.body; readable .pipeTo( new WritableStream({ write: async value => { // value is a Uint8Array, postMessage() here only supports cloning, not transfer externalPort.postMessage(JSON.stringify(value)); }, }) ) .catch(err => { console.warn(err); externalPort.postMessage('done'); }); } catch (err) { console.error(err); } }); } if (message === 'stop') { controller.abort(); chrome.runtime.sendNativeMessage(id, {}, _ => { if (chrome.runtime.lastError) { console.warn(chrome.runtime.lastError.message); } console.log('everything should be done'); }); } }); });
set_externally_connectable.js
(async(set_externally_connectable = ["https://example.com/*"], unset_externally_connectable = true) => { const dir = await self.showDirectoryPicker(); const status = await dir.requestPermission({writable: true}); const fileHandle = await dir.getFileHandle("manifest.json", {create: false}); const file = await fileHandle.getFile(); const manifest_text = await file.text(); const match_extension_id = /\/\/ Extension ID: \w{32}/; const [extension_id] = manifest_text.match(match_extension_id); let text = manifest_text.replace(match_extension_id, `"_": 0,`); const manifest_json = JSON.parse(text); manifest_json.externally_connectable.matches = unset_externally_connectable ? set_externally_connectable : [...manifest_json.externally_connectable.matches, ...set_externally_connectable]; const writer = await fileHandle.createWritable({keepExistingData:false}); await writer.write(JSON.stringify(manifest_json, null, 2).replace(/"_": 0,/, extension_id)); return await writer.close(); })([`${location.origin}/*`]);
host/
native_messaging_stream.json (
$ cp native_messaging_stream.json ~/.config/chromium/NativeMessagingHosts
){ "name": "native_messaging_file_stream", "description": "Capture system audio", "path": "/path/to/host/captureSystemAudio.sh", "type": "stdio", "allowed_origins": [ "chrome-extension://<id>/" ], "author": "guest271314" }
index.php
<?php if($_SERVER['REQUEST_METHOD'] == 'HEAD') { header('Vary: Origin'); header("Access-Control-Allow-Origin: chrome-extension://lmkllhfhgnmbcmhdagfhejkpicehnida"); header("Access-Control-Allow-Methods: GET, OPTIONS, HEADERS"); header("Access-Control-Allow-Headers: Content-Type, Access-Control-Allow-Headers"); header("X-Powered-By:"); header("HTTP/1.1 200 OK"); die(); } if (isset($_GET["start"])) { header('Vary: Origin'); header("Access-Control-Allow-Origin: chrome-extension://lmkllhfhgnmbcmhdagfhejkpicehnida"); header("Access-Control-Allow-Methods: GET, OPTIONS, HEAD"); header("Access-Control-Allow-Headers: Content-Type, Access-Control-Allow-Headers"); header("Content-Type: text/plain"); header("X-Powered-By:"); echo passthru("parec -v --raw -d alsa_output.pci-0000_00_1b.0.analog-stereo.monitor"); exit(); }
captureSystemAudio.sh
#!/bin/bash sendMessage() { # https://stackoverflow.com/a/24777120 message='{"message": "ok"}' # Calculate the byte size of the string. # NOTE: This assumes that byte length is identical to the string length! # Do not use multibyte (unicode) characters, escape them instead, e.g. # message='"Some unicode character:\u1234"' messagelen=${#message} # Convert to an integer in native byte order. # If you see an error message in Chrome's stdout with # "Native Messaging host tried sending a message that is ... bytes long.", # then just swap the order, i.e. messagelen1 <-> messagelen4 and # messagelen2 <-> messagelen3 messagelen1=$(( ($messagelen ) & 0xFF )) messagelen2=$(( ($messagelen >> 8) & 0xFF )) messagelen3=$(( ($messagelen >> 16) & 0xFF )) messagelen4=$(( ($messagelen >> 24) & 0xFF )) # Print the message byte length followed by the actual message. printf "$(printf '\\x%x\\x%x\\x%x\\x%x' \ $messagelen1 $messagelpen2 $messagelen3 $messagelen4)%s" "$message" } captureSystemAudio() { if pgrep -f php > /dev/null; then killall -9 php & sendMessage else php -S localhost:8000 -t /path/to/host/ & sendMessage fi } captureSystemAudio
at the arbitrary web page, in this case, MDN, where we previously set the matching URL pattern at
externally_connectable
for permission to communicate with the Chromium Native Messaging applicationAt web page
var id = 'lmkllhfhgnmbcmhdagfhejkpicehnida'; var port = chrome.runtime.connect(id); var controller; var readable = new ReadableStream({ start(c) { return (controller = c); }, }); var init = false; port.onMessage.addListener(async message => { if (message !== 'done') { if (!init) { init = true; class AudioWorkletProcessor {} class AudioWorkletNativeMessageStream extends AudioWorkletProcessor { constructor(options) { super(options); this.byteSize = 512 * 384 * 60 * 1; // 5 minutes of data this.memory = new Uint8Array(this.byteSize); // TODO: grow memory, dynamically Object.assign(this, options.processorOptions); this.port.onmessage = this.appendBuffers.bind(this); } async appendBuffers({ data: readable }) { Object.assign(this, { readable }); const source = { write: (value, controller) => { console.log(globalThis.currentTime % 3); if (this.totalBytes + value.byteLength < this.byteSize) { this.memory.set(value, this.readOffset); this.readOffset = this.readOffset + value.buffer.byteLength; this.totalBytes = this.readOffset; } else { const last = value.subarray(0, this.byteSize - this.totalBytes); this.memory.set(last, this.readOffset); this.readOffset = this.readOffset + last.length; this.totalBytes = this.readOffset; console.log( value.buffer.byteLength, this.readOffset, this.totalBytes ); controller.close(); } // await 250 milliseconds of audio data if (this.totalBytes >= (512 * 384) / 4 && !this.started) { this.started = true; this.port.postMessage({ started: this.started }); } }, close() { console.log('stopped'); }, }; try { await this.readable.pipeTo(new WritableStream(source)); } catch (e) { console.warn(e); console.log(this.writeOffset, this.totalBytes); this.endOfStream(); } } endOfStream() { this.port.postMessage({ ended: true, currentTime, currentFrame, readOffset: this.readOffset, writeOffset: this.writeOffet, totalBytes: this.totalBytes, }); } process(inputs, outputs) { const channels = outputs.flat(); if ( this.writeOffset >= this.totalBytes || this.totalBytes === this.byteSize ) { console.log(this); this.endOfStream(); return false; } const uint8 = new Uint8Array(512); try { for (let i = 0; i < 512; i++, this.writeOffset++) { if (this.writeOffset === this.byteSize) { break; } uint8[i] = this.memory[this.writeOffset]; } const uint16 = new Uint16Array(uint8.buffer); // https://stackoverflow.com/a/35248852 for (let i = 0, j = 0, n = 1; i < uint16.length; i++) { const int = uint16[i]; // If the high bit is on, then it is a negative number, and actually counts backwards. const float = int >= 0x8000 ? -(0x10000 - int) / 0x8000 : int / 0x7fff; // interleave channels[(n = ++n % 2)][!n ? j++ : j - 1] = float; } // console.log(channels[0]); } catch (e) { console.error(e); } return true; } } // register processor in AudioWorkletGlobalScope function registerProcessor(name, processorCtor) { return `${processorCtor};\nregisterProcessor('${name}', ${processorCtor.name});`; } const worklet = URL.createObjectURL( new Blob( [ registerProcessor( 'audio-worklet-native-message-stream', AudioWorkletNativeMessageStream ), ], { type: 'text/javascript' } ) ); const ac = new AudioContext({ latencyHint: 1, sampleRate: 44100, numberOfChannels: 2, }); ac.onstatechange = e => console.log(ac.state); if (ac.state === 'running') { await ac.suspend(); } await ac.audioWorklet.addModule(worklet); const aw = new AudioWorkletNode(ac, 'audio-worklet-native-message-stream', { numberOfInputs: 1, numberOfOutputs: 2, channelCount: 2, processorOptions: { totalBytes: 0, readOffset: 0, writeOffset: 0, done: false, ended: false, started: false, }, }); aw.onprocessorerror = e => { console.error(e); console.trace(); }; const msd = new MediaStreamAudioDestinationNode(ac); const { stream } = msd; const [track] = stream.getAudioTracks(); aw.connect(msd); const recorder = new MediaRecorder(stream); recorder.ondataavailable = e => console.log(URL.createObjectURL(e.data)); aw.port.onmessage = async e => { console.log(e.data); if ( e.data.started && ac.state === 'suspended' && recorder.state === 'inactive' ) { recorder.start(); await ac.resume(); setTimeout(_ => { port.postMessage('stop'); controller.close(); }, 1000 * 60 * 1); } else if (recorder.state === 'recording') { recorder.stop(); track.stop(); aw.disconnect(); msd.disconnect(); await ac.close(); console.log(track); gc(); } }; aw.port.postMessage(readable, [readable]); } if (readable.locked) controller.enqueue(Uint8Array.from(Object.values(JSON.parse(message)))); } else { if (readable.locked) controller.close(); } }); port.postMessage('start');
A version using
WebAssembly.Memory.grow()
to dynamically grow memory
during the stream. The expected result is 30 minutes of audio being written to
SharedArrayBuffer
. An error is thrown atTesting at a legacy 4.15.0-20-lowlatency 32-bit kernel attempted to grow memory to 30 minutes of data, with maximum set to 1 hour of data
An error was thrown when calling grow(1) when current offset is 177143808before grow 177012736 after grow 177078272 before grow 177078272 after grow 177143808 before grow 177143808 e4056867-5209-4bea-8f76-a86106ab83af:75 RangeError: WebAssembly.Memory.grow(): Unable to grow instance memory. at Memory.grow (<anonymous>) at Object.write (e4056867-5209-4bea-8f76-a86106ab83af:26) appendBuffers @ e4056867-5209-4bea-8f76-a86106ab83af:75 async function (async) appendBuffers @ e4056867-5209-4bea-8f76-a86106ab83af:73 177111040 177139656 {ended: true, currentTime: 1004.0308390022676, currentFrame: 44277760, readOffset: 177139656, writeOffset: 177111040, …} MediaStreamTrack {kind: "audio", id: "e0958602-f0ea-4565-9e34-f624afce7c12", label: "MediaStreamAudioDestinationNode", enabled: true, muted: false, …} blob:https://developer.mozilla.org/f9d716ff-c65e-4ddf-b7c7-5e385d0602ec closed
resulting in 16 minutes and 43 seconds of data being written to memory, observable at recorded audio
Asked the question if 1GB is the maximum the underlying memory can grow at 32-bit architectures at https://bugs.chromium.org/p/v8/issues/detail?id=7881.
if (globalThis.gc) gc(); var id = 'lmkllhfhgnmbcmhdagfhejkpicehnida'; var port = chrome.runtime.connect(id); var controller; var readable = new ReadableStream({ start(c) { return (controller = c); }, }); var init = false; port.onMessage.addListener(async message => { if (message !== 'done') { if (!init) { init = true; class AudioWorkletProcessor {} class AudioWorkletNativeMessageStream extends AudioWorkletProcessor { constructor(options) { super(options); this.initial = (384 * 512) / 65536; // 1 minute this.maximum = (384 * 512 * 60 * 60) / 65536; // 1 hour this.byteSize = this.maximum * 65536; this.memory = new WebAssembly.Memory({ initial: this.initial, maximum: this.maximum, shared: true, }); Object.assign(this, options.processorOptions); this.port.onmessage = this.appendBuffers.bind(this); } async appendBuffers({ data: readable }) { Object.assign(this, { readable }); const source = { write: (value, controller) => { if ( this.totalBytes + value.byteLength > this.memory.buffer.byteLength && this.totalBytes + value.buffer.byteLength < this.byteSize ) { console.log('before grow', this.memory.buffer.byteLength); this.memory.grow(1); console.log('after grow', this.memory.buffer.byteLength); } const uint8_sab = new Uint8Array(this.memory.buffer, this.readOffset); // console.log(this.totalBytes, this.totalBytes % 65536, this.totalBytes / 65536, this.memory.buffer.byteLength); if (this.totalBytes + value.buffer.byteLength < this.byteSize) { for ( let i = 0; i < value.buffer.byteLength; i++, this.readOffset++ ) { uint8_sab[this.readOffset] = value[i]; } this.totalBytes = this.readOffset; } else { const lastBytes = value.subarray(0, this.byteSize - this.totalBytes); // const uint8 = new Uint8Array(this.memory.buffer, this.readOffset); for ( let i = 0; i < lastBytes.buffer.byteLength; i++, this.readOffset++ ) { uint8_sab[this.readOffset] = value[i]; } this.totalBytes = this.readOffset; console.log( value.buffer.byteLength, this.readOffset, this.totalBytes ); this.readable.cancel(); } // accumulate 250 milliseconds of data before resuming AudioContext if (this.totalBytes >= (512 * 384) / 4 && !this.started) { this.started = true; this.port.postMessage({ started: this.started }); } }, close() { console.log('stream closed'); }, }; try { await this.readable.pipeTo(new WritableStream(source)); } catch (e) { console.warn(e); console.log(this.writeOffset, this.totalBytes); this.endOfStream(); } } endOfStream() { this.port.postMessage({ ended: true, currentTime, currentFrame, readOffset: this.readOffset, writeOffset: this.writeOffset, totalBytes: this.totalBytes, }); } process(inputs, outputs) { const channels = outputs.flat(); if ( this.writeOffset >= this.totalBytes || this.totalBytes === this.byteSize ) { console.log(this); this.endOfStream(); return false; } const uint8 = new Uint8Array(512); const uint8_sab = new Uint8Array(this.memory.buffer, this.writeOffset); // .slice(this.writeOffset, this.writeOffset + 512) try { for (let i = 0; i < 512; i++, this.writeOffset++) { if (this.writeOffset === this.byteSize) { break; } uint8[i] = uint8_sab[this.writeOffset]; } const uint16 = new Uint16Array(uint8.buffer); // https://stackoverflow.com/a/35248852 for (let i = 0, j = 0, n = 1; i < uint16.length; i++) { const int = uint16[i]; // If the high bit is on, then it is a negative number, and actually counts backwards. const float = int >= 0x8000 ? -(0x10000 - int) / 0x8000 : int / 0x7fff; // interleave channels[(n = ++n % 2)][!n ? j++ : j - 1] = float; } // console.log(channels[0]); } catch (e) { console.error(e); } return true; } } // register processor in AudioWorkletGlobalScope function registerProcessor(name, processorCtor) { return `${processorCtor};\nregisterProcessor('${name}', ${processorCtor.name});`; } const worklet = URL.createObjectURL( new Blob( [ registerProcessor( 'audio-worklet-native-message-stream', AudioWorkletNativeMessageStream ), ], { type: 'text/javascript' } ) ); const ac = new AudioContext({ latencyHint: 1, sampleRate: 44100, numberOfChannels: 2, }); ac.onstatechange = e => console.log(ac.state); if (ac.state === 'running') { await ac.suspend(); } await ac.audioWorklet.addModule(worklet); const aw = new AudioWorkletNode( ac, 'audio-worklet-native-message-stream', { numberOfInputs: 1, numberOfOutputs: 2, channelCount: 2, processorOptions: { totalBytes: 0, readOffset: 0, writeOffset: 0, done: false, ended: false, started: false, }, } ); aw.onprocessorerror = e => { console.error(e); console.trace(); }; const msd = new MediaStreamAudioDestinationNode(ac); const { stream } = msd; const [track] = stream.getAudioTracks(); aw.connect(msd); const recorder = new MediaRecorder(stream); recorder.ondataavailable = e => console.log(URL.createObjectURL(e.data)); aw.port.onmessage = async e => { console.log(e.data); if ( e.data.started && ac.state === 'suspended' && recorder.state === 'inactive' ) { recorder.start(); await ac.resume(); setTimeout(_ => { port.postMessage('stop'); controller.close(); }, 1000 * 60 * 30); } else if (recorder.state === 'recording') { recorder.stop(); track.stop(); aw.disconnect(); msd.disconnect(); await ac.close(); console.log(track); gc(); } }; aw.port.postMessage(readable, [readable]); } if (readable.locked) controller.enqueue(Uint8Array.from(Object.values(JSON.parse(message)))); } else { if (readable.locked) controller.close(); } }); port.postMessage('start');
Ideally, in this case, we should be able to stream directly to
AudioWorkletProcessor.process()
instead of writing to memory, i.e., the gist of the proposal at Chromium: Use Streams API forSTDIN
andSTDOUT
, flushing memory as we proceed.One approach to reduce memory usage would be to write Opus audio to memory and decode to raw PCM as
Float32Array
values atoutputs
channels, which have not yet tried.
Describe the feature in more detail
Internally RawPCMStreamNode
(or simply StreamNode
or InputStreamNode
) is a TransformStream
that accepts a ReadableStream
(readable
) of raw PCM, transforms the raw data (the terms "interleaved" and "planar" are no longer included in Web Audio API v1) into Float32Array
(s) corresponding to the number of channels contained in the input stream and sets the Float32Array
(s) as outputs
in process()
.
Externally his will work in the same way as MediaElementSourceNode
<body>
<audio controls autoplay crossOrigin="anonymous"></audio>
<script>
(async() => {
const context = new AudioContext({latencyHint:1.0});
await context.suspend();
const mediaElement = document.querySelector("audio");
mediaElement.onloadedmetadata = async e => await context.resume();
const source = new MediaElementAudioSourceNode(context, {
mediaElement
});
await context.audioWorklet.addModule("bypass-processor.js");
const bypasser = new AudioWorkletNode(context, "bypass-processor");
source.connect(bypasser);
bypasser.connect(context.destination);
mediaElement.src = "https://ia800301.us.archive.org/10/items/DELTAnine2013-12-11.WAV/Deltanine121113Pt3Wav.wav";
})();
</script>
</body>
where when connected to an AudioWorkletNode
AudioWorkletProcessor.process()
effectively takes input from src
of HTMLMediaElement
and resulting audio pipeline and outputs Float32Array
s at outputs
.
This will provide an entry point for users to utilize AudioContext
for non-standard or non-implemented device capture and processing with Web Audio API.
WebCodecs does not solve this - unless that solution can be unequivocally proven here by evidence, not conjecture.
Related art:
Related issue:
Describe the feature
encodeAudioData
as a complement to the existing decodeAudioData
method on AudioContext
. This is useful if you want to do things like record audio into the browser from the microphone, apply effects using the Web Audio API and then create a file that people can take away for use in audio editing/playback software outside the browser.
Is there a prototype?
No, but there are a number of libraries which implement similar behavior using either WASM or DataView
s in JavaScript.
For example: https://github.com/Jam3/audiobuffer-to-wav
Describe the feature in more detail
The function would take a buffer instance, and return an ArrayBuffer representing the contents of the encoded file: encodeAudioData(buffer: AudioBuffer, options: EncodeAudioDataOptions) => ArrayBuffer
The options could allow developers to specify the details of various encoding targets. To begin with, having simple encoding to .wav
would be immensely helpful. It would be great if other compressed encodings were also available. For example, mp3
might be a good option now that the patent has run out.
Why have it in the spec?
This could be handled by libraries, but having this built into the spec would avoid having to download large libraries/WASM blobs, and would open the door to more performant implementations.
Unless I'm mistaken, it seems the Web Audio API spec does not define a standard way to manually terminate an OfflineAudioContext.
The closest we have is calling suspend
on an already running OfflineAudioContext, which does not dispose the context however, and will accumulate used memory. close
is specced only to AudioContext (not AudioContextBase or OfflineAudioContext), so we can't use that here.
Shouldn't closing be possible on OfflineAudioContext as well?
Our use case is a DAW that supports hour long compositions causing an OfflineAudioContext.startRendering
call to sometimes take up an entire hour, and it would be nice to offer users a way to cancel the process.
The only workaround I know about at this time is window.location.reload()
, and thus perhaps running the rendering process in an iframe
, which can be reloaded without reloading my entire app.
How can one stream music files while a) setting custom loop points for the loops to generate a seamless looping audio and b) adjusting (by possibly animating) the pitch of the playback?
Currently one can download the music file and .decodeAudioData() it fully, and then use AudioBufferSourceNode and its .loopStart, .loopEnd and .playbackRate attributes to specify the loop points and the pitch. However this consumes 100MB's + of memory and takes an awfully long time to perform the decoding because everything has to be uncompressed in memory for it to work, so it is not a viable solution.
If one uses a MediaElementAudioSourceNode, it is possible to pass the compressed music file directly for playback, and loop it, but looping is restricted to fully looping from begin to the end, which is not enough. It is either not possible to adjust pitch of the audio output (the .playbackRate performs pitch correction for that case, which is undesirable)
If AudioBufferSourceNode was able to be populated with compressed audio data, it would solve this use case. The Web Audio API would then decompress the AudioBufferSourceNode contents while it's playing it back in a streaming manner without having to decode the whole clip in advance. Would that sound like a feasible addition?
(Note that the interest here is in streamed decompression, not in streamed download - the compressed audio assets have been fully downloaded on the page prior)
Describe the feature
The comparision between using transferring a single stream (ReadableStream
; WritableStream
) and transferring hundreds of thousands of TypedArray
s using MessagePort.postMessage()
is not even close.
AudioWorklet
invariably outputs multiple gaps of silence at both Chromium and Firefox when thousands of messages are posted/TypedArray
s transferred.
Is there a prototype?
Yes.
Describe the feature in more detail
See https://bugs.chromium.org/p/chromium/issues/detail?id=910471#c11.
Posting a single ReadableStream
, e.g., from a TransformStream
substantially reduces observable gaps in playback to 1 (that bug is currently isolated to being observable at the end of reading/writing the stream, not the underlying network request in the code used to get the stream) from multiple gaps randomly occurring during playback https://bugzilla.mozilla.org/show_bug.cgi?id=1629384#c24.
Compare for yourself https://plnkr.co/edit/nECtUZ (transferable streams implemented at Chromium behind --enable-experimental-web-platform-features
); https://plnkr.co/edit/yh5A66UhMnlpq0JF (thousands of postMessage(<TypedArray>, [<TypedArray.buffer>])
calls).
Describe the Issue
Need to setup github/travis to display the spec. Currently there's nothing.
Expected Result
https://webaudio.github.io/web-audio-api-v2 should display the v2 version of the spec (that currently has no new content).
Actual Result
404
Describe the issue
A clear and concise description of what the problem is in the spec.
What happens when Float32Array
set at AudioWorkletProcessor
output is less than 128
sample frames?
Where Is It
2.4. Rendering an Audio Graph
Rendering an audio graph is done in blocks of 128 samples-frames. A block of 128 samples-frames is called a render quantum, and the render quantum size is 128.
Additional Information
Provide any additional information
Could not locate any section within the specification where the result of input less than 128
length
is described.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.