webaudio / web-audio-api-v2 Goto Github PK

Describe the feature

Route system or specific application audio output through a dedicated audio context intended to be utilized for this purpose.

Is there a prototype?

Yes.

• captureSystemAudio includes several workarounds for the use case of capturing system audio output, using Native Messaging, File System Access API, inotify-tools, pavucontrol, and other native applications for audio capture, and exposure of captured system audio output to the browser.
• setUserMediaAudioSource includes workarounds to list audio devices that might not be exposed by API's and to dynamically set the source of an audio stream being capture with navigator.mediaDevices.getUserMedia({audio: true}), using WebTransport, pactl, Python.

Describe the feature in more detail

No specification or API, including Web Audio API, exists for the specific purpose of capturing, analyzing, or processing system or application audio output.

• Media Streams and Capture goal is to capture microphone input, not capture or analyze system or application audio output.

• Audio Output Devices API goal is to playback audio, not capture or analyze system or application audio output.

The goal of this proposal is singular:

1. Capture system and specification application audio output.

Use cases for the stated singular goal:

1. Assistive technologies for vision and speech impaired persons.
2. Screen reading.
3. Audio book creation and publication.
4. Primary source document narration coupled with simultaneous or post-video capture of the source document text.
5. System and application audio output analysis.
6. Ability to listen to live meetings, conference calls, music sessions, as an observer, invited attendee, artist or producer, where a microphone is not necessary for joining or participating in the discussion or jam session, or where no microphone is connected to the machine, yet the ability to listen to a live audio output stream is nonetheless valuable in, but not limited to, the above cases.
7. Capture, analysis, processing of specific live application output, where the user-agent is not capable of playing the audio, for example, playback of a .caf. and .aiff by mpv media player.

Permissions are not a necessary

When the user instructs a media player to play a sound that is an affirmative activity. When the user instructs AudioOutputContext to capture either system audio output, which the default, or specific application audio output to speakers or headphones, that too, is an affirmative activity, requiring no further user action. There is no security concern with capturing system or application audio output that is already being routed to speakers or headphones.

This proposal is the opposite of the description of AudioDestinationNode

[1.10. The AudioDestinationNode Interface](1.10. The AudioDestinationNode Interface

This is an AudioNode representing the final audio destination and is what the user will ultimately hear. It can often be considered as an audio output device which is connected to speakers.

The language describing AudioOutputContext can be described as

This is an AudioNode representing what the user hears. The default is system audio output capture. Users can set a specific application that is outputting what the user hears to be captured as a live stream.

API default behaviour

const audioOutput = await new AudoOutputContext();
// output
{ 
  sampleSpecification: 'float32le 1ch 25Hz', 
  stream: <MediaStream>
  application: 'system'
} 

API specification application behaviour

const audioOutput = await new AudoOutputContext({
  application:'mpv'
});
// output
{ 
  application: 'mpv'
  sampleSpecification: 'float32le 2ch 48000Hz', 
  stream: <MediaStream>
} 

application is the default 'system', what the user hears being output to speakers or headphones; can include multiple streams that are playing. If set at constructor and the single application is in use set at the constructor playing back audio to speakers or headphones (including line-out) the application process binary of the application, a string.

sampleSpecification is similar to structure of destination object of BaseAudioCotext.

stream is similar to MediaStreamAudioDestinationNode.stream, a MediaStream containing the MediaStreamTrack of the live audio output, system or specific application.

Extensibility

Prospective

1. When audio is not streaming, no audio output is currently live, the ability to set system or application capture to commence, resulting in the Promise from the constructor fulfilling, when live audio output commences, or at a specific time in the future, or when the application starts playing.

Solutions for other API's

• This will provide solutions for other API's, for example, the ability to listen to Google Meet without having to first capture a microphone with navigator.mediaDevices.getUserMedia({audio :true}) to get deviceId from navigator.mediaDevices.enumerateDevices() to then use setSinkId to route audio through the device listed as deviceId, which can be an incorrect listing at user-agents that do not support capture of non-microphone devices.

• This will provide solutions for the ability within the domain of music to listen to live jam sessions without having to request microphone permission in attempts to gain access to audio output devices, where the specification does not proffer that goal and does not specify any means to capture devices other than microphones, which is a non-goal for this API.

• This will provide multiple solutions for assistive technologies, including outputting text on screen to vision impaired users, and capturing the output of speech synthesis engines for the ability to effectively transform text to captured audio output and captured audio output to text.

• This will provide a means to apply computer generated audio with text to speech or human audio with voice to primary source research, to read aloud and capture original texts and treatises written as recently as two hundred years ago, back through antiquity, for the ability to accurately reproduce primary source documents in audio form.

• Audio book creation, editing, publication.

• Screen reading.

• Capture of audio output from applications of audio codecs that user-agents do not support, for the ability to listen to any audio in the browser processed through a containerless MediaStream.

That is, this feature will provide solutions for API's that currently do not exist, while the use cases do exist, making this API the cutting edge, both to fill gaps in existing API's that have internally or externally imposed restrictions, and to prospectively incorporate by reference the above-listed use case solutions, each a distinct domain of human activity where capture of audio output is useful in the scope of the respective case context.

Describe the issue
A clear and concise description of what the problem is in the spec.

What happens when Float32Array set at AudioWorkletProcessor output is less than 128 sample frames?

Where Is It

2.4. Rendering an Audio Graph
Rendering an audio graph is done in blocks of 128 samples-frames. A block of 128 samples-frames is called a render quantum, and the render quantum size is 128.

Additional Information
Provide any additional information

Could not locate any section within the specification where the result of input less than 128 length is described.

Describe the feature
Add ability to pause/resume AudioBufferSourceNode

Is there a prototype?

Describe the feature in more detail
I think people would love to have that. Not sure why is it not possible and explicitly stated as not being provided.

This is something that most profesionnal audio software on my system can do, we should have a look to see if we can make it work with the current equations.

Describe the issue
There seems to be no consensus so far on what an audio element should do when its sound is piped into the Web Audio API. Firefox stops playing out the sound from the audio element once it is connected to the Web Audio API. Chrome on the other hand keeps on playing the sound from audio element. Until recently it was possible to manually mute the audio element in Chrome which essentially reproduces what Firefox does automatically. The problem is that both Firefox and Chrome (since a few weeks) stop the audio element internally once it is muted which means the workaround doesn't work anymore in Chrome.

Where Is It
I'm not sure if the behavior is specified already somewhere. It could also be part of the spec which defines the media element or the one for MediaStreams ...

Additional Information
I initially filed a bug for Chrome for the same issue.

Describe the Issue
Need to start a new doc for the new version

Describe the feature
Decode hexadecimal input to Float32Array capable of being copied to an existing AudioBuffer or processing with AudioWorkletProcessor. Encode Float32Array to hexadecimal.

From V1 repository

If you need help on how to use WebAudio, ask on your favorite forum or consider visiting
the WebAudio Slack Channel (register here) or
StackOverflow.

Visited web-audio.slack.com first, yet apparently an "invitation" is required

Don't have an account on this workspace yet?
Contact the workspace administrator for an invitation

If this is issue is deemed off-topic for this repository kindly post the "invitation" to the slack page so can ask the question there.

Banned from SO for 4+ more years.

Is there a prototype?
No completely. Though found this example https://babbage.cs.qc.cuny.edu/IEEE-754.old/32bit.html today.

A sampling of research efforts thus far:

• https://stackoverflow.com/questions/42699162/javascript-convert-array-of-4-bytes-into-a-float-value-from-modbustcp-read
• https://stackoverflow.com/questions/15935365/convert-float-to-bytes-in-javascript-without-float32array/16043259#16043259
• https://stackoverflow.com/questions/57803/how-to-convert-decimal-to-hexadecimal-in-javascript
• https://stackoverflow.com/questions/39275175/how-can-i-convert-javascript-float-pcm-audio-array-to-little-endian-byte-string
• https://stackoverflow.com/questions/39275175/how-can-i-convert-javascript-float-pcm-audio-array-to-little-endian-byte-string/40660469#40660469
• https://stackoverflow.com/questions/25371713/playing-raw-audio-pcm-samples-in-web-audio
• https://github.com/samirkumardas/pcm-player/blob/master/pcm-player.min.js
• https://stackoverflow.com/questions/49221045/javascript-hexadecimal-string-to-ieee-754-floating-point

Describe the feature in more detail
Provide more information how it does and how it works.

Decoding PCM from hexadecimal to Float32Array and encoding Float32Array to hexadecimal PCM (or other audio codec).

The use case is creating N discrete files, for example using a WebM container template where the values can be read from or written to the template for the purpose of "streaming" (encoding and decoding, reading, writing) those N files for "seamless" playback using AudioWorklet without using decodeAudioData(). Since Chromium is capable of playing Matroska(TM) container with PCM codec (MediaRecorder.isTypeSupported("audio/webm;codecs=pcm")) and Firefox is currently moving towards supporting WebM with FLAC encoded audio (https://bugzilla.mozilla.org/show_bug.cgi?id=1606822) this will provide a means to encode and decode Matroska and WebM files "on the fly" for various use cases.

Sample input

Header (in an XML file output by mkv2xml )

<mkv2xml>
<EBML>
  <EBMLVersion>1</EBMLVersion>
  <EBMLReadVersion>1</EBMLReadVersion>
  <EBMLMaxIDLength>4</EBMLMaxIDLength>
  <EBMLMaxSizeLength>8</EBMLMaxSizeLength>
  <DocType>matroska</DocType>
  <DocTypeVersion>4</DocTypeVersion>
  <DocTypeReadVersion>2</DocTypeReadVersion>
</EBML>
<Segment>
<Info>
  <TimecodeScale>1000000</TimecodeScale>
  <MuxingApp>Chrome</MuxingApp>
  <WritingApp>Chrome</WritingApp>
</Info>
<Tracks>
  <TrackEntry>
    <TrackNumber>1</TrackNumber>
    <TrackUID>66451940004784068</TrackUID>
    <TrackType>2</TrackType>
    <CodecID>A_PCM/FLOAT/IEEE</CodecID>
    <Audio>
      <SamplingFrequency>22050.0</SamplingFrequency>
      <Channels>1</Channels>
      <BitDepth>32</BitDepth>
    </Audio>
  </TrackEntry>
</Tracks>

Hexadecimal values within a <data> tag

      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      0000000000000000000000000000000000000000000000000000000000000000
      9201c93b4201a13b4201a13b9201c93be001703be001703b00000000000072bb
      004017bd009080bd1a018d3c7f813f3d8981443de201f13b00c0ddbce001703b
      0601833ccf81e73c9201c93b000000006a01353ce381713c1a018d3c4001a03a
      4201a13be201f13b6a01353c4201213ce201f13b19810c3c9201c93b6a01353c
      e381713cba015d3c19810c3c4201213c9201c93be201f13b9201c93be201f13b
      6a01353ce201f13b19810c3c4201a13b9201c93b9201c93be201f13b9201c93b
      4001203b9201c93b4001203b9201c93be001703b4001203b000000004001203b
      6a01353c0080c9bc00602bbd9201c93b2e21173d6041303d00805dbc004021bd
      e001703ba7a1533d0181803d7e41bf3c00401cbd00208dbd2e01973c4c41263d
      ba015d3c4201a13b6a41b53c6b61353d56412b3d9201c93b00c0e7bc0601833c
      cf81e73c6041303df801fc3c000000001a018d3c6a01353c1a018d3c0000fcbc
      00a049bd00200dbd008049bc0601833ce201f13b4001a03a19810c3c9241c93c
      ba015d3c4201213ca781d33c0000f2bb008035bc4201a13b000072bb0601833c
      19810c3c000022bbe381713c9201c93b0000cabb0000a4ba1a018d3ce4c1f13c
      0601033d9201c93b0080bfbc4201a13ce001703b9201493c4201a13b00000dbc
      0080c9bc4201213ccf81e73c00805dbccf81e73c008049bc008035bc5641ab3c
      7e41bf3c1a010d3d4001a03a008035bc4001203be4c1f13cf801fc3c4001a03a
      0000a4ba008035bc0000a1bb0000a4ba5641ab3c00408dbc0601833c00000000
      2e01973c19810c3c0000000000000000000000009381493d4201213ccf81e73c
      9201c93b008049bc75613a3d0000000038211c3d6a01353c1a018d3ca781d33c
      ba015d3c2e01973c000000002e01973c008049bc00c062bd0000f2bbbb81dd3c
      e4c1f13ce201f13bba015d3c4201a13be381713c00000000000000002e01973c
      bb81dd3c008035bc7e41bf3c9201c93b008035bce4c1f13c0000a4bae001703b
      e201f13b0040a1bc19810c3c19810c3c0000f2bb4201213c4201a13bba015d3c
      ba015d3c008035bc9201c93b4001a03a5641ab3c9201c93b0000a4ba000072bb
      0000a1bb1a018d3c6a01353c000000004201a13be001703be381713ce001703b
      004097bc1a010d3d00000000000021bc0601833c00805dbc4201a13c0000a4ba
      4201a13bba015d3c9201c93b4201a13c0080c9bcbb81dd3c1a010d3d0080b5bc
      6a41b53c00c0d3bce4c1f13c9381493d00409cbdeee1763d9201c93b0000cabb
      0b81853d0080c9bc4201213ce001703b4001203bcf81e73c004097bc0000a1bb
      e4c1f13c00805dbc4c41263de201f13b0080c9bc75613a3d00c0e7bc1a018d3c
      e201f13b0080b5bc9201493ce4c1f13c000072bc0040a1bc0601833c2e01973c
      008035bce001703bc5a1623d00e06cbd0000000038211c3dba015d3c0080b5bc
      0080abbcc5a1623d000022bb000000000080abbc19810c3ccf81e73c004097bc
      e201f13b4201a13ba781d33c0000f2bb19810c3c00c0ddbc000072bbc5a1623d
      000021bc0080b5bca781d33c4001a03a0601833c0601833c004083bc000021bc
      19810c3c1a018d3c000000004201a13c000022bb5641ab3c0000a4ba0040a1bc
      9241c93c4201213c4001a03a0000a4ba9241c93c0040a1bcbb81dd3c8981443d
      004097bda7a1533d4001a03a0080c9bc1a118d3d0080bfbc00805dbc4001a03a
      00a03fbd4c41a63d008035bd000072bbcf81e73c00401cbd0601033d002003bd
      7e41bf3c00c0d3bc2e01973c9201493c002012bdf801fc3c9201c93b4201213c
      4c41263d9201493c00000dbc000072bb56412b3d4001a03ae381713c0601033d
      f801fc3c00000dbc7e41bf3c002012bdf8017c3d4001a03a2e01973c9241c93c
      008035bd8981443d7e41bf3ce201f13b2421123d00602bbd0080abbc1011883d
      19810c3c008049bcba015d3c000022bbe201f13b0601833cf801fc3c00200dbd
      e4c1f13c0601033d004097bd0601833d56412b3d00e0a8bde201f13b2e21173d
      4221213d4201a13b00805dbc00c0d3bc29a1943d00408dbcba015d3c2e01973c
      004021bd1a118d3d4001203b00e067bd2e31973d004017bd000021bc0181803d
      00a044bd1a118d3d4001a03a00c0d8bdfd81fe3d0000fcbc1a018d3c7e41bf3c
      0080bfbcdac16c3d9201c93b0080bfbc2e01973cba015d3c5641ab3c4001203b
      00e06cbdc5a1623d9201493c4001203be001703b00408dbc7f813f3d4001a03a
      00408dbc5641ab3c19810c3c4001203b0080b5bc4201a13c56412b3d4001203b
      6a01353c4201a13c009085bdb631db3d00e06cbd1a018d3cbba15d3d00f0b2bd
      e4e1713d2421923d00a0c9bd2421123da7a1533d0060b0bde4f1f13d00b094bd
      e4e1713dba015d3c00409cbdca41e53d000072bde4c1f13c66e1b23d00a0cebd
      00602bbd4001203e00401cbd000072bb0000000000c0d3bc1a010d3d6041303d
      e001703b00200dbd00000000002003bd4201a13c0601833c5641ab3c2e21173d

that we can retrieve using DOMParser()

    fetch("data.xml")
      .then(response => response.text())
      .then(xml => {
        const res = [];
        const parser = new DOMParser();
        const doc = parser.parseFromString(xml, "application/xml");
        const mkv = doc.documentElement;
        const str = [...mkv.querySelectorAll("data")].map(({
          textContent
        }) => textContent.match(/\w+/g)).flat().join("");
        console.log(str);
      })

though am not certain about the canonical procedure to convert the above values to Float32Array(s).

Thanks to the author of https://github.com/vi/mkvparse we know that the input is playable using ffplay

$ cat mediarecorder_pcm.xml | xml2 | grep '/mkv2xml/Segment/Cluster/SimpleBlock/data=' | cut -f2-2 -d= | xxd -r -p | ffplay -f f32le -ar 22050 -ac 1 -.

Expected output is a Float32Array 1/4 the length of input hexadecimal string input that is suitable for playback using Web Audio API (i.e., within AudioWorkletProcessor).

Is there a simpler algorithm to convert the hexadecimal values than the approach at https://babbage.cs.qc.cuny.edu/IEEE-754.old/32bit.html?

Describe the Issue
The PR #95, the files were added to the master branch instead of the main branch

Expected Result
Everything should be on the main branch. We need to rename and/or move the master branch to the main branch.

we have two online daw use cases where the user needs to A) export a song as a wav, and B) freeze a (midi) track into an audio clip. currently we can reconstruct the realtime graph in an OfflineAudioContext on demand, but this doubles the memory requirements, and may even take some time with complex graphs.

feature request: would be cool to just call AudioContext.suspend() to turn the realtime context into an offline one, call startRendering() and then happily resume() back to realtime in oncomplete. i guess this would also comply with autoplaypolicy.

would inheritance chain AudioContext extends OfflineAudioContext extends BaseAudioContext make any sense?

Describe the feature
Define TextEncoderStream(), and TextDecoderStream() (and TextEncoder and TextDecoder) in AudioWorkletGlobalScope.

Is there a prototype?

TextEncoder, TextDecoder, TextEncoderStream(), and TextDecoderStream() are defined in Window and Worker scope.

Describe the feature in more detail

Consider a (potentially infinite) raw PCM stream source from fetch() to set as outputs within AudioWorletProcessor.process(). We need to convert text to a TypedArray which can be achieved with

const controller;
const readable = new ReadableStream({
  start(_) {
    return controller = _;
  }
});

readable
.pipeThrough(new TextEncoderStream())
.pipeTo(new WritableStream({
  write(value) {
    console.log(value);
  },
  close() {
    console.log('stream closed');
  }
}));
port.onMessage.addListener(async message => {
  if (message !== 'done') {
    controller.enqueue(message);
  }
  else {
    controller.close();
  }
});

at Window, however throws an error in AudioWorkletGlovbalScope

constructor(options) {
    super(options);
    this.byteSize = 512 * 344 * 60 * 50;
    this.memory = new Uint8Array(this.byteSize);
    Object.assign(this, options.processorOptions);
    console.log(this);
    this.port.onmessage = this.appendBuffers.bind(this);
  }
  async appendBuffers({ data: readable }) {
    Object.assign(this, { readable });
    const source = {
      write: (value, c) => {
        if (this.totalBytes < this.byteSize) {
          this.memory.set(value, this.readOffset);
          this.readOffset = this.readOffset + value.buffer.byteLength;
          this.totalBytes = this.readOffset;
        } else {
          console.log(
            value.buffer.byteLength,
            this.readOffset,
            this.totalBytes
          );
        }
        if (this.totalBytes >= (344 * 512) / 2 && !this.started) {
          this.started = true;
          this.port.postMessage({ started: this.started });
        }
      },
      close() {
        console.log('stopped');
      },
    };
    try {
      // throws error
      await this.readable
        .pipeThrough(new TextEncoderStream())
        .pipeTo(new WritableStream(source));
    } catch (e) {
      console.warn(e);
      console.log(this.writeOffset, this.totalBytes);
      this.endOfStream();
    }
  }

a9bc72ac-b7ac-43ba-96b0-9f8e58322a2b:37 ReferenceError: TextEncoderStream is not defined
    at AudioWorkletProcessor.appendBuffers (a9bc72ac-b7ac-43ba-96b0-9f8e58322a2b:35)

Describe the feature
It would be nice if the OfflineAudioContext could be configured to not allocate any memory for the rendered AudioBuffer in case it is not needed.

Is there a prototype?
No.

Describe the feature in more detail
Sometimes an OfflineAudioContext is only used to compute a side effect and not really for rendering a buffer.

Let's imagine you have an AudioWorkletNode which just scans the signal for its maximum value. Its usage might look like this:

const offlineAudioContext = new OfflineAudioContext({ length: 44100, sampleRate: 44100 });
const audioBufferSourceNode = new AudioBufferSourceNode(
    offlineAudioContext,
    { buffer: aBufferFromSomewhere }
);
// This could be an AudioWorkletNode which scans the signal for its maximum value. 
const maximumValueNode = new MaximumValueNode(offlineAudioContext);

audioBufferSourceNode.start(0);
audioBufferSourceNode
    .connect(maximumValueNode)
    .connect(offlineAudioContext.destination);

offlineAudioContext
    .startRendering()
    .then(() => {
        // Now that the rendering is done ask the MaximumValueNode for the maximum value.
        console.log(maximumValueNode.maximumValue);
    });

In the example above the OfflineAudioContext is only used to drive the processing of the data but the rendered buffer is not really needed. I think it would be beneficial if the buffer doesn't need to be created in any case.

Describe the issue

Am not sure if this is a specification issue or a Chromium bug (Chromium/Chrome is the only browser implementation am aware of at this time).

Initially ignored the issue that first observed when calling port.postMessage() to main thread where resume() is executed. Occasionally the processing would not commence, so refreshed the page.

When a user-defined method is created in the extended class if values used in process() were set in the method those values were not immediately available in process() function.

Upon further inquiry found that process() can be executed exactly 32 times, then appears to
not be capable of reading the value set in the method of the class.

Where Is It
https://webaudio.github.io/web-audio-api/#AudioWorkletProcessor-methods

Additional Information

  async appendBuffers({ data: { readable, processStream } }) {
    processStream = new Function(`return ${processStream}`)();
    let ready = false;
    let next = [];
    let overflow = [[], []];
    let init = false;

    globalThis.console.log({
      currentTime,
      currentFrame,
      buffers: this.buffers,
    });

    const strategy = new ByteLengthQueuingStrategy({
      highWaterMark: 32 * 1024,
    });

    const source = {
      write: async value => {
        // value (Uint8Array) length is not guaranteed to be multiple of 2 for Uint16Array
        // store remainder of value in next array
        if (value.length % 2 !== 0 && next.length === 0) {
          next.push(...value.slice(value.length - 1));
          value = value.slice(0, value.length - 1);
        } else {
          const prev = [...next.splice(0, next.length), ...value];
          while (prev.length % 2 !== 0) {
            next.push(...prev.splice(-1));
          }
          value = new Uint8Array(prev);
        }
        // we do not need length here, we process input until no more, or infinity
        let data = new Uint16Array(value.buffer);
        if (!init) {
          init = true;
          data = data.subarray(22);
          console.log('init');
        }
        let { ch0, ch1 } = processStream(data);
        // send  128 sample frames to process()
        // to reduce, not entirely avoid, glitches
        while (ch0.length && ch1.length) {
          let __ch0, __ch1;
          let _ch0 = ch0.splice(0, 128);
          let _ch1 = ch1.splice(0, 128);
          let [overflow0, overflow1] = overflow;
          if (_ch0.length < 128 || _ch1.length < 128) {
            overflow0.push(..._ch0);
            overflow1.push(..._ch1);
            break;
          }
          if (overflow0.length) {
            __ch0 = overflow0.splice(0, overflow0.length);
            __ch1 = overflow1.splice(0, overflow1.length);
            while (__ch0.length < 128 && _ch0.length) {
              let [float] = _ch0.splice(0, 1);
              __ch0.push(float);
            }
            while (__ch1.length < 128 && _ch1.length) {
              let [float] = _ch1.splice(0, 1);
              __ch1.push(float);
            }
          }
          const channel0 = new Float32Array(__ch0 || _ch0);
          const channel1 = new Float32Array(__ch1 || _ch1);
          this.buffers.set(this.i, {
            channel0,
            channel1,
          });
          // this is where the pause in execution of process() is observable
          if (this.buffers.size < 64) {
          console.log(this.buffers.size);
          }
          ++this.i;
          if (!ready) {
            ready = true;
            console.log(this.buffers.get(this.n));
            this.port.postMessage({
              start: true,
            });
          }
        }
      },
      close: _ => {
        console.log('writable close');
        // handle overflow floats < 128 length
        let [overflow0, overflow1] = overflow;
        if (overflow0.length || overflow1.length) {
          const channel0 = new Float32Array(overflow0.splice(0));
          const channel1 = new Float32Array(overflow1.splice(0));
          this.buffers.set(this.i, {
            channel0,
            channel1,
          });
          ++this.i;
        }
      },
    };

    const writable = new WritableStream(source, strategy);

    Object.assign(this, { readable, writable });

    await readable.pipeTo(writable, {
      preventCancel: true,
    });

process

process(inputs, outputs) {
    if (currentTime > 0.9 && this.buffers.size === 0) {
      console.log(
        this.i,
        this.n,
        this.buffers.size,
        this.readable,
        this.writable
      );
      this.endOfStream();
      return false;
    }
    let channel0, channel1;
    // this is where the pause/inability to read the value set is observable
    // the pause always lasts exactly at least 32 executions of process
    try {
      if (this.n < 33) {
        console.log(this.n, this.buffers.size);
      }
      // process() can be executed before appendBuffers()
      // where this.buffers is set with values
      // handle this.buffers.get(this.n) being undefined
      // for up to 32 calls to process()
      ({ channel0, channel1 } = this.buffers.get(this.n));
      // glitches can occur when sample frame size is less than 128
      if (
        (channel0 && channel0.length < 128) ||
        (channel1 && channel1.length < 128)
      ) {
        console.log(
          channel0.length,
          channel1.length,
          currentTime,
          currentFrame,
          this.buffers.size
        );
      }
    } catch (e) {
      console.error(e, this.buffers.get(this.n), this.i, this.n);
      // handle this.buffers.size === 0 while this.readable.locked (reading)
      // where entry in this.buffers (Map) deleted and this.writable (writing)
      // not having set new entry this.buffers,
      // resulting in no data to set at currentTime
      // example of JavaScript being single threaded?
      return true;
    }
    const [[outputChannel0], [outputChannel1]] = outputs;
    outputChannel0.set(channel0);
    outputChannel1.set(channel1);
    this.buffers.delete(this.n);
    ++this.n;
    return true;
  }

Logs

previewer.79829115fde743502576.js:14 0.18575963718820862 // baseLatency
2previewer.79829115fde743502576.js:14 suspended
previewer.79829115fde743502576.js:14 durationchange
previewer.79829115fde743502576.js:14 loadedmetadata
previewer.79829115fde743502576.js:14 play
previewer.79829115fde743502576.js:14 playing
audioWorklet.js:17 {currentTime: 0, currentFrame: 0, buffers: Map(0)}
audioWorklet.js:46 init
audioWorklet.js:80 1
audioWorklet.js:85 {channel0: Float32Array(128), channel1: Float32Array(128)}
audioWorklet.js:80 2
audioWorklet.js:80 3
audioWorklet.js:80 4
audioWorklet.js:80 5
audioWorklet.js:80 6
audioWorklet.js:80 7
audioWorklet.js:80 8
audioWorklet.js:80 9
audioWorklet.js:80 10
audioWorklet.js:80 11
audioWorklet.js:80 12
audioWorklet.js:80 13
audioWorklet.js:80 14
audioWorklet.js:80 15
audioWorklet.js:80 16
audioWorklet.js:80 17
audioWorklet.js:80 18
audioWorklet.js:80 19
audioWorklet.js:80 20
audioWorklet.js:80 21
audioWorklet.js:80 22
audioWorklet.js:80 23
audioWorklet.js:80 24
audioWorklet.js:80 25
audioWorklet.js:80 26
audioWorklet.js:80 27
audioWorklet.js:80 28
audioWorklet.js:80 29
audioWorklet.js:80 30
audioWorklet.js:80 31
audioWorklet.js:80 32
audioWorklet.js:80 33
audioWorklet.js:80 34
audioWorklet.js:80 35
audioWorklet.js:80 36
audioWorklet.js:80 37
audioWorklet.js:80 38
audioWorklet.js:80 39
audioWorklet.js:80 40
audioWorklet.js:80 41
audioWorklet.js:80 42
audioWorklet.js:80 43
audioWorklet.js:80 44
audioWorklet.js:80 45
audioWorklet.js:80 46
audioWorklet.js:80 47
audioWorklet.js:80 48
audioWorklet.js:80 49
audioWorklet.js:80 50
audioWorklet.js:80 51
audioWorklet.js:80 52
audioWorklet.js:80 53
audioWorklet.js:80 54
audioWorklet.js:80 55
audioWorklet.js:80 56
audioWorklet.js:80 57
audioWorklet.js:80 58
audioWorklet.js:80 59
audioWorklet.js:80 60
audioWorklet.js:80 61
audioWorklet.js:80 62
audioWorklet.js:80 63

Above, 64 (x2) Float32Arrays are created having length set to 128 to avoid glitches due to the fact that if a Float32Array is set at outputs in process() where the length is less than 128 the remainder of the TypedArray remains filled with 0s.

That means there are at least 128 Float32Arrays set at this.buffers (Map), 64 for each channel.

audioWorklet.js:133 0 "process first call" 63 // first call to process()
audioWorklet.js:149 0 63
audioWorklet.js:149 1 62
previewer.79829115fde743502576.js:14 running
audioWorklet.js:149 2 61
audioWorklet.js:149 3 60
audioWorklet.js:149 4 59
audioWorklet.js:149 5 58
audioWorklet.js:149 6 57
audioWorklet.js:149 7 56
audioWorklet.js:149 8 55
audioWorklet.js:149 9 54
audioWorklet.js:149 10 53
audioWorklet.js:149 11 52
audioWorklet.js:149 12 51
audioWorklet.js:149 13 50
audioWorklet.js:149 14 49
audioWorklet.js:149 15 48
audioWorklet.js:149 16 47
audioWorklet.js:149 17 46
audioWorklet.js:149 18 45
audioWorklet.js:149 19 44
audioWorklet.js:149 20 43
audioWorklet.js:149 21 42
audioWorklet.js:149 22 41
audioWorklet.js:149 23 40
audioWorklet.js:149 24 39
audioWorklet.js:149 25 38
audioWorklet.js:149 26 37
audioWorklet.js:149 27 36
audioWorklet.js:149 28 35
audioWorklet.js:149 29 34
audioWorklet.js:149 30 33
audioWorklet.js:149 31 32
audioWorklet.js:149 32 31

However, after 32 calls this.buffers.size is 0

audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 63 63
process @ audioWorklet.js:170
audioWorklet.js:80 1
audioWorklet.js:80 2
audioWorklet.js:80 3
audioWorklet.js:80 4
audioWorklet.js:80 5
audioWorklet.js:80 6
audioWorklet.js:80 7
audioWorklet.js:80 8
audioWorklet.js:80 9
audioWorklet.js:80 10
audioWorklet.js:80 11
audioWorklet.js:80 12
audioWorklet.js:80 13
audioWorklet.js:80 14
audioWorklet.js:80 15
audioWorklet.js:80 16
audioWorklet.js:80 17
audioWorklet.js:80 18
audioWorklet.js:80 19
audioWorklet.js:80 20
audioWorklet.js:80 21
audioWorklet.js:80 22
audioWorklet.js:80 23
audioWorklet.js:80 24
audioWorklet.js:80 25
audioWorklet.js:80 26
audioWorklet.js:80 27
audioWorklet.js:80 28
audioWorklet.js:80 29
audioWorklet.js:80 30
audioWorklet.js:80 31
audioWorklet.js:80 32
audioWorklet.js:80 33
audioWorklet.js:80 34
audioWorklet.js:80 35
audioWorklet.js:80 36
audioWorklet.js:80 37
audioWorklet.js:80 38
audioWorklet.js:80 39
audioWorklet.js:80 40
audioWorklet.js:80 41
audioWorklet.js:80 42
audioWorklet.js:80 43
audioWorklet.js:80 44
audioWorklet.js:80 45
audioWorklet.js:80 46
audioWorklet.js:80 47
audioWorklet.js:80 48
audioWorklet.js:80 49
audioWorklet.js:80 50
audioWorklet.js:80 51
audioWorklet.js:80 52
audioWorklet.js:80 53
audioWorklet.js:80 54
audioWorklet.js:80 55
audioWorklet.js:80 56
audioWorklet.js:80 57
audioWorklet.js:80 58
audioWorklet.js:80 59
audioWorklet.js:80 60
audioWorklet.js:80 61
audioWorklet.js:80 62
audioWorklet.js:80 63

Another example of logs

running // AudioContext state
audioWorklet.js:133 0 "process first call" 31 // first call to process()
audioWorklet.js:149 0 31
audioWorklet.js:149 1 30
audioWorklet.js:149 2 29
audioWorklet.js:149 3 28
audioWorklet.js:149 4 27
audioWorklet.js:149 5 26
audioWorklet.js:149 6 25
audioWorklet.js:149 7 24
audioWorklet.js:149 8 23
audioWorklet.js:149 9 22
audioWorklet.js:149 10 21
audioWorklet.js:149 11 20
audioWorklet.js:149 12 19
audioWorklet.js:149 13 18
audioWorklet.js:149 14 17
audioWorklet.js:149 15 16
audioWorklet.js:149 16 15
audioWorklet.js:149 17 14
audioWorklet.js:149 18 13
audioWorklet.js:149 19 12
audioWorklet.js:149 20 11
audioWorklet.js:149 21 10
audioWorklet.js:149 22 9
audioWorklet.js:149 23 8
audioWorklet.js:149 24 7
audioWorklet.js:149 25 6
audioWorklet.js:149 26 5
audioWorklet.js:149 27 4
audioWorklet.js:149 28 3
audioWorklet.js:149 29 2
audioWorklet.js:149 30 1
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:149 31 0
audioWorklet.js:170 TypeError: Cannot destructure property 'channel0' of 'this.buffers.get(...)' as it is undefined.
    at AudioWorkletProcessor.process (audioWorklet.js:155) 0 31 31
process @ audioWorklet.js:170
audioWorklet.js:80 1
audioWorklet.js:80 2
audioWorklet.js:80 3
audioWorklet.js:80 4
audioWorklet.js:80 5
audioWorklet.js:80 6
audioWorklet.js:80 7
audioWorklet.js:80 8
audioWorklet.js:80 9
audioWorklet.js:80 10
audioWorklet.js:80 11
audioWorklet.js:80 12
audioWorklet.js:80 13
audioWorklet.js:80 14
audioWorklet.js:80 15
audioWorklet.js:80 16
audioWorklet.js:80 17
audioWorklet.js:80 18
audioWorklet.js:80 19
audioWorklet.js:80 20
audioWorklet.js:80 21
audioWorklet.js:80 22
audioWorklet.js:80 23
audioWorklet.js:80 24
audioWorklet.js:80 25
audioWorklet.js:80 26
audioWorklet.js:80 27
audioWorklet.js:80 28
audioWorklet.js:80 29
audioWorklet.js:80 30
audioWorklet.js:80 31
audioWorklet.js:80 32
audioWorklet.js:80 33
audioWorklet.js:80 34
audioWorklet.js:80 35
audioWorklet.js:80 36
audioWorklet.js:80 37
audioWorklet.js:80 38
audioWorklet.js:80 39
audioWorklet.js:80 40
audioWorklet.js:80 41
audioWorklet.js:80 42
audioWorklet.js:80 43
audioWorklet.js:80 44
audioWorklet.js:80 45
audioWorklet.js:80 46
audioWorklet.js:80 47
audioWorklet.js:80 48
audioWorklet.js:80 49
audioWorklet.js:80 50
audioWorklet.js:80 51
audioWorklet.js:80 52
audioWorklet.js:80 53
audioWorklet.js:80 54
audioWorklet.js:80 55
audioWorklet.js:80 56
audioWorklet.js:80 57
audioWorklet.js:80 58
audioWorklet.js:80 59
audioWorklet.js:80 60
audioWorklet.js:80 61
audioWorklet.js:80 62
audioWorklet.js:80 63
audioWorklet.js:149 31 160
audioWorklet.js:149 32 159

This effectively means that when process() is being executed no other methods defined in extended AudioWorkletProcess are being executed.

Is that by design?

Or, the result of JavaScript being "single threaded"?

The WG will flesh out some plans for V2 document migration. There are a couple of issues, but the general consensus is to separate out the V2 spec.

Proposal

• When adding a complete new feature, it can go directly to V2.

• When adding improvements on an existing feature, we can copy V1 text over to V2 and make some modifications.

• Good-first-issues:

  • NoiseGenerator (a brand new spec): issue
  • AudioParam power law (improvements): issue

Describe the feature
Take two oscillators. Say I have one oscillatorA that is ramping linearly, I'd like there to be a way such that you can make the other oscillatorB scale as a function of the initial one.

For example:
oscNodeA.frequency.setValueAtTime(1, context.currentTime); oscNodeA.frequency.linearRampToValueAtTime( 3000, context.currentTime + 10 ); oscNodeB.frequency.value = oscNodeA.frequency.value * 2;

Which should operate like if I pass oscNodeA.frequency.value in an object, it will constantly update the frequency as oscNodeA changes frequency. It's useful for shifting frequencies of whole chords made by Oscillators

Probably want to have issue templates for bugs and feature requests like we have for https://github.com/WebAudio/web-audio-api/issues/new/choose

Could just steal the templates from there; don't think we need to make any changes for v2 for that.

Describe the issue
A clear and concise description of what the problem is in the spec.

For some reason an instance of AudioWorklet played back twice, after disconnect() was called on AudioWorklet and close() called on AudioContext.

After setting own requirement to

1. fetch() a audio file
2. Use Response.body to create a ReadableStream
3. Read audio file bytes (parse WAV file) by converting values to Float32Array and begin playback of audio file as soon as possible, without awaiting completion of reading the entire file

achieved that requirement approximately 2 days ago using AudioWorklet at Chromium 81 at localhost.

Today, re-ran the same code (with Web Audio tab at DevTools open) and observed 3 issues

1. The first 30 seconds of playback is "choppy", with a decreased playback rate
2. Web Audio DevTools displays Render Capacity toggling between ~1% and ~98% - tentatively meaning the two Worklet instances which are apparently being used by AudioWorklet are not balanaced in the processing of the audio;
   3. After executing AudioWorklet.disconnect() and AudioContext.close() and gc() (exposed via Chromium flag --js-flags=--expose-gc) the audio played back a second time - meaning the duration of the WAV file is 6:42 yet playback completed at 13:30

If you need help on how to use WebAudio, ask on your favorite forum or consider visiting
the WebAudio Slack Channel (register here) or
StackOverflow.

Where Is It
Provide a link to where the problem is in the spec

The relevant sections of the specification which contain language that could possibly explain the implementation behaviour that have been able to locate

https://webaudio.github.io/web-audio-api/#active-source

Every AudioWorkletProcessor has an associated active source flag, initially true. This flag causes the node to be retained in memory and perform audio processing in the absence of any connected inputs.

https://webaudio.github.io/web-audio-api/#AudioWorkletProcessor-attributes

Note: Authors that register a event listener on the "message" event of this port should call close on either end of the MessageChannel (either in the AudioWorkletProcessor or the AudioWorkletNode side) to allow for resources to be collected.

Additional Information
Provide any additional information

Was trying to determine the reason for the same code playing back "choppy" and with decreased playback rate when 2 days earlier the playback was not "choppy" for the first ~30 seconds.

Opened Web Audio tab at DevTools in an attempt to debug the issue. Noticed the imbalanced printing of Render Capacity toggling between ~1% and ~98% - apparently toggling usage of the Worklets created by AudioWorklet constructor.

Was not expecting the audio to be played back a second time after AudioContext.close() being executed.

Describe the Issue
Need to have the PR review bot and Travis CI set up when we start editing the spec.

Describe the issue
Is defining fetch() in AudioWorkletGlobalScope an implementer choice?

Where Is It
AFAICT not addressed.

Additional Information
fetch is not defined in AudioWorkletGlobalScope at Chromium. Why?

Is defining fetch within AudioWorkletGlobalScope at Firefox, Nightly within the parameters of the specification, or is fetch intentionally omitted from being defined in AudioWorkletGlobalScope?

In https://webaudio.github.io/web-audio-api/#constructors-4, there is no description for the options parameter for the AudioBuffer constructor. Maybe we want to add a short sentence for completeness.

Describe the feature
The comparision between using transferring a single stream (ReadableStream; WritableStream) and transferring hundreds of thousands of TypedArrays using MessagePort.postMessage() is not even close.

AudioWorklet invariably outputs multiple gaps of silence at both Chromium and Firefox when thousands of messages are posted/TypedArrays transferred.

Is there a prototype?

Yes.

• https://docs.google.com/document/d/1_KuZzg5c3pncLJPFa8SuVm23AP4tft6mzPCL5at3I9M/edit
• https://bugs.chromium.org/p/chromium/issues/detail?id=894838

Describe the feature in more detail

See https://bugs.chromium.org/p/chromium/issues/detail?id=910471#c11.

Posting a single ReadableStream, e.g., from a TransformStream substantially reduces observable gaps in playback to 1 (that bug is currently isolated to being observable at the end of reading/writing the stream, not the underlying network request in the code used to get the stream) from multiple gaps randomly occurring during playback https://bugzilla.mozilla.org/show_bug.cgi?id=1629384#c24.

Compare for yourself https://plnkr.co/edit/nECtUZ (transferable streams implemented at Chromium behind --enable-experimental-web-platform-features); https://plnkr.co/edit/yh5A66UhMnlpq0JF (thousands of postMessage(<TypedArray>, [<TypedArray.buffer>]) calls).

Please forgive my ignorance here. I'm new to audio synthesis...

WebAudio supports using custom waveforms with oscillator nodes, but the API (createPeriodicWave) only accept arguments defined in the frequency domain (real and imag).

I have periodic waves in the time domain (defined as functions), but without an FFT, there seems to be no way to pass them into an oscillator node. I understand there are a couple of ways to feed arbitrary signals into the routing graph, but this request is about the signature of createPeriodicWave.

There are a few JavaScript libraries online with FFT implementations, but none have been updated for years. I could probably figure it out for myself, or compile some C to WebAssembly or something, but it would obviously be much nicer if createPeriodicWave also accepted time-domain arguments.

More generally, the API could provide FFT and IFFT helpers (especially if they are already implicitly required by the implementation), but even then, the createPeriodicWave method should still be able to understand time-domain arguments.

Again, sorry if I've misunderstood something.

I am developing a 3D environment that connects two microphones with a specific geometry in my scene , runs locally for now.
I want to be able to switch between which microphone is being used ( when I press key A use mic1 and when I press key B use mic 2 ).
The error I get is :
Firefox ----> Error:NotReadableError: Concurrent mic process limit.
Chrome -> no Error, it just doesn't switch devices

How can I fix that ? I tried to stop the stream but it doesn't work , any suggestions ?

Describe the feature
encodeAudioData as a complement to the existing decodeAudioData method on AudioContext. This is useful if you want to do things like record audio into the browser from the microphone, apply effects using the Web Audio API and then create a file that people can take away for use in audio editing/playback software outside the browser.

Is there a prototype?
No, but there are a number of libraries which implement similar behavior using either WASM or DataViews in JavaScript.

For example: https://github.com/Jam3/audiobuffer-to-wav

Describe the feature in more detail
The function would take a buffer instance, and return an ArrayBuffer representing the contents of the encoded file: encodeAudioData(buffer: AudioBuffer, options: EncodeAudioDataOptions) => ArrayBuffer

The options could allow developers to specify the details of various encoding targets. To begin with, having simple encoding to .wav would be immensely helpful. It would be great if other compressed encodings were also available. For example, mp3 might be a good option now that the patent has run out.

Why have it in the spec?
This could be handled by libraries, but having this built into the spec would avoid having to download large libraries/WASM blobs, and would open the door to more performant implementations.

Describe the feature

WebCodecs proposal is mentioned at V1 in several issues

• WebAudio/web-audio-api#335 (comment)

  This will now be handled by https://discourse.wicg.io/t/webcodecs-proposal/3662

• WebAudio/web-audio-api#337 (comment)

  This will now be handled by https://discourse.wicg.io/t/webcodecs-proposal/3662

• WebAudio/web-audio-api#371 (comment)

  This will now be handled by https://discourse.wicg.io/t/webcodecs-proposal/3662

• WebAudio/web-audio-api#1468 (comment)

  Likely to be solved by webcodecs. No response since Oct 2018

• WebAudio/web-audio-api#1850 (comment)

  Closing this, it is not an area of the api that we will be working on in V2. However we hope that this will be covered by https://github.com/WICG/web-codecs
  https://github.com/WICG/web-codecs/blob/master/explainer.md
  https://discourse.wicg.io/t/webcodecs-proposal/3662

• WebAudio/web-audio-api#1850 (comment)

  Don't know how it will work, but having two apis that overlap a little is not a great pattern. It seems more beneficial for everyone if WebCodecs can solve this, rather than grafting some kind of partial solution to decodeAudioData but not be able to do other things that WebCodecs can do. It's up to you and us to do our best to make it happen.

• WebAudio/web-audio-api#1872 (comment)

  This will not be considered for V2, as it is better placed in webcodecs.
  https://github.com/WICG/web-codecs

• WebAudio/web-audio-api#2135 (comment)

  This is what we're doing in Web Codecs.

• WebAudio/web-audio-api#2135 (comment)

  • implement what is needed here in js/wasm has been done numerous times before, and works well, allowing authors that need to solve this use case to wait until WebCodec is ready.

• WebAudio/web-audio-api#2135 (comment)

  Let me also add that we will work closely with the WebCodecs group to make sure that it can provide solutions to the problems people have had with decodeAudioData. Ideally, WebCodecs should be able to do everything that decodeAudioData does and more.

• WebAudio/web-audio-api#337 (comment)

  New feature requests go in the v2 repo. New feature requests about audio decoding and encoding go in the Web Codecs repo, but this is use-case has been handled from day one there.

Is there a prototype?

No.

It is not at all clear how WebCodecs will solve any of the issues where Web Audio API specification authors have suggested the same might eventually be possible.

So far there are only unsubstantiated claims without any hint of an algorithm or proof-of-concept in code where using WebCodecs would produce any result different than is currently possible using Web Audio API V1.

Since Web Audio API only processes PCM

1.4. The AudioBuffer Interface
This interface represents a memory-resident audio asset. Its format is non-interleaved 32-bit floating-point linear PCM values with a normal range of [−1,1]
, but values are not limited to this range. It can contain one or more channels. Typically, it would be expected that the length of the PCM data would be fairly short (usually somewhat less than a minute). For longer sounds, such as music soundtracks, streaming should be used with the audio element and MediaElementAudioSourceNode.

5.1. Audio sample format
Linear pulse code modulation (linear PCM) describes a format where the audio values are sampled at a regular interval, and where the quantization levels between two successive values are linearly uniform.

Whenever signal values are exposed to script in this specification, they are in linear 32-bit floating point pulse code modulation format (linear 32-bit float PCM), often in the form of Float32Array objects.

5.2. Rendering
The range of all audio signals at a destination node of any audio graph is nominally [-1, 1]. The audio rendition of signal values outside this range, or of the values NaN, positive infinity or negative infinity, is undefined by this specification.

without any documentation, example code, or even a concept to prove otherwise the gist of WebCodecs relevant to Web Audio API V1 or V2 would essentially be an arbitrary audio codec input to PCM output. Nothing more can be gleaned from the comments thus far, besides "hope" for some furture capability proferred by WebCodecs that will somehow be connectable to Web Audio API. For individuals that do not entertain hope, rather base conclusions on facts, there are currently no facts available which support the claims made thus far that WebCodecs will provide some means to solve the issues closed that point to WebCodecs proposal as being capable of remedying the problems described in the issues.

It is not clear how WebCodecs will change or modify at all Web Audio API processing model re "5.1. Audio sample format" or "5.2. Rendering", presently making any reference to "This will now be handled by https://discourse.wicg.io/t/webcodecs-proposal/3662" moot, null and void, particularly without any example or basic flow-chart detailing how precisely Web Audio API will connect to WebCodecs.

Describe the feature in more detail

This issue is intended to be a meta thread for WebCodecs prospective connection with the as-yet un-sepecified and un-deployed WebCodecs.

As a concrete example of the problem with simply suggesting WebCodecs will solve a current problem with the Web Audio API in the future, without evidence to support such a claim, following up on WebAudio/web-audio-api#2135 where decodeAudioData() does not support decoding partial content, leading to memory usage that eventually crashes the tab, was able to cobble together a working example of streaming Opus audio attempting to solve this MediaSource issue w3c/media-source#245 using Web Audio API by first splitting the file into N 15 second Opus files with ffmpeg or mkvmerge then sorting and merging the files again into a single file using Files

      const sorted = [...e.target.files].sort((a, b) => 
                               +b.name.replace(/\D/g, "") > a.name.replace(/\D/g, "") ? -1 : 0);
      const offsets = sorted.map(({size}) => size);
      console.log(offsets);
      const blob = new Blob(sorted);
      const a = document.createElement("a");
      a.href = URL.createObjectURL(blob);
      a.download = "house_opus_merged";
      a.click();

then serving the map of sorted offsets with the merged file

<?php 
  if ($_GET["opus"]) {
    $offsets = "[135567,127418,120207,124575,129572,120537,120918,133856,137974,135627,138015,128146,128113,127626,129090,124280,127461,133903,133995,137177,137409,148202,147645,143197,126504,127171,94884]";
    header("Content-Type: application/octet-stream");
    header("Content-Description: " . $offsets);
    $file = file_get_contents("./house_opus_merged", FILE_USE_INCLUDE_PATH);
    echo $file;
  }
?>

which allows executing decodeAudioData() on one of the included N merged parts of the single served file then commencing playback of that audio segment without decoding all of the N parts of the file first - and without discernable gaps in playback (not achieved with first attempt at splitting the file). Included is the first execution of the spittling of the file and original file in attached zip archive.

(async() => {
  try {
  const request = await fetch("http://localhost:8000?opus=true");
  const response = await request;
  const offsets = JSON.parse(response.headers.get("Content-Description"));
  const inputStream = response.body.getReader();

  const audioData = [];
  let n = 0;
  let offset;
  let controller; 
  const ac = new AudioContext({
    numberOfChannels: 2,
    sampleRate: 48000
  });

  const msd = new MediaStreamAudioDestinationNode(ac);
  const {
    stream
  } = msd;
  const gainNode = new GainNode(ac, {
    gain: 1
  });

  gainNode.connect(msd);

  const [track] = stream.getAudioTracks();
  track.enabled = false;

  const audio = new Audio();

  audio.autoplay = true;
  audio.controls = true;

  document.body.appendChild(audio);

  const outputStream = new ReadableStream({
    async start(c) {
      return controller = c;
    }
  });
  const audioOutputReader = outputStream.getReader();
  async function processAudioOutputStream({
    value: sourceBufferSegmentEnded,
    done
  }) {
    if (done) {
      await outputStream.closed;
      return "processAudioOutputStream done:" + done;
    }
    const {
      segment, trackOffset
    } = await sourceBufferSegmentEnded();
    if (segment === offsets.length - 1) {
      track.stop();
      track.disabled = true;
      audioOutputReader.cancel();
    }
    return processAudioOutputStream(await audioOutputReader.read());
  }

  async function processAudioInputStream({
    value, done
  }) {
    if (done) {
      await inputStream.closed;
      return "processAudioInputStream done:" + done;
    }
    offset = 0;
    do {
      audioData.push(value[offset]);
      offset++;
      if (audioData.length === offsets[n]) {
        const uint8array = new Uint8Array(audioData.splice(0, offsets[n]));

        controller.enqueue((props => () => {
          return new Promise(async resolve => {
            const ab = await ac.decodeAudioData(uint8array.buffer);
            const source = ac.createBufferSource();
            source.connect(gainNode);
            source.buffer = ab;

            source.start();
           
            source.onended = _ => {              
              source.disconnect(gainNode);
              resolve(props);
            };
            if (audio.srcObject !== stream) {
              track.enabled = true;
              audio.srcObject = stream;
            };

          })
        })({
          segment: n,
          trackOffset: offsets[n]
        }));
        n++;
      }
    } while (offset < value.length);
    return processAudioInputStream(await inputStream.read());
  }
  const input = await processAudioInputStream(await inputStream.read());
  console.log({input})
  const output = await processAudioOutputStream(await audioOutputReader.read());
  console.log({output});
  } catch (e) {
    console.trace();
  }
})();

The problem is twofold

• obvious gaps in playback (probably due to waiting on ended event of source buffer)
• posting the data to AudioWorklet to avoid gaps requires substantial attention to the 128 byte per process() execution, where when the test is not correctly configured can crash the browser tab and the underlying OS.

The suggested solution for this case as evidenced by the closure of similar use cases is now to use WebCodecs. However, it is not immediately clear at all how WebCodecs will overcome Web Audio API "Audio sampling format" and "Rendering" sections. This corresponding subject-matter WebCodecs issue w3c/webcodecs#28 is not resolved.

Given the code example above and cross-referencing to WebCodecs examples https://github.com/WICG/web-codecs/blob/master/explainer.md#examples the only code where Web Audio API would potentially connect to output of WebCodecs is by way of a MediaStreamTrack, not PCM

// Finally the decoders are created and the encoded media is piped through the decoder
// and into the TrackerWriters which converts them into MediaStreamTracks.
const audioDecoder = new AudioDecoder({codec: 'opus'});

The question then becomes would the output MediaStreamTrack be connect()ed to a an audio node? If no post-processing or output modification is required why would Web Audio API be used at all in that case?

Given the above example code that combines N files into a single file to preserve metadata for each file for decodeAudioData() what pattern would fulfill the suggestion that WebCodecs would solve the problems described in the linked closed issues that refer to WebCodecs?

Ideally we should be able to parse any input file having any audio codec to AudioWorklet as Float32Array without having to also use MediaStreamTrack, or again, why use Web Audio API at all in that case where we can simply audio.srcObject = new MediaStream([await audioDecorder])?

Is

audioTrackNode = new MediaStreamTrackAudioSourceNode(context, {mediaStreamTrack:await audioDecoder});

expected to be used for connect to WebCodecs, where we could stream the Opus file without splitting the file into N parts - and playback will begin without parsing the entire input, especially where the input might not have a definitive end - even though that method is only available at Firefox?

The answer to, but not limited to the above questions, relevant to the reference to WebCodecs dependence, reliance or referral should be clear.

Re

This is what we're doing in Web Codecs.

Can the specification authors who have made the above claims kindly clearly write out precisely what you are "doing" re WebCodecs being connectable in any meaningful way to Web Audio API (V1 and V2)?

A nice feature of this API is the ability to write custom nodes for modulation - things like parameterized envelopes or step sequencers w/ parameterized slew. This can be achieved now with a-rate outputs, but often k-rate would work just as well and it would be significantly more efficient.

This could be accomplished by adding outputRate alongside outputChannelCount to AudioWorkletNode options:

new AudioWorkletNode(context, 'ModulationProcessor', {
  numberOfInputs: 0,
  numberOfOutputs: 1,
  outputChannelCount: [1],
  outputRate: ['k-rate'],
  channelCount: 1,
  channelCountMode: 'explicit',
  channelInterpretation: 'discrete',
});

Connecting a k-rate output to an a-rate param or to the input of another AudioNode could cause the value to be upscaled by filling a 128 length buffer.

First, https://webaudio.github.io/web-audio-api/#channel-limitations describes some limitations. But this doesn't really add anything that can't be determined from the (fixed!) channelCountMode and channelCount values specified for the node. With these values, all inputs are mixed to stereo on input so the output is always stereo.

Second, https://webaudio.github.io/web-audio-api/#panner-channel-limitations references https://webaudio.github.io/web-audio-api/#channel-limitations, so we don't need this section either (for the same reasons).

AnalyserNode already performs a windowed FFT. However, it provides only access to a smoothed power spectrum (log-magnitudes of the complex FFT result vector) through getFloatFrequencyData() resp. getByteFrequencyData(). There is no access to raw magnitude nor any phase information, even though the FFT must compute it. This needlessly limits the applicability of AnalyserNode.

Please add an accessor getComplexFrequencyData() that returns the unsmoothed complex-valued FFT result vector (called X[k] here directly. Whatever format is used internally is fine - it could be a float[2][fftsize], float[fftsize][2], or float[2*fftsize] with real and imaginary components interleaved or not. The idea here is to provide direct access to the FFT result without incurring any performance penalties.

A getPolarFrequencyData() that converts the FFT result into polar (magnitude & phase) coordinates would also be useful, as this is a frequent use case. Alternatively, since the existing getFloatFequencyData() infrastructure (with smoothing, log, and scaling) already provides the magnitudes, a getPhaseData() that computes just the phase vector would suffice. Here is a fast atan2() approximation that would provide added value. Cheers!

Describe the issue

Does AudioContext.latencyHint affect AudioWorkletProcessor::process Callback Interval?

If so that language and effect is not clearly specified.

Where Is It

• https://webaudio.github.io/web-audio-api/#dictionary-audiocontextoptions-members
• https://webaudio.github.io/web-audio-api/#processing-model-background
• https://webaudio.github.io/web-audio-api/#latency
• https://webaudio.github.io/web-audio-api/#AudioWorkletProcessor-instantiation
• https://heycam.github.io/webidl/#construct-a-callback-function

Additional Information

At DevTools open WebAudio tab. Observe the values of Callback Interval corresponding to different AudioContext.latencyHint settings

"playback" => Callback Interval 23.217
"interactive" => Callback Interval 11.626
"balanced" => Callback Interval 9.9
0.5 => Callback Interval 185.729

Describe the Issue
Need to setup github/travis to display the spec. Currently there's nothing.

Expected Result
https://webaudio.github.io/web-audio-api-v2 should display the v2 version of the spec (that currently has no new content).

Actual Result
404

I was recently thinking about some ideas with web audio and I came to my head that,
could it be possible to add a oversample and bufferSize property to AudioWorklet?
This is could be a very useful feature for when making custom nodes.
While the oversample property does make the bufferSize property seem useless since oversampling is basically (bufferSize * n or basically 128 * n), I'm suggesting that:

• the oversample property:

  • is used for oversampling the selected bufferSize that is used for processing to the best it can achieve.
  • can be used to achieve high quality
  • can be set with the same values as WaveShaperNode.prorotype.oversample.

• the bufferSize property:

  • can be set to a preferred buffer size. This could be good for performance, even though it is on worker thread.
  • can only be set on the AudioWorkletOptions and the can be only a read only property after initialization.
  • value is applied to all inputs and outputs of the processing block.
  • either can accept values limited to what the ScriptProcessorNode uses (a.k.a 128 - 16384). Or to a wider range base on performance and support of the worker thread and or hardware (a.k.a 128 - ?).
  • default value should be 128.
  • could be achieved with ringbuffer since the audioWorklet process only 128 frames per block. Take a look at this code of a ring buffer used in p5.js-sound library. I have tested it and works well and very handy. I definitely suggest to take a look at the way it's been used.

I also realized that this could also affect the custom parameters especially with automation rates. My suggestion to work around this is that for the parameters with a k-rate that if the oversample is not set to 'none' then the value is multiplied and for when the buffer size is greater than 128 frames the parameter produces values similar to a a-rate parameter but has 1 value for each block (a.k.a per 128 frames) from the total buffer size.

For example: say the buffer size is 512. 512 divided by 128 equal 4 blocks per 128 frames. Since there a 4 blocks in 512 the k-rate parameter should result in 4 different (or the same depending on the value prop in the main thread), values for each 128 frames per block out of 512 frames.

And for the a-rate parameters, the value will result the same method for the oversample property. In other words the value is multipled if the oversample property is not set to 'none'. Besides that buffer size shouldn't matter since an a-rate parameter does not produce 1 value per 128 frames If you know what I mean.

Also (I'm not sure about this part since it's on a worker thread but), since oversampling and different buffer sizes can cause a little bit of latency (I think), Could it be possible to report the latency in ms as a property in both the main and worker thread?

Also this should also work with respect of the AudioContext sampleRate as well.
Similar to the step process that WaveShaperNode does that must follow when using the oversample property according to the spec.

here's an example of what it could look like:

//main thread:
let worklet = new AudioWorkletNode(context, name, {
  bufferSize: 256,
  oversample: "none",
  //etc...
});

worklet.oversample;  //setter and getter. returns "none"
worklet.bufferSize; //read only! returns 256
worklet.latency; //read only!  returns a number in ms the latency of the processor from input to output


//worker thread:
class MyProcessor extends AudioWorkletProcessor {
   
   process(inputs, outputs, params) {
      this.latency; //read only!  returns a number in ms the latency of the processor from input to output
      this.oversampleType; // read only! number - returns 0 for 'none' and a number for the oversample factor. i.e returns 2 if oversample = "2x"
      this.bufferSize; //read only! number - returns the original selected buffer size from the AudioWorkletNodeOptions.
     
     //this code here is used to get the result of the buffer size for when it's oversampled:
     let oversampledSize = this.bufferSize * this.oversampleType;

     //params:

     //for k-rate parameters: 
     //since the buffer size is 256. 256 / 128 is 2 blocks. So there will be 2 values for each 128 frames for k-rate parameters. 
     //params["myParam"].length === 2
     let a = params["myParam"][0]; //returns the value for the first 128 frames
     let b = params["myParam"][1]; //returns the value for the next 128 frames

     //for a-rate parameters:
     //the buffer size doesn't matter. It's basically 'params["myParam"].length === bufferSize (256 in this example)
     let c = params["myParam"][0]; //returns the first value out of 256 values
     

   //for when either a k-rate or an a-rate parameter with oversample property is not set to "none",
   //the value is multiplied otherwise it's the original value
   //To achieve the original value, just divide it by the oversample amount:
    let originalValue = params["myParam"][0] / this.oversampleType;
   
   }
}

The goal of this feature is to achieve high quality processing and better performance in the worker thread with the audio worklet.
I hope i made sense and it's not too complicated. Also please correct if I'm wrong as well.
Do you think this is cool idea?
And could this be possible or not?
Thanks :)

Describe the feature

An alternative to module loading of AudioWorklet.

Is there a prototype?

The closest that I am aware of is Anonymous inline modules

Describe the feature in more detail

When I run this code https://github.com/guest271314/webtransport/blob/main/webTransportAudioWorkletWebAssemblyMemoryGrow.js at console on GitHub I get this error

Refused to load the script 'blob:https://github.com/66922005-0952-462d-abbe-a0983c78079f' because it violates the following Content Security Policy directive: "script-src github.githubassets.com". Note that 'script-src-elem' was not explicitly set, so 'script-src' is used as a fallback.

VM187:238 DOMException: The user aborted a request.
webTransportAudioWorkletMemoryGrow @ VM187:238
async function (async)
webTransportAudioWorkletMemoryGrow @ VM187:113
(anonymous) @ VM194:1

which is the Blob URL at

    const worklet = URL.createObjectURL(
      new Blob(
        [
          registerProcessor(
            'audio-worklet-quic-transport-stream',
            AudioWorkletQuicTransportStream
          ),
        ],
        {
          type: 'text/javascript',
        }
      )
    );

We should be able to run AudioWorklet using inline code

    class AudioWorkletProcessor {}
    class AudioWorkletQuicTransportStream extends AudioWorkletProcessor {
    // ...
    }

to avoid making a fetch request.

i'm trying to figure out how to have a lower latency when playing wav samples.

i'm digging into audio worklet to find some informations but no solutions (if it exist)

i was thinking about loading my wav samples in an internal buffer when loading page
and playing it back with a lower latency but is it even possible for now ?

Let's say you have a very simple AudioWorklet having a single input and a single output with no AudioParams. Let's say you made a mistake and have

process(inputs, outputs, params) {
  let input = inputs[0];
  let output = outputs[0];
  output.fill(input[0][0]);
  return true;
}

This accidentally modifies outputs[0] to be an array of floats. But it's supposed to be a sequence<Float32Array>.

How is the worklet supposed to handle this? I'm kind of thinking this should cause an error to be thrown. I guess that also implies that if output isn't exactly sequence<sequence<Float32Array>> an error is thrown. Similarly, if the length of the Float32Array isn't 128, it's an error.

(In case it matters, I actually did this by accident.)

Describe the feature
WebCodecs defines AudioData. In the WebCodecs specification this note appears:

NOTE: The Web Audio API currently uses f32-planar exclusively.

However, the format of AudioData from AudioDecoder is 'f32' not 'f32-planar'.

Even though sampleRate set at AudioDecoder configuration sampleRate is other than 48000 (and opusenc supports --raw-rate option to specifically set sample rate for Opus encoded audio) the resulting WebCodecs AudioData instance always has sampleRate set to 48000.

The effective result is that there is no way that I am aware of to convert the data from AudioData.copyTo(ArrayBuffer, {planeIndex: 0}) to an AudioBuffer instance that can be played with AudioBufferSourceNode or resampled to a different sampleRate, for example, 22050.

Since MediaStreamTrackGenerator suffers from "overflow" and no algorithm exists in the WebCodecs specification to handle the overflow outside of one defined by the user it is necessary for the user to write the algorithm. After testing a user might find a magic number to delay the next call to MediaStreamTrackGenerator.writable.WritableStreamDefaultWriter.write() https://plnkr.co/edit/clbdVbhaRhCKWmPS that approach does not achieve the same result when attempting to use a Web Audio API AudioBuffer and AudioSourceNode

async function main() {
  const oac = new AudioContext({
    sampleRate: 48000,
  });
  let channelData = [];
  const decoder = new AudioDecoder({
    error(e) {
      console.error(e);
    },
    async output(frame) {
      const { duration: d } = frame;
      const size = frame.allocationSize({ planeIndex: 0 });
      const data = new ArrayBuffer(size);
      frame.copyTo(data, { planeIndex: 0 });
      const view = new Float32Array(data);
      let i = 0;
      for (let i = 0; i < view.length; i++) {
        if (channelData.length === 220) {
          const floats = new Float32Array(220);
          floats.set(channelData.splice(0, 220));
          const ab = new AudioBuffer({
            sampleRate: 48000,
            length: floats.length,
            numberOfChannels: 1,
          });
          ab.getChannelData(0).set(floats);
          const source = new AudioBufferSourceNode(oac, { buffer: ab });
          source.connect(oac.destination);
          console.log(ab.duration, ab.sampleRate);
          source.start();
          await new Promise((r) => {
            console.log(ab);
            source.onended = r;
          });
        }
        channelData.push(view[i]);
      }
      if (decoder.decodeQueueSize === 0) {
        if (channelData.length) {
          const floats = new Float32Array(220);
          floats.set(channelData.splice(0, 220));
          const ab = new AudioBuffer({
            sampleRate: 48000,
            length: floats.length,
            numberOfChannels: 1,
          });
          ab.getChannelData(0).set(floats);
          console.log(ab.duration, ab.sampleRate);
          const source = new AudioBufferSourceNode(oac, { buffer: ab });
          source.connect(oac.destination);
          source.start();
          await new Promise((r) => (source.onended = r));
          await decoder.flush();
          return;
        }
      }
    },
  });

  const encoded = await (await fetch('./encoded.json')).json();
  let base_time = encoded[encoded.length - 1].timestamp;
  console.assert(encoded.length > 0, encoded.length);
  console.log(JSON.stringify(encoded, null, 2));
  const metadata = encoded.shift();
  console.log(encoded[encoded.length - 1].timestamp, base_time);
  metadata.decoderConfig.description = new Uint8Array(
    base64ToBytesArr(metadata.decoderConfig.description)
  ).buffer;
  console.log(await AudioEncoder.isConfigSupported(metadata.decoderConfig));
  decoder.configure(metadata.decoderConfig);
  while (encoded.length) {
    const chunk = encoded.shift();
    chunk.data = new Uint8Array(base64ToBytesArr(chunk.data)).buffer;
    const eac = new EncodedAudioChunk(chunk);
    decoder.decode(eac);
  }
}

verifying the AudioData data and AudioBuffer channel data are incompatible.

Is there a prototype?
No.

Describe the feature in more detail

Web Audio API AudioBuffer <=> WebCodecs AudioData

Provide an algorithm and method to convert WebCodecs AudioData to Web Audio API AudioBuffer with option to set sample rate of the resulting object.

Christian Hoene:

in order to support high quality spatial audio rendering, individualized HRTF should be used.

Recently, the AES has standardized the SOFA format, which describes HRTFs.

So far, each browser just uses a default HRTF. I am wondering, whether Web Audio should support the SOFA format and thus individualized HRTFs.

Any plans?

Describe the feature
Add computedValue readonly property to AudioParam.

Is there a prototype?
I attempted to create an AudioWorkletNode which assisted with this but wasn't able to achieve anything useful since there is also no getter for the current computedValue of an AudioNode, and using the MessagePort available was either too intensive or too high latency, and timing too unpredictable to be considered a suitable workaround.

Describe the feature in more detail
It seems that previous inconsistent implementations of the getter on AudioParam.value in Chrome (which retrieved the computed value after summing any connected inputs) has been removed, in order to follow the spec which states it should return the current interpolated value based on the current automation event (AudioParam.linearRampToValueAtTime for example).

As stated in WebAudio/web-audio-api#1788, the AudioParam.value getter is often used to drive visual representations of the parameter, which means incorporating any inputs' contributions is desired. While I understand the need and reasoning behind going with the decision made in WebAudio/web-audio-api#1788, access to the computed value is still very much needed. As far as I can tell there is no straightforward way to access this value anymore.

As suggested in WebAudio/web-audio-api#1788, a readonly property of AudioParam (and ideally AudioNode too) called computedValue would solve this without introducing ambiguity into the current spec of setters and getters behaviour.

It was brought up that performance concerns might arise depending on the implementation, which is reasonable, but considering those performance issues would have existed in any previous implementations that did this, I don't see this as a major issue. Users of the API will realise if computedGetter causes performance issues and find a workaround if one exists for them, but in many cases, such a getter might be called 60 times a second or less, certainly not at audio rates - performance might not be a great concern for many users of the API, including myself.

Describe the feature
For example, the constructor for OfflineAudioContext takes an unsigned long for the length parameter. This means the actual length is the given length modulo 2^32, according to https://heycam.github.io/webidl/#abstract-opdef-converttoint

It might be useful if we added the [EnforceRange] attribute to cause an error to be thrown when the parameter is outside the range.

If this makes sense, we should probably apply this everywhere we use unsigned long.

It's certainly easier to understand if we get an error when we ask for an OfflineAudioContext of length 4294967297 and end up with no errors and a context with length 1.

Describe the feature

const ac = new AudioContext();
const aw = new AudioWorkletNode(ac, 'processor');
const {body: readable} = await fetch('/path/to/raw_pcm_output');
const rawPCMStreamNode = new RawPCMStreamNode(ac, readable);
aw.connect(rawPCMStreamNode);

Is there a prototype?

Yes. Rather involved, illustrating some of the deficiencies of SharedArrayBuffer and even dynamic SharedArrayBuffer memory growth using WebAssembly.Memory.prototype.grow(), thus this feature request to simplify the process.

From wasmerio/wasmer-php#121 (comment)

Put together a minimal, verifiable, complete working example of streaming from fetch() to AudioWorkletProcessor via Native Messaging, including the observable restriction which lead to filing this issue, and those issues referenced above, specifically, RangeError being thrown, and limitations on the amount of memory that can be pre-allocated at an ArrayBuffer or SharedArrayBuffer.

Posting the code here as not infrequently crash the browser and, or the OS in the midst of ongoing experiments.

Flow chart

• Create Native Messaging Chromium app (app)
• At an arbitrary web page ynamically set permissions for the app using Native File System to re-write manifest.json in app/ folder by setting the property externally_connectable value
• Connect to the backgound script, background.js, that passes messages between the Native Messaging host (in this case we use bash and execute PHP within the shell script) and the arbitrary web page
• We do not use the Native Messaging protocol to pass STDOUT to background.js then to the web page, instead we begin PHP built-in server process, localhost, to handle the subsequent call to fetch() with AbortController set within object passed as send parameter where we read STDOUT as a ReadableStream, converting to and from JSON and Uint8Array
• Create a MediaStreamAudioDestinationNode for the ability to record, perform other tasks with MediaStream, and MediaStreamTrack of live audio stream of system audio capture
• Use Transferable Streams to post response.body (ReadableStream) to AudioWorkletProcessor, where we pipe the stream to WritableStream and write Uint8Arrays to a single Uint8Array pre-allocated to a set amount (344 * 512 * 60 * 60 throws cannot allocate memory, here)
• Store minimal data before process() is started by resuming AudioContext
• Pass message to background.js to kill PHP built-in server process
• Close ReadableStreamDefaultController

Chromium Native Messaging extenstion.

app/

manifest.json

{
  // Extension ID: <id>
  "key": "<key>",
  "name": "native_messaging_stream",
  "version": "1.0",
  "manifest_version": 2,
  "description": "Capture system audio",
  "icons": {},
  "permissions": [
    "nativeMessaging", "*://localhost/"
  ],
  "app": {
    "background": {
      "scripts": [
        "background.js"
      ],
      "persistent": false
    }
  },

  "externally_connectable": {
    "matches": [
      "https://developer.mozilla.org/*",
      "http://localhost/*"
    ],
    "ids": [
      "lmkllhfhgnmbcmhdagfhejkpicehnida"
    ]
  },
  "author": "guest271314"
}

background.js

const id = 'native_messaging_stream';
let externalPort, controller, signal;

chrome.runtime.onConnectExternal.addListener(port => {
  console.log(port);
  externalPort = port;
  externalPort.onMessage.addListener(message => {
    if (message === 'start') {
      chrome.runtime.sendNativeMessage(id, {}, async _ => {
        console.log(_);
        if (chrome.runtime.lastError) {
          console.warn(chrome.runtime.lastError.message);
        }
        controller = new AbortController();
        signal = controller.signal;
        // wait until bash script completes, server starts
        for await (const _ of (async function* stream() {
          while (true) {
            try {
              if ((await fetch('http://localhost:8000', { method: 'HEAD' })).ok)
                break;
            } catch (e) {
              console.warn(e.message);
              yield;
            }
          }
        })());
        try {
          const response = await fetch('http://localhost:8000?start=true', {
            cache: 'no-store',
            mode: 'cors',
            method: 'get',
            signal
          });
          console.log(...response.headers);
          const readable = response.body;
          readable
            .pipeTo(
              new WritableStream({
                write: async value => {
                  // value is a Uint8Array, postMessage() here only supports cloning, not transfer
                  externalPort.postMessage(JSON.stringify(value));
                },
              })
            )
            .catch(err => {
              console.warn(err);
              externalPort.postMessage('done');
            });
        } catch (err) {
          console.error(err);
        }
      });
    }
    if (message === 'stop') {
      controller.abort();
      chrome.runtime.sendNativeMessage(id, {}, _ => {
        if (chrome.runtime.lastError) {
          console.warn(chrome.runtime.lastError.message);
        }
        console.log('everything should be done');
      });
    }
  });
});

set_externally_connectable.js

(async(set_externally_connectable = ["https://example.com/*"], unset_externally_connectable = true) => {
  const dir = await self.showDirectoryPicker();
  const status = await dir.requestPermission({writable: true});
  const fileHandle = await dir.getFileHandle("manifest.json", {create: false});
  const file = await fileHandle.getFile();
  const manifest_text = await file.text();
  const match_extension_id = /\/\/ Extension ID: \w{32}/;
  const [extension_id] = manifest_text.match(match_extension_id);
  let text = manifest_text.replace(match_extension_id, `"_": 0,`);
  const manifest_json = JSON.parse(text);
  manifest_json.externally_connectable.matches = unset_externally_connectable ? set_externally_connectable :
    [...manifest_json.externally_connectable.matches, ...set_externally_connectable];
  const writer = await fileHandle.createWritable({keepExistingData:false});
  await writer.write(JSON.stringify(manifest_json, null, 2).replace(/"_": 0,/, extension_id)); 
  return await writer.close();
})([`${location.origin}/*`]);

host/

native_messaging_stream.json ($ cp native_messaging_stream.json ~/.config/chromium/NativeMessagingHosts)

{
  "name": "native_messaging_file_stream",
  "description": "Capture system audio",
  "path": "/path/to/host/captureSystemAudio.sh",
  "type": "stdio",
  "allowed_origins": [
    "chrome-extension://<id>/"
  ],
  "author": "guest271314"
}

index.php

<?php 
  if($_SERVER['REQUEST_METHOD'] == 'HEAD') {
    header('Vary: Origin');
    header("Access-Control-Allow-Origin: chrome-extension://lmkllhfhgnmbcmhdagfhejkpicehnida");
    header("Access-Control-Allow-Methods: GET, OPTIONS, HEADERS");
    header("Access-Control-Allow-Headers: Content-Type, Access-Control-Allow-Headers");    
    header("X-Powered-By:");
    header("HTTP/1.1 200 OK");
    die();
  }
  if (isset($_GET["start"])) {
    header('Vary: Origin');
    header("Access-Control-Allow-Origin: chrome-extension://lmkllhfhgnmbcmhdagfhejkpicehnida");
    header("Access-Control-Allow-Methods: GET, OPTIONS, HEAD");
    header("Access-Control-Allow-Headers: Content-Type, Access-Control-Allow-Headers");    
    header("Content-Type: text/plain");
    header("X-Powered-By:");
    echo passthru("parec -v --raw -d alsa_output.pci-0000_00_1b.0.analog-stereo.monitor");
    exit();
  }

captureSystemAudio.sh

#!/bin/bash
sendMessage() {
    # https://stackoverflow.com/a/24777120
    message='{"message": "ok"}'
    # Calculate the byte size of the string.
    # NOTE: This assumes that byte length is identical to the string length!
    # Do not use multibyte (unicode) characters, escape them instead, e.g.
    # message='"Some unicode character:\u1234"'
    messagelen=${#message}
    # Convert to an integer in native byte order.
    # If you see an error message in Chrome's stdout with
    # "Native Messaging host tried sending a message that is ... bytes long.",
    # then just swap the order, i.e. messagelen1 <-> messagelen4 and
    # messagelen2 <-> messagelen3
    messagelen1=$(( ($messagelen      ) & 0xFF ))               
    messagelen2=$(( ($messagelen >>  8) & 0xFF ))               
    messagelen3=$(( ($messagelen >> 16) & 0xFF ))               
    messagelen4=$(( ($messagelen >> 24) & 0xFF ))               
    # Print the message byte length followed by the actual message.
    printf "$(printf '\\x%x\\x%x\\x%x\\x%x' \
        $messagelen1 $messagelpen2 $messagelen3 $messagelen4)%s" "$message"
}

captureSystemAudio() {
  if pgrep -f php > /dev/null; then
    killall -9 php & sendMessage  
  else
    php -S localhost:8000 -t /path/to/host/ & sendMessage
  fi
}
captureSystemAudio

at the arbitrary web page, in this case, MDN, where we previously set the matching URL pattern at externally_connectable for permission to communicate with the Chromium Native Messaging application

At web page

var id = 'lmkllhfhgnmbcmhdagfhejkpicehnida';
var port = chrome.runtime.connect(id);
var controller;
var readable = new ReadableStream({
  start(c) {
    return (controller = c);
  },
});
var init = false;
port.onMessage.addListener(async message => {
  if (message !== 'done') {
    if (!init) {
      init = true;
      class AudioWorkletProcessor {}
      class AudioWorkletNativeMessageStream extends AudioWorkletProcessor {
        constructor(options) {
          super(options);
          this.byteSize = 512 * 384 * 60 * 1; // 5 minutes of data
          this.memory = new Uint8Array(this.byteSize); // TODO: grow memory, dynamically
          Object.assign(this, options.processorOptions);
          this.port.onmessage = this.appendBuffers.bind(this);
        }
        async appendBuffers({ data: readable }) {
          Object.assign(this, { readable });
          const source = {
            write: (value, controller) => {
              console.log(globalThis.currentTime % 3);
              if (this.totalBytes + value.byteLength < this.byteSize) {
                this.memory.set(value, this.readOffset);
                this.readOffset = this.readOffset + value.buffer.byteLength;
                this.totalBytes = this.readOffset;
              } else {
                const last = value.subarray(0, this.byteSize - this.totalBytes);
                this.memory.set(last, this.readOffset);
                this.readOffset = this.readOffset + last.length;
                this.totalBytes = this.readOffset;
                console.log(
                  value.buffer.byteLength,
                  this.readOffset,
                  this.totalBytes
                );
                controller.close();

              }
              // await 250 milliseconds of audio data
              if (this.totalBytes >= (512 * 384) / 4 && !this.started) {
                this.started = true;
                this.port.postMessage({ started: this.started });
              }
            },
            close() {
              console.log('stopped');
            },
          };
          try {
            await this.readable.pipeTo(new WritableStream(source));
          } catch (e) {
            console.warn(e);
            console.log(this.writeOffset, this.totalBytes);
            this.endOfStream();
          }
        }
        endOfStream() {
          this.port.postMessage({
            ended: true,
            currentTime,
            currentFrame,
            readOffset: this.readOffset,
            writeOffset: this.writeOffet,
            totalBytes: this.totalBytes,
          });
        }
        process(inputs, outputs) {
          const channels = outputs.flat();
          if (
            this.writeOffset >= this.totalBytes ||
            this.totalBytes === this.byteSize
          ) {
            console.log(this);
            this.endOfStream();
            return false;
          }

          const uint8 = new Uint8Array(512);
          try {
            for (let i = 0; i < 512; i++, this.writeOffset++) {
              if (this.writeOffset === this.byteSize) {
                break;
              }
              uint8[i] = this.memory[this.writeOffset];
            }
            const uint16 = new Uint16Array(uint8.buffer);
            // https://stackoverflow.com/a/35248852
            for (let i = 0, j = 0, n = 1; i < uint16.length; i++) {
              const int = uint16[i];
              // If the high bit is on, then it is a negative number, and actually counts backwards.
              const float =
                int >= 0x8000 ? -(0x10000 - int) / 0x8000 : int / 0x7fff;
              // interleave
              channels[(n = ++n % 2)][!n ? j++ : j - 1] = float;
            }
            // console.log(channels[0]);
          } catch (e) {
            console.error(e);
          }

          return true;
        }
      }
      // register processor in AudioWorkletGlobalScope
      function registerProcessor(name, processorCtor) {
        return `${processorCtor};\nregisterProcessor('${name}', ${processorCtor.name});`;
      }
      const worklet = URL.createObjectURL(
        new Blob(
          [
            registerProcessor(
              'audio-worklet-native-message-stream',
              AudioWorkletNativeMessageStream
            ),
          ],
          { type: 'text/javascript' }
        )
      );
      const ac = new AudioContext({
        latencyHint: 1,
        sampleRate: 44100,
        numberOfChannels: 2,
      });
      ac.onstatechange = e => console.log(ac.state);
      if (ac.state === 'running') {
        await ac.suspend();
      }
      await ac.audioWorklet.addModule(worklet);
      const aw = new AudioWorkletNode(ac, 'audio-worklet-native-message-stream', {
        numberOfInputs: 1,
        numberOfOutputs: 2,
        channelCount: 2,
        processorOptions: {
          totalBytes: 0,
          readOffset: 0,
          writeOffset: 0,
          done: false,
          ended: false,
          started: false,
        },
      });

      aw.onprocessorerror = e => {
        console.error(e);
        console.trace();
      };

      const msd = new MediaStreamAudioDestinationNode(ac);
      const { stream } = msd;
      const [track] = stream.getAudioTracks();
      aw.connect(msd);
      const recorder = new MediaRecorder(stream);
      recorder.ondataavailable = e => console.log(URL.createObjectURL(e.data));
      aw.port.onmessage = async e => {
        console.log(e.data);
        if (
          e.data.started &&
          ac.state === 'suspended' &&
          recorder.state === 'inactive'
        ) {
          recorder.start();
          await ac.resume();
          setTimeout(_ => {
            port.postMessage('stop');
            controller.close();
          }, 1000 * 60 * 1);

        } else if (recorder.state === 'recording') {
          recorder.stop();
          track.stop();
          aw.disconnect();
          msd.disconnect();
          await ac.close();
          console.log(track);
          gc();
        }
      };

      aw.port.postMessage(readable, [readable]);

    }
    if (readable.locked)
      controller.enqueue(Uint8Array.from(Object.values(JSON.parse(message))));
  } else {
    if (readable.locked) controller.close();
  }
});

port.postMessage('start');

A version using WebAssembly.Memory.grow() to dynamically grow memory
during the stream. The expected result is 30 minutes of audio being written to
SharedArrayBuffer. An error is thrown at

Testing at a legacy 4.15.0-20-lowlatency 32-bit kernel attempted to grow memory to 30 minutes of data, with maximum set to 1 hour of data
An error was thrown when calling grow(1) when current offset is 177143808

before grow 177012736
after grow 177078272
before grow 177078272
after grow 177143808
before grow 177143808
e4056867-5209-4bea-8f76-a86106ab83af:75 RangeError: WebAssembly.Memory.grow(): Unable to grow instance memory.
    at Memory.grow (<anonymous>)
    at Object.write (e4056867-5209-4bea-8f76-a86106ab83af:26)
appendBuffers @ e4056867-5209-4bea-8f76-a86106ab83af:75
async function (async)
appendBuffers @ e4056867-5209-4bea-8f76-a86106ab83af:73
177111040 177139656
{ended: true, currentTime: 1004.0308390022676, currentFrame: 44277760, readOffset: 177139656, writeOffset: 177111040, …}
MediaStreamTrack {kind: "audio", id: "e0958602-f0ea-4565-9e34-f624afce7c12", label: "MediaStreamAudioDestinationNode", enabled: true, muted: false, …}
blob:https://developer.mozilla.org/f9d716ff-c65e-4ddf-b7c7-5e385d0602ec
closed

resulting in 16 minutes and 43 seconds of data being written to memory, observable at recorded audio

Asked the question if 1GB is the maximum the underlying memory can grow at 32-bit architectures at https://bugs.chromium.org/p/v8/issues/detail?id=7881.

if (globalThis.gc) gc();
var id = 'lmkllhfhgnmbcmhdagfhejkpicehnida';
var port = chrome.runtime.connect(id);
var controller;
var readable = new ReadableStream({
  start(c) {
    return (controller = c);
  },
});
var init = false;
port.onMessage.addListener(async message => {
  if (message !== 'done') {
    if (!init) {
      init = true;
      class AudioWorkletProcessor {}
      class AudioWorkletNativeMessageStream extends AudioWorkletProcessor {
        constructor(options) {
          super(options);
          this.initial = (384 * 512) / 65536; // 1 minute
          this.maximum = (384 * 512 * 60 * 60) / 65536; // 1 hour
          this.byteSize = this.maximum * 65536;
          this.memory = new WebAssembly.Memory({
            initial: this.initial,
            maximum: this.maximum,
            shared: true,
          });
          Object.assign(this, options.processorOptions);
          this.port.onmessage = this.appendBuffers.bind(this);
        }
        async appendBuffers({ data: readable }) {
          Object.assign(this, { readable });
          
          const source = {
            write: (value, controller) => {
              if (
                this.totalBytes + value.byteLength >
                  this.memory.buffer.byteLength &&
                this.totalBytes + value.buffer.byteLength < this.byteSize
              ) {
                console.log('before grow', this.memory.buffer.byteLength);
                this.memory.grow(1);
                console.log('after grow', this.memory.buffer.byteLength);
              }

              const uint8_sab = new Uint8Array(this.memory.buffer, this.readOffset);
              // console.log(this.totalBytes, this.totalBytes % 65536, this.totalBytes / 65536, this.memory.buffer.byteLength);
              if (this.totalBytes + value.buffer.byteLength < this.byteSize) {
                

                for (
                  let i = 0;
                  i < value.buffer.byteLength;
                  i++, this.readOffset++
                ) {
                  uint8_sab[this.readOffset] = value[i];
                }

                this.totalBytes = this.readOffset;
              } else {
                const lastBytes = value.subarray(0, this.byteSize - this.totalBytes);
                // const uint8 = new Uint8Array(this.memory.buffer, this.readOffset);
                for (
                  let i = 0;
                  i < lastBytes.buffer.byteLength;
                  i++, this.readOffset++
                ) {
                  uint8_sab[this.readOffset] = value[i];
                }
                this.totalBytes = this.readOffset;
                console.log(
                  value.buffer.byteLength,
                  this.readOffset,
                  this.totalBytes
                );
                this.readable.cancel();
              }
              // accumulate 250 milliseconds of data before resuming AudioContext
              if (this.totalBytes >= (512 * 384) / 4 && !this.started) {
                this.started = true;
                this.port.postMessage({ started: this.started });
              }
            },
            close() {
              console.log('stream closed');
            },
          };
          try {
            await this.readable.pipeTo(new WritableStream(source));
          } catch (e) {
            console.warn(e);
            console.log(this.writeOffset, this.totalBytes);
            this.endOfStream();
          }
        }
        endOfStream() {
          this.port.postMessage({
            ended: true,
            currentTime,
            currentFrame,
            readOffset: this.readOffset,
            writeOffset: this.writeOffset,
            totalBytes: this.totalBytes,
          });
        }
        process(inputs, outputs) {
          const channels = outputs.flat();
          if (
            this.writeOffset >= this.totalBytes ||
            this.totalBytes === this.byteSize
          ) {
            console.log(this);
            this.endOfStream();
            return false;
          }

          const uint8 = new Uint8Array(512);
          const uint8_sab = new Uint8Array(this.memory.buffer, this.writeOffset); // .slice(this.writeOffset, this.writeOffset + 512)
          
          try {
            for (let i = 0; i < 512; i++, this.writeOffset++) {
              if (this.writeOffset === this.byteSize) {
                break;
              }
              uint8[i] = uint8_sab[this.writeOffset];
            }

            const uint16 = new Uint16Array(uint8.buffer);
            // https://stackoverflow.com/a/35248852
            for (let i = 0, j = 0, n = 1; i < uint16.length; i++) {
              const int = uint16[i];
              // If the high bit is on, then it is a negative number, and actually counts backwards.
              const float =
                int >= 0x8000 ? -(0x10000 - int) / 0x8000 : int / 0x7fff;
              // interleave
              channels[(n = ++n % 2)][!n ? j++ : j - 1] = float;
            }
            // console.log(channels[0]);
          } catch (e) {
            console.error(e);
          }

          return true;
        }
      }
      // register processor in AudioWorkletGlobalScope
      function registerProcessor(name, processorCtor) {
        return `${processorCtor};\nregisterProcessor('${name}', ${processorCtor.name});`;
      }
      const worklet = URL.createObjectURL(
        new Blob(
          [
            registerProcessor(
              'audio-worklet-native-message-stream',
              AudioWorkletNativeMessageStream
            ),
          ],
          { type: 'text/javascript' }
        )
      );
      const ac = new AudioContext({
        latencyHint: 1,
        sampleRate: 44100,
        numberOfChannels: 2,
      });
      ac.onstatechange = e => console.log(ac.state);
      if (ac.state === 'running') {
        await ac.suspend();
      }
      await ac.audioWorklet.addModule(worklet);
      const aw = new AudioWorkletNode(
        ac,
        'audio-worklet-native-message-stream',
        {
          numberOfInputs: 1,
          numberOfOutputs: 2,
          channelCount: 2,
          processorOptions: {
            totalBytes: 0,
            readOffset: 0,
            writeOffset: 0,
            done: false,
            ended: false,
            started: false,
          },
        }
      );

      aw.onprocessorerror = e => {
        console.error(e);
        console.trace();
      };

      const msd = new MediaStreamAudioDestinationNode(ac);
      const { stream } = msd;
      const [track] = stream.getAudioTracks();
      aw.connect(msd);
      const recorder = new MediaRecorder(stream);
      recorder.ondataavailable = e => console.log(URL.createObjectURL(e.data));
      aw.port.onmessage = async e => {
        console.log(e.data);
        if (
          e.data.started &&
          ac.state === 'suspended' &&
          recorder.state === 'inactive'
        ) {
          recorder.start();
          await ac.resume();
          setTimeout(_ => {
            port.postMessage('stop');
            controller.close();
          }, 1000 * 60 * 30);
        } else if (recorder.state === 'recording') {
          recorder.stop();
          track.stop();
          aw.disconnect();
          msd.disconnect();
          await ac.close();
          console.log(track);
          gc();
        }
      };

      aw.port.postMessage(readable, [readable]);
    }
    if (readable.locked)
      controller.enqueue(Uint8Array.from(Object.values(JSON.parse(message))));
  } else {
    if (readable.locked) controller.close();
  }
});

port.postMessage('start');

Ideally, in this case, we should be able to stream directly to AudioWorkletProcessor.process() instead of writing to memory, i.e., the gist of the proposal at Chromium: Use Streams API for STDIN and STDOUT, flushing memory as we proceed.

One approach to reduce memory usage would be to write Opus audio to memory and decode to raw PCM as Float32Array values at outputs channels, which have not yet tried.

Describe the feature in more detail

Internally RawPCMStreamNode (or simply StreamNode or InputStreamNode) is a TransformStream that accepts a ReadableStream (readable) of raw PCM, transforms the raw data (the terms "interleaved" and "planar" are no longer included in Web Audio API v1) into Float32Array(s) corresponding to the number of channels contained in the input stream and sets the Float32Array(s) as outputs in process().

Externally his will work in the same way as MediaElementSourceNode

<body>
  <audio controls autoplay crossOrigin="anonymous"></audio>
  <script>
    (async() => {
      const context = new AudioContext({latencyHint:1.0});
      await context.suspend();
      const mediaElement = document.querySelector("audio");
      mediaElement.onloadedmetadata = async e => await context.resume();
      const source = new MediaElementAudioSourceNode(context, {
        mediaElement
      });
      await context.audioWorklet.addModule("bypass-processor.js");
      const bypasser = new AudioWorkletNode(context, "bypass-processor");
      source.connect(bypasser);
      bypasser.connect(context.destination);
      mediaElement.src = "https://ia800301.us.archive.org/10/items/DELTAnine2013-12-11.WAV/Deltanine121113Pt3Wav.wav";
    })();
  </script>
</body>

where when connected to an AudioWorkletNode AudioWorkletProcessor.process() effectively takes input from src of HTMLMediaElement and resulting audio pipeline and outputs Float32Arrays at outputs.

This will provide an entry point for users to utilize AudioContext for non-standard or non-implemented device capture and processing with Web Audio API.

WebCodecs does not solve this - unless that solution can be unequivocally proven here by evidence, not conjecture.

Related art:

• https://github.com/w3c/webrtc-insertable-streams/blob/master/explainer.md

Related issue:

• w3c/webrtc-encoded-transform#41

Describe the Issue
Once we get spec text, we need Travis to verify the spec is correct

Expected Result
Travis should run bikeshed to generate the doc and verify that things are correct.

Actual Result
Travis CI isn't enabled yet.

How can one stream music files while a) setting custom loop points for the loops to generate a seamless looping audio and b) adjusting (by possibly animating) the pitch of the playback?

Currently one can download the music file and .decodeAudioData() it fully, and then use AudioBufferSourceNode and its .loopStart, .loopEnd and .playbackRate attributes to specify the loop points and the pitch. However this consumes 100MB's + of memory and takes an awfully long time to perform the decoding because everything has to be uncompressed in memory for it to work, so it is not a viable solution.

If one uses a MediaElementAudioSourceNode, it is possible to pass the compressed music file directly for playback, and loop it, but looping is restricted to fully looping from begin to the end, which is not enough. It is either not possible to adjust pitch of the audio output (the .playbackRate performs pitch correction for that case, which is undesirable)

If AudioBufferSourceNode was able to be populated with compressed audio data, it would solve this use case. The Web Audio API would then decompress the AudioBufferSourceNode contents while it's playing it back in a streaming manner without having to decode the whole clip in advance. Would that sound like a feasible addition?

(Note that the interest here is in streamed decompression, not in streamed download - the compressed audio assets have been fully downloaded on the page prior)

WebCodecs uses AudioBuffer as a container of unencoded audio.
https://wicg.github.io/web-codecs/#dom-audioframe-buffer

The AudioFrame, which wraps AudioBuffer, is provided as an input to AudioEncoder and as an output from AudioDecoder. We've exposed the encoder and decoder interfaces to DedicatedWorker to facilitate offloaded codec IO.

I think the change is as simple as changing the Exposed= parameter for just AudioBuffer. I'll send a PR to demonstrate.

Describe the issue
The description for cancelTime parameter for both cancelScheduledValues and cancelAndHoldAtTime says that non-finite values should throw a RangeError. However, since the declared type is double and not unrestricted double, a TypeError is thrown instead.

We should update the text to say that a RangeError is thrown for negative values (deleting the part about non-finite values).

Where Is It
For cancelScheduledValues, see https://webaudio.github.io/web-audio-api/#dom-audioparam-cancelscheduledvalues-canceltime

For cancelAndHoldAtTime, see https://webaudio.github.io/web-audio-api/#dom-audioparam-cancelandholdattime-canceltime

Describe the Issue
See WebAudio/web-audio-api#2106 for a PR to add a pull request template. We're not going to do that for v1, but we will for v2, as mentioned in WebAudio/web-audio-api#2106 (comment)

This is a reminder for us to take that merge request and move it/duplicate it to this repo.

https://webaudio.github.io/web-audio-api/#fft-windowing-and-smoothing-over-time specifies the post-processing of the complex FFT result vector to obtain smoothed log-magnitudes for getFloatFrequencyData() in AnalyserNode. I want to suggest an efficiency improvement here.

Leaving out smoothing for the moment, the post-processing computes each element of the result vector as Y[k] = 20*log10(sqrt(r[k]*r[k] + i[k]*i[k])), where r[k] and i[k] are the real and imaginary components of the complex FFT result X[k]. However, the sqrt() operation here can be eliminated, since Y[k] = 10*log10(r[k]*r[k] + i[k]*i[k]) is mathematically identical.

Smoothing complicates this picture, since the two forms (with and without square root) are now no longer identical. Taking a step back, however, we can ask why there is smoothing before taking logs - what does that achieve? I believe that the unstated purpose here is to create a peak detector: exponential smoothing before taking logs creates an impulse response with exponentially fast attack but slow linear decay to zero, both at a rate governed by the smoothing constant tau.

The proposed sqrt-free version would make this asymmetry more pronounced - essentially doubling the attack and halving the decay rate. We could improve backwards compatibility by internally substituting tau := 2*tau - 1 to re-double the decay rate, at the cost of now having quadruple the previous attack rate.

If this sounds bad, there are three mitigating factors at work here:

1. The slow decay is far more visible than the fast attack, so users use tau primarily to control the decay rate, which will remain the same.
2. Arguably most users want an instantaneous attack - a true peak detector. If they notice the quadrupled attack rate at all, they're more likely than not to be happy about it.
3. In the rare cases where a faster attack is not welcome, it is always possible to compensate by introducing additional smoothing after the log, i.e., on the user side.

Finally, I can provide a fast approximate log if there's interest. Cheers!

Unless I'm mistaken, it seems the Web Audio API spec does not define a standard way to manually terminate an OfflineAudioContext.

The closest we have is calling suspend on an already running OfflineAudioContext, which does not dispose the context however, and will accumulate used memory. close is specced only to AudioContext (not AudioContextBase or OfflineAudioContext), so we can't use that here.

Shouldn't closing be possible on OfflineAudioContext as well?

Use case

Our use case is a DAW that supports hour long compositions causing an OfflineAudioContext.startRendering call to sometimes take up an entire hour, and it would be nice to offer users a way to cancel the process.

The only workaround I know about at this time is window.location.reload(), and thus perhaps running the rendering process in an iframe, which can be reloaded without reloading my entire app.

Following up on issue WebAudio/web-audio-api#1848, which briefly discusses why a blocking wait like Atomics.wait is detrimental to real-time audio processing, in particular this comment made by @hoch:

For real-time audio use case, synchronously blocking the thread is generally not a good idea.

... and this comment by @padenot:

one should never ever ever wait in a real-time audio callback, so we disabled Atomic.wait [...] if they are trying to wait on anything, they are doing something wrong

To briefly reflect on these thoughts, I think it should be pointed out that, strictly speaking, none of these apply to offline audio processing, and consequently, I think there is probably no reason whatsoever to disable Atomics.wait in AudioWorklets associated with an OfflineAudioContext (no reason other than the consistency of available APIs, that is).

In fact, for offline audio rendering that involves cross-thread audio processing, a blocking wait is essentially necessary at one point to guarantee cross-thread processing completion, especially if low or zero latency is necessary, due to how an offline rendering thread schedules its process calls (i.e. as fast as computationally possible, without considering cross-thread audio processing load).

This is currently possible and feasible with a busy wait (and there are plenty of high-level workarounds as well).

Any thoughts on making Atomics.wait available in offline rendering, to replace busy waiting in these cases?

Describe the issue

console.log('compileStreaming' in globalThis.WebAssembly);
console.log('instantiateStreaming' in globalThis.WebAssembly);

both return true at Nightly 78 and Chromium 81.

However, those methods cannot be used in AudioWorkletGlobalScope without throwing a TypeError because Response is not defined in AudioWorkletGlobalScope

https://webassembly.org/docs/web/#process-a-potential-webassembly-response

Process a potential WebAssembly response
The above two functions both reuse much of the same infrastructure for extracting bytes from an appropriate Response object, differing only in what they do with those bytes in the end. As such we define the following shared spec-level procedure:

...

Given these values, to process a potential WebAssembly response:

1. Let returnValue be a new promise.
2. Let sourceAsPromise be a promise resolved with argument.
3. Upon fulfillment of sourceAsPromise with value unwrappedSource:
   i. If unwrappedSource is not a Response object, reject returnValue with a TypeError exception and abort these substeps.

Specification authors object to defining fetch(), thus Response, in AudioWorkletGlobalScope where the methods now are only capable of throwing a TypeError per the WebAsssembly specification, thus should be removed from AudioWorkletGlobalScope.WebAssembly implementation.

Where Is It

1.32.1. Concepts

The AudioWorklet object allows developers to supply scripts (such as JavaScript or WebAssembly code) to process audio on the rendering thread, supporting custom AudioNodes. This processing mechanism ensures synchronous execution of the script code with other built-in AudioNodes in the audio graph.

https://webassembly.org/docs/web/#additional-web-embedding-api

Additional Web Embedding API

In Web embeddings, the following methods are added.

Note that it is expected that compileStreaming and instantiateStreaming be either both present or both absent.

Additional Information

Where specification authors appear to be in firm agreement to not define fetch(), and thus Response, in AudioWorkletGlobalScope #73, where if that decision remains unchanged compileStreaming() and instantiateStreaming() will always only throw a TypeError rendering the methods useless in AudioWorkletGlobalScope.

The WebAssembly specification provides the solution to the bug "both absent" must apply in this case.

If the specification authors conclude that a TypeError being thrown for WebAssembly.compileStreaming() and WebAssembly.instantiateStreaming() which is the only possible outcome due to Response not being defined in AudioWorkletGlobalScope at a minimum a note should be included in the specification to notify users that while compileStreaming() and instantiateStreaming() are defined methods within AudioWorkletGlobalScope using either will only result in a TypeError being thrown as Response is not defined in the AudioWorkletGlobalScope.