Demoting AudioContext into an OfflineAudioContext about web-audio-api-v2 HOT 5 CLOSED

webaudio commented on August 24, 2024 3

Demoting AudioContext into an OfflineAudioContext

from web-audio-api-v2.

Comments (5)

padenot commented on August 24, 2024

We talked about this during today's call.

This is hard to do, because AudioContext have methods and nodes that are not present in OfflineAudioContext. What happens if you're receiving data from a PeerConnection using MediaStreamTrackAudioSourceNode, and then you turn it into an OfflineAudioContext, where this AudioNode doesn't exist?

How are AudioParam changes scheduled? I assume one would have to re-do all the AudioParam automation calls? AudioContext.currentTime would reset to 0?

Additionally, changing the inheritance is a breaking change, this wouldn't evaluate to true:

Object.getPrototypeOf(AudioContext).name == "BaseAudioContext"

from web-audio-api-v2.

guest271314 commented on August 24, 2024

For

export a song as a wav

as long as the Float32Array representation of the audio is accessible the WAV file header can be set at any time. To that end if the song is live AudioWorkletProcessor can be used as a bypass for any connected audio node https://github.com/web-platform-tests/wpt/blob/d5be80a86d4f938250c075ac12414ad47516969c/webaudio/js/worklet-recorder.js then WAV headers can be set once the start and end times of the audio to be exported are reached using, for example, https://github.com/mattdiamond/Recorderjs/blob/master/src/recorder.js.

Alternatively, at Chromium, Chrome MediaRecorder with mimeType set to "video/x-matroska;codecs=pcm" or "audio/webm;codecs=pcm" can be used to record all audio as PCM, which can then be extracted from the Matroska file (e.g., using mkv2xml or ts-ebml) and converted to Float32Array then further processed to converted to a WAV file proper, see #63.

Another alternative is using Native Messaging to convert any file or data input type to WAV. E.g., opus-tools to convert Opus to WAV with opusdec, e.g., https://github.com/guest271314/native-messaging-espeak-ng downloads opus-tools to convert WAV output by espeak-ng to OGG.

from web-audio-api-v2.

jariseon commented on August 24, 2024

thanks for discussing the feature request in the group. i kind of expected that i failed to see the big picture :)

suspend() was indeed a bad idea. instead of suspending could a new AudioContext state maybe work better? For example, going offline with context.offline() - which would fail should the context contain MediaElement/Stream nodes or something else unavailable in the BaseAudioContext - and calling context.resume() to switch the state back to realtime.

re-scheduling AudioParams sounds right at least for the present use case. no idea how to resolve the breaking change in the inheritance pattern though, or how to respond to OfflineAudioContext methods when the state is realtime.

but yeah, there might well be further issues that are outside my limited perspective, so pls feel free to close the issue as you see best.

from web-audio-api-v2.

padenot commented on August 24, 2024

We discussed this again in the call.

There are too many differences and problems to solve for this to be solved at the Web Audio API level, and I think in general that a DAW always will have its own transport layer and automation scheduling on this transport, different from the AudioContext.currentTime, so suspend wouldn't work as you say.

You can alleviate the memory requirements by rendering track by track and performing a mix down, or creating nodes at the right time using suspend(), or destroying your real-time Web Audio API graph to create an Offline one.

from web-audio-api-v2.

guest271314 commented on August 24, 2024

we have two online daw use cases where the user needs to A) export a song as a wav,

Minimally modifying existing code from https://github.com/web-platform-tests/wpt/blob/d5be80a86d4f938250c075ac12414ad47516969c/webaudio/js/worklet-recorder.js to use AudioWorklet for processing "real-time" playback and https://github.com/chromium/chromium/blob/77578ccb4082ae20a9326d9e673225f1189ebb63/third_party/blink/web_tests/webaudio/resources/audio-file-utils.js to process either real-time or non-real-time input audio data to wav output as a file

<!DOCTYPE html>

<html>
  <head>
    <title>Audio input to wav file</title>
  </head>

  <body>
    <input type="file" />
    <audio controls></audio>
    <script type="worklet">
      // https://github.com/web-platform-tests/wpt/blob/d5be80a86d4f938250c075ac12414ad47516969c/webaudio/js/worklet-recorder.js
      /**
       * @class RecorderProcessor
       * @extends AudioWorkletProcessor
       *
       * A simple recorder AudioWorkletProcessor. Returns the recorded buffer to the
       * node when recording is finished.
       */
      class RecorderProcessor extends AudioWorkletProcessor {
        /**
         * @param {*} options
         * @param {number} options.processorOption.duration A duration to record in seconds.
         * @param {number} options.processorOptions.channelCount A channel count to record.
         */
        constructor(options) {
          super();
          console.log(options);
          this._createdAt = currentTime;
          this._elapsed = 0;
          this._recordDuration =
            (options.processorOption && options.processorOptions.duration) || 1;
          this._recordChannelCount =
            (options.processorOptions && options.processorOptions.channelCount) || 1;
          this._recordBufferLength = sampleRate * this._recordDuration;
          this._recordBuffer = [];
          for (let i = 0; i < this._recordChannelCount; ++i) {
            this._recordBuffer[i] = [];
            // new Float32Array(this._recordBufferLength);
          }
          this.stop = false;
          this.port.onmessage = e => {
            this.stop = true;
            this.port.onmessage = null;
          }
        }

        process(inputs, outputs) {
          if (this.stop) {
            this.port.postMessage({
              currentFrame,
              sampleRate,
              type: 'recordfinished',
              recordBuffer: this._recordBuffer.map(floats => new Float32Array(floats))
            });
            this.port.close();
            return false;
          }

          // Records the incoming data from |inputs| and also bypasses the data to
          // |outputs|.
          const input = inputs[0];
          const output = outputs[0];
          if (!input.length) {
            console.log(input);
            return true;
          }
          for (let channel = 0; channel < input.length; channel++) {
            let inputChannel, outputChannel;
            try {
              inputChannel = input[channel];
              outputChannel = output[channel];
              outputChannel.set(inputChannel);
              // console.log(this._recordBuffer, channel);
              const buffer = this._recordBuffer[channel];
              // const capacity = buffer.length - currentFrame;
              buffer.push(...inputChannel.slice(0));
            } catch (e) {
            console.error(e, {channel, inputs, outputs, input, output});
              // throw e;
            }
          }

          return true;
        }
      }

      registerProcessor('recorder-processor', RecorderProcessor);
    </script>
    <script>
      // https://github.com/chromium/chromium/blob/77578ccb4082ae20a9326d9e673225f1189ebb63/third_party/blink/web_tests/webaudio/resources/audio-file-utils.js
      // Utilities for creating a 16-bit PCM WAV file from an AudioBuffer
      // when using Chrome testRunner, and for downloading an AudioBuffer as
      // a float WAV file when running in a browser.
      function writeString(s, a, offset) {
        for (let i = 0; i < s.length; ++i) {
          a[offset + i] = s.charCodeAt(i);
        }
      }

      function writeInt16(n, a, offset) {
        n = Math.floor(n);

        let b1 = n & 255;
        let b2 = (n >> 8) & 255;

        a[offset + 0] = b1;
        a[offset + 1] = b2;
      }

      function writeInt32(n, a, offset) {
        n = Math.floor(n);
        let b1 = n & 255;
        let b2 = (n >> 8) & 255;
        let b3 = (n >> 16) & 255;
        let b4 = (n >> 24) & 255;

        a[offset + 0] = b1;
        a[offset + 1] = b2;
        a[offset + 2] = b3;
        a[offset + 3] = b4;
      }

      // Return the bits of the float as a 32-bit integer value.  This
      // produces the raw bits; no intepretation of the value is done.
      function floatBits(f) {
        let buf = new ArrayBuffer(4);
        new Float32Array(buf)[0] = f;
        let bits = new Uint32Array(buf)[0];
        // Return as a signed integer.
        return bits | 0;
      }

      function writeAudioBuffer(audioBuffer, a, offset, asFloat) {
        let n = audioBuffer.length;
        // let n = audioBuffer.reduce((a, b) => a + b.length, 0);
        let channels = audioBuffer.numberOfChannels;
        // let channels = audioBuffer.length;

        for (let i = 0; i < n; ++i) {
          for (let k = 0; k < channels; ++k) {
            let buffer = audioBuffer.getChannelData(k);
            // let buffer = audioBuffer[k];
            if (asFloat) {
              let sample = floatBits(buffer[i]);
              writeInt32(sample, a, offset);
              offset += 4;
            } else {
              let sample = buffer[i] * 32768.0;

              // Clip samples to the limitations of 16-bit.
              // If we don't do this then we'll get nasty wrap-around distortion.
              if (sample < -32768) sample = -32768;
              if (sample > 32767) sample = 32767;

              writeInt16(sample, a, offset);
              offset += 2;
            }
          }
        }
      }

      // See http://soundfile.sapp.org/doc/WaveFormat/ and
      // http://www-mmsp.ece.mcgill.ca/Documents/AudioFormats/WAVE/WAVE.html
      // for a quick introduction to the WAVE PCM format.
      function createWaveFileData(audioBuffer, asFloat) {
        let bytesPerSample = asFloat ? 4 : 2;
        let frameLength = audioBuffer.length; // audioBuffer[0].length
        let numberOfChannels = audioBuffer.numberOfChannels; // audioBuffer.length
        let sampleRate = audioBuffer.sampleRate; // ac.sampleRate; sampleRate
        let bitsPerSample = 8 * bytesPerSample;
        let byteRate = (sampleRate * numberOfChannels * bitsPerSample) / 8;
        let blockAlign = (numberOfChannels * bitsPerSample) / 8;
        let wavDataByteLength = frameLength * numberOfChannels * bytesPerSample;
        let headerByteLength = 44;
        let totalLength = headerByteLength + wavDataByteLength;

        let waveFileData = new Uint8Array(totalLength);

        let subChunk1Size = 16; // for linear PCM
        let subChunk2Size = wavDataByteLength;
        let chunkSize = 4 + (8 + subChunk1Size) + (8 + subChunk2Size);

        writeString('RIFF', waveFileData, 0);
        writeInt32(chunkSize, waveFileData, 4);
        writeString('WAVE', waveFileData, 8);
        writeString('fmt ', waveFileData, 12);

        writeInt32(subChunk1Size, waveFileData, 16); // SubChunk1Size (4)
        // The format tag value is 1 for integer PCM data and 3 for IEEE
        // float data.
        writeInt16(asFloat ? 3 : 1, waveFileData, 20); // AudioFormat (2)
        writeInt16(numberOfChannels, waveFileData, 22); // NumChannels (2)
        writeInt32(sampleRate, waveFileData, 24); // SampleRate (4)
        writeInt32(byteRate, waveFileData, 28); // ByteRate (4)
        writeInt16(blockAlign, waveFileData, 32); // BlockAlign (2)
        writeInt32(bitsPerSample, waveFileData, 34); // BitsPerSample (4)

        writeString('data', waveFileData, 36);
        writeInt32(subChunk2Size, waveFileData, 40); // SubChunk2Size (4)

        // Write actual audio data starting at offset 44.
        writeAudioBuffer(audioBuffer, waveFileData, 44, asFloat);

        return waveFileData;
      }

      function createAudioData(audioBuffer, asFloat) {
        return createWaveFileData(audioBuffer, asFloat);
      }

      function finishAudioTest(event) {
        let audioData = createAudioData(event.renderedBuffer);
        testRunner.setAudioData(audioData);
        testRunner.notifyDone();
      }

      // Save the given |audioBuffer| to a WAV file using the name given by
      // |filename|.  This is intended to be run from a browser.  The
      // developer is expected to use the console to run downloadAudioBuffer
      // when necessary to create a new reference file for a test.  If
      // |asFloat| is given and is true, the WAV file produced uses 32-bit
      // float format (full WebAudio resolution).  Otherwise a 16-bit PCM
      // WAV file is produced.
      function downloadAudioBuffer(
        audioBuffer,
        /* filename, */ asFloat = false
      ) {
        console.log(asFloat);
        // Don't download if testRunner is defined; we're running a layout
        // test where this won't be useful in general.
        // if (window.testRunner) return false;
        // Convert the audio buffer to an array containing the WAV file
        // contents.  Then convert it to a blob that can be saved as a WAV
        // file.
        let wavData = createAudioData(audioBuffer, /* filename, */ asFloat);
        let blob = new Blob([wavData], { type: 'audio/wav' });
        // Manually create html tags for downloading, and simulate a click
        // to download the file to the given file name.
        return blob;
        // audio.src = URL.createObjectURL(blob);
        /*
        let a = document.createElement('a');
        a.style.display = 'none';
        a.download = filename;
        let audioURL = window.URL.createObjectURL(blob);
        let audio = new Audio();
        audio.src = audioURL;
        a.href = audioURL;
        document.body.appendChild(a);
        a.click();
        */
        return true;
      }

      function audioInputToWav(audioInput, asFloat = false) {
        // TODO handle possible audio inputs
        let mediaElement,
          mediaStream,
          mediaStreamTrack,
          blob,
          file,
          blobURL,
          mediaFragmentURI,
          url,
          audioBuffer,
          array,
          typedArray;
        switch (Object.getPrototypeOf(audioInput).constructor.name) {
          case 'AudioBuffer':
            console.log(audioInput);
          case 'HTMLAudioElement':
          case 'HTMLVideoElement':
            console.log(audioInput);
            break;
          case 'MediaStream':
            console.log(audioInput);
            break;
          case 'MediaStreamTrack':
            console.log(audioInput);
            break;
          case 'Blob':
          case 'File':
            console.log(audioInput);
            break;
          case 'String':
            console.log(audioInput);
            break;
          case 'Array':
            console.log(audioInput);
            break;
          case 'Float32Array':
            console.log(audioInput);
            break;
          case 'ArrayBuffer':
            console.log(audioInput);
            break;
          case 'Uint8Array':
            console.log(audioInput);
            break;
          case 'Uint16Array':
            console.log(audioInput);
            break;
          default:
            break;
        }
      }

      const input = document.querySelector('input[type=file]');
      const mediaElement = document.querySelector('audio');

      input.onchange = async e => {
        const [file] = e.target.files;
        const ac = new AudioContext();
        // const ab = await ac.decodeAudioData(await file.arrayBuffer());
        ac.onstatechange = e => console.log(e);
        await ac.suspend();
        const audioWorklet = URL.createObjectURL(
          new Blob(
            [document.querySelector('script[type=worklet]').textContent],
            { type: 'text/javascript' }
          )
        );
        await ac.audioWorklet.addModule(audioWorklet);
        const source = new MediaElementAudioSourceNode(ac, { mediaElement });
        // const msd = new MediaStreamAudioDestinationNode(ac);

        // const { stream: mediaStream } = msd;
        // const ms = new MediaStreamAudioSourceNode(ac, { mediaStream });

        // const [track] = mediaStream.getAudioTracks();
        // track.onmute = track.onunmute = track.onended = e => {
        //  console.log(e);
        // };
        // audioInputToWav(mediaStream);
        audioInputToWav(mediaElement);
        audioInputToWav(file);
        // audioInputToWav(track);
        const aw = new AudioWorkletNode(ac, 'recorder-processor', {
          processorOptions: { channelCount: 2 },
        });
        aw.onprocessorerror = console.trace;
        aw.port.onmessage = async e => {
          const { recordBuffer, sampleRate, currentFrame } = e.data;
          // an Array could be substituted for AudioBuffer, here,
          // with changes made where numberOfChannels,
          // length, sampleRate are used in original code
          const audioBuffer = new AudioBuffer({
            length: currentFrame,
            sampleRate,
            numberOfChannels: recordBuffer.length,
          });
          recordBuffer.forEach((floats, index) => {
            audioBuffer.getChannelData(index).set(floats);
          });
          console.log(audioBuffer);
          const blob = downloadAudioBuffer(audioBuffer, true);
          audioInputToWav(blob);

          const audio = new Audio(URL.createObjectURL(blob));
          audio.controls = true;
          audio.onloadedmetadata = e => {
            console.log(e.target.duration, mediaElement.currentTime);
            audio.play();
          };
          audio.onpause = e => {
            console.log(e.target.duration, mediaElement.currentTime);
          };
          document.body.appendChild(audio);
        };
        source.connect(aw);
        // ms.connect(aw);
        aw.connect(ac.destination);

        const blobURL = URL.createObjectURL(file);
        audioInputToWav(blobURL);

        mediaElement.src = blobURL + '#t=0,5';

        mediaElement.oncanplaythrough = async _ => {
          console.log(mediaElement.src);
          mediaElement.oncanplaythrough = null;
          await ac.resume();
          await mediaElement.play();
          if ('mozCaptureStream' in mediaElement)
            setTimeout(_ => mediaElement.pause(), 5000);
        };
        mediaElement.onplay = async _ => {};
        mediaElement.onpause = e => {
          aw.port.postMessage('stop');
          // track.stop();
          mediaElement.onpause = null;
          console.log(e, source);
        };
      };
    </script>
  </body>
</html>

https://github.com/guest271314/audioInputToWav/blob/master/index.html

and B) freeze a (midi) track into an audio clip. currently we can reconstruct the realtime graph in an OfflineAudioContext on demand

Am not sure gather if there is any difference between a midi track and a MediaStreamTrack streaming midi data or what the exact expected output is. MediaRecorder.pause() and resume() could be used to record real-time media stream, omitting audio input not intended to be captured for further processing. Alternatively, an oscillator node could be created to stream silence continuously, and RTCRtpSender.replaceTrack() could be used to substitute stream of silence for midi track then replace the track back to midi or other track.

from web-audio-api-v2.

Demoting AudioContext into an OfflineAudioContext about web-audio-api-v2 HOT 5 CLOSED

Comments (5)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent