Currently we incorrectly upmix/downmix across ports instead of mixing multiple inputs to a single port.

We also don't support fan-in and fan-out for ports at all, and don't support multiple connections between nodes.

We need to:

• Modify Edge so it contains a smallvec of port pairs. Modify the graph code to support this well
• Modify the processing code to clone chunks in case of fan-out
• Modify the caching code to store a smallvec of chunks
• Modify the processing code to mix these chunks together (and ultimately sum them)

This would make it possible to test on CI and make it easy to build servo without gstreamer support if desired (such as when porting to a new platform).

Having OfflineAudioContext in place should fix some of the existing WPTs errors and should allow us to start running performance tests.

We must expose a way to control and know the current state of the audio context.

When I try this I get a single frozen videosink, and this error in the logs:

0:00:00.650705511 29662 0x7f35a8004ad0 WARN                 basesrc gstbasesrc.c:3055:gst_base_src_loop:<nicesrc3> error: Internal data stream error.
0:00:00.650721783 29662 0x7f35a8004ad0 WARN                 basesrc gstbasesrc.c:3055:gst_base_src_loop:<nicesrc3> error: streaming stopped, reason not-linked (-1)
0:00:00.907446112 29662 0x7f356048a6d0 WARN           basetransform gstbasetransform.c:1415:gst_base_transform_reconfigure:<videoconvert1> warning: not negotiated
0:00:00.907457873 29662 0x7f356048a6d0 WARN           basetransform gstbasetransform.c:1415:gst_base_transform_reconfigure:<videoconvert1> warning: not negotiated
0:00:00.956709545 29662 0x7f35bc02ee80 WARN                 basesrc gstbasesrc.c:3055:gst_base_src_loop:<nicesrc0> error: Internal data stream error.
0:00:00.956748859 29662 0x7f35bc02ee80 WARN                 basesrc gstbasesrc.c:3055:gst_base_src_loop:<nicesrc0> error: streaming stopped, reason not-negotiated (-4)

the "not linked" error is probably the relevant one, i suspect this cascades and breaks caps negotiation

the following patch which effectively reverts #186 fixes the problem. I'm not sure what's going on here, they're almost equivalent.

diff --git a/backends/gstreamer/src/webrtc.rs b/backends/gstreamer/src/webrtc.rs
index eff0af7..e5df76f 100644
--- a/backends/gstreamer/src/webrtc.rs
+++ b/backends/gstreamer/src/webrtc.rs
@@ -268,11 +268,13 @@ fn handle_media_stream(
             let q = gst::ElementFactory::make("queue", None).unwrap();
             let conv = gst::ElementFactory::make("audioconvert", None).unwrap();
             let resample = gst::ElementFactory::make("audioresample", None).unwrap();
+            let sink = gst::ElementFactory::make("autoaudiosink", None).unwrap();
 
-            pipe.add_many(&[&q, &conv, &resample])?;
-            gst::Element::link_many(&[&q, &conv, &resample])?;
+            pipe.add_many(&[&q, &conv, &resample, &sink])?;
+            gst::Element::link_many(&[&q, &conv, &resample, &sink])?;
 
             resample.sync_state_with_parent()?;
+            sink.sync_state_with_parent()?;
 
             let elements = vec![q.clone(), conv.clone(), resample];
             (q, conv, elements)
@@ -280,9 +282,11 @@ fn handle_media_stream(
         StreamType::Video => {
             let q = gst::ElementFactory::make("queue", None).unwrap();
             let conv = gst::ElementFactory::make("videoconvert", None).unwrap();
+            let sink = gst::ElementFactory::make("autovideosink", None).unwrap();
 
-            pipe.add_many(&[&q, &conv])?;
-            gst::Element::link_many(&[&q, &conv])?;
+            pipe.add_many(&[&q, &conv, &sink])?;
+            gst::Element::link_many(&[&q, &conv, &sink])?;
+            sink.sync_state_with_parent()?;
 
             let elements = vec![q.clone(), conv.clone()];
             (q, conv, elements)
diff --git a/backends/gstreamer/src/media_stream.rs b/backends/gstreamer/src/media_stream.rs
index 0623e1f..867fd55 100644
--- a/backends/gstreamer/src/media_stream.rs
+++ b/backends/gstreamer/src/media_stream.rs
@@ -156,6 +156,7 @@ impl MediaSink {
 
 impl MediaOutput for MediaSink {
     fn add_stream(&mut self, stream: Box<MediaStream>) {
+        return;
         {
             let stream = stream.as_any().downcast_ref::<GStreamerMediaStream>().unwrap();
             let last_element = stream.elements.last();

https://html.spec.whatwg.org/multipage/media.html#dom-navigator-canplaytype

For reference, this is how WebKit implements this https://trac.webkit.org/browser/webkit/trunk/Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp#L2341

This is not a high priority right now, but we need to start thinking about this.

We need to accommodate the current design to support playback for HTMLMediaElements. We can start with support for <audio>.

Some random thoughts about this:

• We need a different backend.
  For WebAudio we have a GStreamer src element that gets and deals with raw audio data.
  For <audio> (and <video>) we probably need a different element to, for example, read from http:// URIs, which typically means raw unidentified data that could be encoded or not, could be a live stream or not, etc.
  We may use GstPlayer for this, that does most part of the required work (via playbin). This is what @philn and @sdroege used for their

• We need to support having <audio> elements as sources of WebAudio pipelines as well (check MediaElementAudioSourceNode), which makes things a bit harder. I think we can instruct GstPlayer to output the processed stream into a custom audio sink from where we can inject the resulting buffer into the WebAudio pipeline.

• We need to differentiate between real-time and non-real-time audio rendering and figure out in general how (or even if we want/need) to reuse the current WebAudio graph as a wider media graph to manage the different streams coming from different media APIs.

With #54 in place, we can now deinterleave the decoded audio and create the AudioBuffer with the correct number of channels.

We can use GStreamer's deinterleave element for this.

As described in the spec.

From this comment:

For cycles: we can do the traversal a little bit more manually to support it, the algorithm isn't too hard. I was thinking once this is implemented we can parallelize out that work to support all the little graph things.

Just like we have PortIds, we should have ParamIds, and for a given kind of node, each ParamId corresponds to a specific parameter.

This will let us refer to them from the DOM side and correctly implement nodes-connected-to-params

(I'll implement this soon)

We switched from BaseSrc to AppSrc in #4 and this broke the Android example. We get sound in desktop but no sound at all in Android. I see this in the logcat:

05-17 11:23:57.494 7738 8344 W GStreamer+basesrc: 0:00:08.982054176 0xd4ae4af0 gstbasesrc.c:3055:gst_base_src_loop: error: Internal data stream error.
05-17 11:23:57.494 7738 8344 W GStreamer+basesrc: 0:00:08.982140165 0xd4ae4af0 gstbasesrc.c:3055:gst_base_src_loop: error: streaming stopped, reason not-negotiated (-4)
05-17 11:23:57.494 7738 8344 W GStreamer+basetransform: 0:00:08.982345530 0xd4ae4af0 gstbasetransform.c:1355:gst_base_transform_setcaps: transform could not transform audio/x-raw, format=(string)F32LE, layout=(string)interleaved, rate=(int)48000, channels=(int)1 in anything we support
05-17 11:23:57.495 7738 8344 W GStreamer+basetransform: 0:00:08.982536884 0xd4ae4af0 gstbasetransform.c:1355:gst_base_transform_setcaps: transform could not transform audio/x-raw, format=(string)F32LE, layout=(string)interleaved, rate=(int)48000, channels=(int)1 in anything we support
05-17 11:23:57.495 7738 8344 W GStreamer+basetransform: 0:00:08.982646415 0xd4ae4af0 gstbasetransform.c:1355:gst_base_transform_setcaps: transform could not transform audio/x-raw, format=(string)F32LE, layout=(string)interleaved, rate=(int)48000, channels=(int)1 in anything we support
05-17 11:23:57.495 7738 8344 W GStreamer+audiobasesink: 0:00:08.982688968 0xd4ae4af0 gstaudiobasesink.c:1119:gst_audio_base_sink_wait_event: error: Sink not negotiated before eos event.

• AudioScheduledSourceNode.start
• AudioScheduledSourceNode.stop

To support the AudioScheduledSourceNode.ended DOM event

Depends on https://gitlab.freedesktop.org/gstreamer/gstreamer-rs/merge_requests/161

BaseAudioContext.currentTime

Right now we implement only the generation of sine waves, but there are other sine wave types that we need to implement, including the periodic waveform. This is not a high priority at the moment though.

The decoding work needs to happen on a dedicated thread. We can probably use GStreamer decodebin for this.

It should have a configuration that builds with android and build the dummy backend as well.

https://webaudio.github.io/web-audio-api/#AudioBufferSourceNode

So currently we're not actually thread safe. There's an unsafe impl of Send/Sync on AudioGraphThread, which shouldn't exist.

We simultaneously need:

• mutable access from the graph thread for things like adding nodes and play/pause
• mutable access from gstreamer for running the graph.

This isn't good. We could solve it with locks, but that introduces two competing synchronization mechanisms. Instead I think we should have everything run by the graph thread (IIRC this was the original plan but we didn't really explore it), with gstreamer being isolated in a different thread requesting data.

The plan would be:

• The graph thread is the only thing that touches the graph
• when the sink needs data, it requests it from the graph thread, and waits
• the graph thread processes incoming graph change messages, as well as incoming need_data messages. Whenever the source needs data, it sends it over.

We don't actually need a separate "gstreamer thread" setup, the graph thread can own the gstreamer sink and the need_data callbacks let this work asynchronously.

I'm a bit afraid that the render quantum of ~3ms may be affected by this model, though.

The audio_decoder example occasionally panics on this assertion. This happens a lot on CI (which has "1.5 threads" IIRC) but less so locally.

I investigated it a bit (it's pretty easy to reproduce under rr's chaos mode, rr record -h target/debug/audio_decoder).

The problem is that gstreamer decodes things on multiple threads (per-channel as far as I can tell), which both call new_sample(). When it reaches the end, eos() gets called. However, the new_sample() from the other thread may not have run yet -- eos() isn't guaranteed to run after all of the new_sample() calls finish.

One possible fix is to have a second RWlock -- always read-acquired by new_sample(). eos() write-acquires it and first drains appsink.pull_sample() into the buffers before sending the actual EOS signal. This is kinda ugly, I'm hoping there's a way we can ensure eos gets called after everything else, or have a way to wait in eos for everything to finish.

cc @ferjm @sdroege

Currently, node.start(t) doesn't seem to work for scheduled source nodes. The following patch fixes it, but overall I'm not sure we're doing the right thing with the two-return-value should_play_at function.

cc @ferjm could you look into this and figure out if this is the right fix? I'm using this fix so I can move ahead with my WPT investigation, but we should probably improve this code overall

diff --git a/audio/src/buffer_source_node.rs b/audio/src/buffer_source_node.rs
index 7847581..68d6ca2 100644
--- a/audio/src/buffer_source_node.rs
+++ b/audio/src/buffer_source_node.rs
@@ -118,10 +118,14 @@ impl AudioNodeEngine for AudioBufferSourceNode {
 
         let len = { self.buffer.as_ref().unwrap().len() as usize };
 
-        if self.playback_offset >= len || self.should_play_at(info.frame) == (false, true) {
+        let should = self.should_play_at(info.frame);
+        if self.playback_offset >= len || should == (false, true) {
             self.maybe_trigger_onended_callback();
             inputs.blocks.push(Default::default());
             return inputs;
+        } else if !should.0 {
+            inputs.blocks.push(Default::default());
+            return inputs;
         }

We should be listening for pipeline bus error messages and firing the audio decoder error callback when needed.

philn> ferjm: there you go :)
12:27:54 travis needs ogg/vorbis libs
12:28:21 <ferjm> Missing decoder: Ogg (audio/ogg)
12:28:33 cool
12:30:18 we shouldn't be freezing there though :\
12:33:03 philn: I guess decodebin is throwing errors for missing decoders, right?
12:34:30 ⇐ BK1603 quit ([email protected]) Quit: Connection closed for inactivity
12:36:35 <philn> yeah
12:38:58 you should be able to reproduce the error locally by moving out libgstogg.so
12:39:16 <@Manishearth> philn: thanks :)
12:39:36 <ferjm> philn: ok, cool. Are these errors sent through as a pipeline bus message or is there an error callback I can handle in decodebin?
12:39:59 <@Manishearth> ferjm: in the example we wait for it to respond, we should .unwrap() the .recv() call
12:40:02 that's why it hangs
12:40:03 <philn> if you use gst-player you can connect to the error signal iirc
12:40:08 <@Manishearth> we get stuck on .recv()
12:40:26 <philn> otherwise yes, an error message is sent over the bus
12:41:10 <@Manishearth> philn: its a bug in the test's error handling
12:41:16 → rohitpaulk23833258 joined ([email protected])
12:41:22 <philn> ah, ok!
12:42:15 <ferjm> philn: ok, we are not using gst-player in this case
12:42:43 Manishearth:  https://github.com/servo/media/blob/master/examples/audio_decoder.rs#L50
12:43:17 <@Manishearth> ferjm: oh hm
12:43:18 <ferjm> we should be firing the error callback for the missing decoder and send a message through the channel there https://github.com/servo/media/blob/master/examples/audio_decoder.rs#L34
12:43:23 <@Manishearth> yeah
12:43:34 it's an eror handler so it doesn't panic, my bad
12:43:34 <ferjm> but we need to get the error from the pipeline bus first 

https://webaudio.github.io/web-audio-api/#dom-audiobuffersourcenode-start

Hello,

I'm getting the following error when building servo media:

error: aborting due to previous error

error: Could not compile examples.

Caused by:
process didn't exit successfully: rustc --crate-name offline_context examples/offline_context.rs --color always --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=aed0fb0205c5093d -C extra-filename=-aed0fb0205c5093d --out-dir /home/zimio/workspace/mozilla/media/target/debug/deps -C incremental=/home/zimio/workspace/mozilla/media/target/debug/incremental -L dependency=/home/zimio/workspace/mozilla/media/target/debug/deps --extern env_logger=/home/zimio/workspace/mozilla/media/target/debug/deps/libenv_logger-6ae0db42cd5a81be.rlib --extern euclid=/home/zimio/workspace/mozilla/media/target/debug/deps/libeuclid-240f7d3a788cc1ac.rlib --extern gleam=/home/zimio/workspace/mozilla/media/target/debug/deps/libgleam-a418cd17a3b482fc.rlib --extern glutin=/home/zimio/workspace/mozilla/media/target/debug/deps/libglutin-db916739f4394186.rlib --extern ipc_channel=/home/zimio/workspace/mozilla/media/target/debug/deps/libipc_channel-b969ce4051c98805.rlib --extern rand=/home/zimio/workspace/mozilla/media/target/debug/deps/librand-9f582797a01a5572.rlib --extern servo_media=/home/zimio/workspace/mozilla/media/target/debug/deps/libservo_media-b045ad578ba4e888.rlib --extern time=/home/zimio/workspace/mozilla/media/target/debug/deps/libtime-db1609a5ae642d87.rlib --extern webrender=/home/zimio/workspace/mozilla/media/target/debug/deps/libwebrender-7ceb9a0165526e89.rlib --extern winit=/home/zimio/workspace/mozilla/media/target/debug/deps/libwinit-2f6826a86cd892ac.rlib -L native=/home/zimio/workspace/mozilla/media/target/debug/build/libloading-b7b9b3538b7e1fe0/out -L native=/usr/local/lib -L native=/usr/local/lib -L native=/usr/local/lib -L native=/usr/local/lib -L native=/usr/local/lib -L native=/usr/local/lib -L native=/usr/lib/x86_64-linux-gnu (exit code: 1)

It seems there is code in examples which it cannot compile.

Here is the output of rustup show:

(base) ➜ media git:(master) ✗ rustup show
Default host: x86_64-unknown-linux-gnu

installed toolchains

stable-x86_64-unknown-linux-gnu
nightly-2018-10-05-x86_64-unknown-linux-gnu (default)

active toolchain

nightly-2018-10-05-x86_64-unknown-linux-gnu (default)
rustc 1.31.0-nightly (8c4ad4e9e 2018-10-04)

So I'm using nightly correctly, yet it seems that I can't build this.

Currently we just have a vector of nodes. We need the ability to have a full graph, using the cycle-breaking algorithm from the spec.

Probably should use petgraph.

I would like to hear your opinion to merge rust-playground into this crate.

The general idea is to simplify the merge to servo, provide a similar API for audio and video, and share common resources (I'm thinking, for example, the player objects or playbin).

We only need to download Android dependencies once, so if we already have them locally, we should skip the fetch step

With #17 we have a chunk and block abstraction, but the block abstraction only supports a single channel.

We should expand this to allow for more channels (stored in the same buffer, probably), as well as upmixing/downmixing.

https://webaudio.github.io/web-audio-api/#audiobuffer

We made several changes to the API and on the repo layout with the multi-crate split up that broke the example for Android.

This test case https://ferjm.github.io/web-api-tests/webaudio/decode.html takes ~465ms in Firefox vs ~40sec in Servo to start playing sound.

https://webaudio.github.io/web-audio-api/#playback-AudioBufferSourceNode

most parameters can be set to be tweaked dynamically.

They can both be such that they get updated each block, and that they get updated each frame (k-type vs a-type)

We need some abstractions for both kinds of params, and for each kind of way they can vary.

We'll also need some way of identifying each parameter whilst messaging the thread. (Kind of needs #19 to exist first, in some rudimentary form)

https://github.com/servo/media/blob/master/backends/gstreamer/build.rs#L48 ("build script") returns a 404 as build.rs no longer exists.

This is required to support parts of the Media Elements spec.

Spec: https://www.w3.org/TR/media-frags/

Gecko implementation:

• https://dxr.mozilla.org/mozilla-central/source/netwerk/base/nsMediaFragmentURIParser.h
• https://dxr.mozilla.org/mozilla-central/source/netwerk/base/nsMediaFragmentURIParser.cpp

Blink implementation:

• https://chromium.googlesource.com/chromium/blink/+/master/Source/core/html/MediaFragmentURIParser.h
• https://chromium.googlesource.com/chromium/blink/+/master/Source/core/html/MediaFragmentURIParser.cpp

We disabled GStreamer for non-x86 in #96 so we could land the WebAudio API in Servo. We should turn it on again as soon as possible.

https://github.com/servo/servo/blob/master/tests/wpt/web-platform-tests/webaudio/the-audio-api/the-audiobuffersourcenode-interface/audiobuffersource-multi-channels.html fails because of this, we assume that all decoded audio is single-channel, but it doesn't have to be.

We should pass the channel count back through one of the callbacks (perhaps progress or eos, or maybe both)

(It also fails because of #107)

There is a link to https://github.com/ferjm/media/blob/master/build.rs#L19 in the Readme which no longer works ("build script"). Can this be replaced with an updated link or removed?

Currently the PlayerError enum in gstreamer-rs is just a stub. We could extend it with the errors that we care about based on our uses cases.

This issue will have one part to update gstreamer-rs and another part to make servo-media use the new set of errors.

https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/merge_requests/167 is a recent merge requrest for gstreamer. It looks like something we might care about, and will require changes to gstreamer-rs before we can make use of them.

https://webaudio.github.io/web-audio-api/#dom-audionode-disconnect

We can currently create connections between AudioNodes but we cannot undo them.

This code creates a video or audio stream with default constraints. The constraint API should be lifted out of the gstreamer backend into the high level library, so that it's possible for consumers of this crate to request an input stream that matches the given constraints. This will require adding additional arguments to the create_videoinput_stream and create_audioinput_stream functions and passing them down the stack until the create_input_stream method in the gstreamer backend.

See #122 (comment)

They are currently disabled.

From this comment:

For AudioParams the plan was to allow for params to "create" new input ports. We can have an additional boolean flag on PortId, and when flipped it enumerates params instead.