Membrane plugin for video and audio mixing/composing

Home Page: https://membraneframework.org/

License: Apache License 2.0

Elixir 100.00%

elixir membrane membrane-framework video video-processing

membrane_video_compositor_plugin's Introduction

Membrane Video Compositor Plugin

Membrane SDK for VideoCompositor, that takes multiple input streams, transforms them according to provided transformations and composes them into output streams / videos.

It is part of Membrane Multimedia Framework.

Installation

The package can be installed by adding membrane_video_compositor_plugin to your list of dependencies in mix.exs:

def deps do
  [
    {:membrane_video_compositor_plugin, "~> 0.8.0"}
  ]
end

VideoCompositor requires having locally installed:

FFmpeg 6.0 - for streaming inputs / outputs to VideoCompositor
wget - for downloading VideoCompositor binary file
tar - for decompressing VideoCompositor binary file

Examples

Examples can be found in examples directory.

To run example run:

cd examples
mix deps.get
mix run lib/[example_name].exs

Layout with shader example

The example presents dynamically added video arranged onto a tiled layout and "twisted" with the simple shader. Shaders can be used to create custom visual effects.

Transition example

The example presents dynamic transition of input videos. Transitions are used for smooth, dynamical animations.

Dynamic outputs example

The example presents dynamic outputs linking. Multiple outputs are useful for live-streaming for multiple platforms (e.g. different layout for mobile devices), target resolutions or any other case, when user want to process input videos differently.

membrane_video_compositor_plugin's People

Contributors

Stargazers

Watchers

membrane_video_compositor_plugin's Issues

Wait time for compositor to start is to short

When setting it up on AWS EC2 instance initial startup takes some time. I was able to make it work by starting the pipeline, kill the process manually and start again, because startup on the second run is much quicker.

10:43:06.323 [error] GenServer #PID<0.1760.0> terminating
** (MatchError) no match of right hand side value: :error
    (membrane_video_compositor_plugin 0.8.0) lib/membrane/video_compositor/video_compositor.ex:320: Membrane.VideoCompositor.handle_setup/2
    (membrane_core 1.0.0) lib/membrane/core/callback_handler.ex:139: Membrane.Core.CallbackHandler.exec_callback/4
    (membrane_core 1.0.0) lib/membrane/core/callback_handler.ex:69: Membrane.Core.CallbackHandler.exec_and_handle_callback/5
    (membrane_core 1.0.0) lib/membrane/core/parent/lifecycle_controller.ex:27: Membrane.Core.Parent.LifecycleController.handle_setup/1
    (membrane_core 1.0.0) lib/membrane/core/bin.ex:149: Membrane.Core.Bin.handle_continue/2
    (stdlib 5.2) gen_server.erl:1085: :gen_server.try_handle_continue/3
    (stdlib 5.2) gen_server.erl:995: :gen_server.loop/7
    (stdlib 5.2) proc_lib.erl:241: :proc_lib.init_p_do_apply/3
Last message: {:continue, :setup}
State: %Membrane.Core.Bin.State{module: Membrane.VideoCompositor, synchronization: %{timers: %{}, clock: nil, parent_clock: #PID<0.1723.0>, clock_provider: %{choice: :auto, clock: nil, provider: nil}, clock_proxy: #PID<0.1761.0>, latency: 0, stream_sync: :membrane_no_sync}, subprocess_supervisor: #PID<0.1759.0>, resource_guard: #PID<0.1762.0>, stalker: %Membrane.Core.Stalker{pid: #PID<0.1721.0>, ets: #Reference<0.1425352200.2932211713.165109>}, internal_state: %Membrane.VideoCompositor{framerate: 30, port_range: {6000, 10000}, init_web_renderer?: false, stream_fallback_timeout: 10000000000, start_composing_strategy: :on_init, vc_server_config: {:start_on_port, 8081}}, children: %{}, name: :video_compositor, pads_info: %{input: %{name: :input, options: [input_id: [spec: "input_id()"]], availability: :on_request, accepted_formats_str: ["%Membrane.H264{alignment: :nalu, stream_structure: :annexb}"], direction: :input}, output: %{name: :output, options: [output_id: [spec: "output_id()"]], availability: :on_request, accepted_formats_str: ["%Membrane.H264{alignment: :nalu, stream_structure: :annexb}"], direction: :output}}, pads_data: %{}, parent_pid: #PID<0.1720.0>, crash_groups: %{}, children_log_metadata: [], links: %{}, pending_specs: %{}, playback: :stopped, initialized?: false, terminating?: false, setup_incomplete?: false, handling_action?: false}

10:43:06.324 [debug] <0.1720.0>/ Child :video_compositor crashed but was not a member of any crash group.
Terminating.


10:43:06.324 [debug] <0.1720.0>/:video_compositor/ subprocess supervisor got exit request from parent, reason: :shutdown, shutting down children

10:43:06.325 [error] <0.1720.0>/ Terminating with reason: {:membrane_child_crash, :video_compositor, {{:badmatch, :error}, [{Membrane.VideoCompositor, :handle_setup, 2, [file: ~c"lib/membrane/video_compositor/video_compositor.ex", line: 320]}, {Membrane.Core.CallbackHandler, :exec_callback, 4, [file: ~c"lib/membrane/core/callback_handler.ex", line: 139]}, {Membrane.Core.CallbackHandler, :exec_and_handle_callback, 5, [file: ~c"lib/membrane/core/callback_handler.ex", line: 69]}, {Membrane.Core.Parent.LifecycleController, :handle_setup, 1, [file: ~c"lib/membrane/core/parent/lifecycle_controller.ex", line: 27]}, {Membrane.Core.Bin, :handle_continue, 2, [file: ~c"lib/membrane/core/bin.ex", line: 149]}, {:gen_server, :try_handle_continue, 3, [file: ~c"gen_server.erl", line: 1085]}, {:gen_server, :loop, 7, [file: ~c"gen_server.erl", line: 995]}, {:proc_lib, :init_p_do_apply, 3, [file: ~c"proc_lib.erl", line: 241]}]}}

[VideoCompositor SDK] Add custom VC requests

`merge_frames` benchmark is not very accurate

While testing wgpu's performance, I discovered that the function timed by the merge_frames benchmark seems to be too short to measure accurately. I think it should be run multiple times per time measurement.

Handling pad removing

Right now there is no handle_pad_removed callback in compositor_element.ex. This can cause problems when we get handle_pad_added without handle_start_of_stream in this scenario we won't always get "end_of stream" message.
That's why we have to implement 'handle_pad_removed` to be sure that we handle all cases of removing pads.

Also there is another problem with a stream that was collected but not mixed. If compositor_element get "end_of_stream" from certain pad it won't try to mix frames that were blocked by this pad so if we don't get any new buffers we will never mix those saved frames. My suggestion to fix that is to invoke some function that will try to mix as much frames as it can in two callbacks: handle_pad_removed and handle_end_of_stream

Fix CI for Rust

Add Rust to the docker image
Add Rust static tools to the ci config

[VideoCompositor SDK] Add livebook examples

Fix ports for input and output streams

Automatic VC binary download

Registering output is picking free port incorrectly

https://github.com/membraneframework/membrane_video_compositor_plugin/blob/master/lib/membrane/video_compositor/streams_handler.ex

Not sure why this even works, but the output port can't be picked based on the response from the compositor. Only the listening side knows if the port is free or not.

Also slightly related, this logic is consuming all their response errors and returns just :error atom. If there is a code where we try range of values and all of them fail we should at least return last of first error.

Jan 23 14:46:10 ip-10-0-1-215 bash[1332]: 14:46:10.135 [error] GenServer #PID<0.1919.0> terminating
Jan 23 14:46:10 ip-10-0-1-215 bash[1332]: ** (MatchError) no match of right hand side value: {:error_response_code, %Req.Response{status: 400, headers: %{"content-type" => ["application/json"], "date" => ["Tue, 23 Jan 2024 14:46:10 GMT"], "server" => ["tiny-http (Rust)"]}, body: %{"error_code" => "COMPOSITOR_ALREADY_INITIALIZED", "msg" => "Compositor was already initialized.", "stack" => ["Compositor was already initialized."]}, trailers: %{}, private: %{}}}
Jan 23 14:46:10 ip-10-0-1-215 bash[1332]:     (membrane_video_compositor_plugin 0.8.0) lib/membrane/video_compositor/video_compositor.ex:322: Membrane.VideoCompositor.handle_setup/2
Jan 23 14:46:10 ip-10-0-1-215 bash[1332]:     (membrane_core 1.0.0) lib/membrane/core/callback_handler.ex:139: Membrane.Core.CallbackHandler.exec_callback/4
Jan 23 14:46:10 ip-10-0-1-215 bash[1332]:     (membrane_core 1.0.0) lib/membrane/core/callback_handler.ex:69: Membrane.Core.CallbackHandler.exec_and_handle_callback/5
Jan 23 14:46:10 ip-10-0-1-215 bash[1332]:     (membrane_core 1.0.0) lib/membrane/core/parent/lifecycle_controller.ex:27: Membrane.Core.Parent.LifecycleController.handle_setup/1
Jan 23 14:46:10 ip-10-0-1-215 bash[1332]:     (membrane_core 1.0.0) lib/membrane/core/bin.ex:149: Membrane.Core.Bin.handle_continue/2
Jan 23 14:46:10 ip-10-0-1-215 bash[1332]:     (stdlib 5.2) gen_server.erl:1085: :gen_server.try_handle_continue/3
Jan 23 14:46:10 ip-10-0-1-215 bash[1332]:     (stdlib 5.2) gen_server.erl:995: :gen_server.loop/7
Jan 23 14:46:10 ip-10-0-1-215 bash[1332]:     (stdlib 5.2) proc_lib.erl:241: :proc_lib.init_p_do_apply/3
Jan 23 14:46:10 ip-10-0-1-215 bash[1332]: Last message: {:continue, :setup}

adding / removing inputs / outputs
pads change handler

membraneframework / membrane_video_compositor_plugin Goto Github PK