Hello, Great project! I would like to use this library for a specifi

Synchronized Video Streaming with Raspberry PI Zero 2 W,about tzuhuantai/raspberrypi_webrtc

Comments (48)

eliabruni commented on May 19, 2024 1

Ah, that's great! I'll try out the timestamp code then :) As for time sync, I can use ntp sync, shouldn't be a problem.

As for production warning, message received. I already like how well the code is structured, looking forward to seeing the further progress! If you need some help in testing, let me know, especially with pi zero 2!

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024 1

Alright, will do the same then and let you know!

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024 1

excellent, i will do the same then (suing RPI3 and transfer); as for the QUEMU aproach, let me open another issue for you with my current configuration on ubuntu 22.04 (but should work on 20.04 as well); maybe we can work on the env together

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024 1

Glad to hear. When attempting to compile to project on rock5, I indeed get issues such as

RaspberryPi_WebRTC/src/common/recorder.cpp:80:14: error: cannot initialize a variable of type 'AVCodec *' with an rvalue of type 'const AVCodec *'
    AVCodec *codec = avcodec_find_encoder_by_name(encoder_name.c_str());
             ^       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

which I could fix by replacing

AVCodec *codec = avcodec_find_encoder_by_name(encoder_name.c_str());

with

const AVCodec *codec = avcodec_find_encoder_by_name(encoder_name.c_str());

I am working on the alternative signaling and am also writing the receiver side (i.e., I am writing a conductor.h for the receiver as well as its server; for the server, I am trying mosquitto).

from raspberrypi_webrtc.

TzuHuanTai commented on May 19, 2024 1

In this project, the ffmpeg is only used for recording the video file that I'm doing in issue #91 so far.

from raspberrypi_webrtc.

TzuHuanTai commented on May 19, 2024

I record timestamps while receiving the frame from the camera,

RaspberryPi_WebRTC/src/track/v4l2_track_source.cpp

Lines 76 to 80 in df98d7b

 OnFrame(webrtc::VideoFrame::Builder() 

 .set_video_frame_buffer(dst_buffer) 

 .set_rotation(webrtc::kVideoRotation_0) 

 .set_timestamp_us(translated_timestamp_us) 

 .build());

This timestamp is appended to the encoded frame before sending it to the webrtc client side.

RaspberryPi_WebRTC/src/v4l2_codecs/raw_buffer_encoder.cpp

Line 82 in df98d7b

encoded_image_.SetTimestamp(frame.timestamp());

So It may work for your case😕. But you must ensure the clocks from different devices are synchronized, or there is always a fixed time difference between each device.

There are still a lot of codes that need to be cleaned or refactored in the future. I don't recommend using this project in production so far. I'm refactoring the DMA mechanism to enhance the performance in those few days, I plan to migrate this project to my pi zero 2 as well😄.

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024

Btw, for testing on pi zero 2, would you recommend installing RaspberryPi OS 64bit?

from raspberrypi_webrtc.

TzuHuanTai commented on May 19, 2024

Yes, I'm using Raspberry Pi OS 64bit. I haven't tested on others yet.😅

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024

Alright, will try that (Bullseye 64bit)!

One more question: do build webrtc directly on RPI or do you cross-compile? I am asking because it might take a long time on pizero and am not even sure if it can do that. I am trying to figure out a good cross-compilation recipe..

from raspberrypi_webrtc.

TzuHuanTai commented on May 19, 2024

I built on my WSL with Ubuntu 20.04, and it only took 10 minutes.
The compiling arguments are here.

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024

Thanks. I have trying on bare ubuntun 22.04, but could also give WSL on 20.04 a try. Besides the compiling arguments you pointed at, you must also have a cross compile configuration/procedure right? I have been trying to cook one, but I am getting clang errors (despite using your recommended one). I am getting weird CLANG errors, e.g., there are several compiler flags that the current version of clang does not recognize (that's both with your optional procedures or the default vresion)

-Wno-thread-safety-reference-return
-Wno-delayed-template-parsing-in-cxx20
-split-threshold-for-reg-with-hint=0

from raspberrypi_webrtc.

TzuHuanTai commented on May 19, 2024

Try to download the newer clang version here.
It usually happens because the webrtc team uses the relatively new clang.
It would show unrecognized errors If your clang is too old.

from raspberrypi_webrtc.

TzuHuanTai commented on May 19, 2024

I don't know whether you check out the specific branch of webrtc or the newest one. I'm using m115 now.

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024

That worked, thank you! I am cross-compiling on ubuntu 22.04. Later on, if you want I can share the whole recipe from creating the cross-compilation env onward, maybe it could be useful info for this repo.

Now I am moving to microsoft-signalr.so. Did you also cross-compile that on WSL or did you instead directly compiled and install it on RPI?

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024

In the meantime I made progress with cross-compilation, however I now get stuck at Install [cpprestsdk](https://github.com/Microsoft/cpprestsdk/wiki/How-to-build-for-Linux).

all works (and I followed carefully your instructions about c++11 --> c++14) until when i try to run

ninja

at that point i have a ton of compilation errors; i create a gist with the output

from raspberrypi_webrtc.

TzuHuanTai commented on May 19, 2024

I built all other libs on Raspberry Pi. webrtc.lib can be compiled on WSL because it provides a cross-compile arg target_cpu="arm64". We need to make sure all libs are compatible with arm based platforms.

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024

Compiling on pizero seems to be quite hard though. I am now at casablanca but in 8 hours it is at this step:

[5/160] Building CXX object Release/src/CMakeFiles/cpprest.dir/json/json.cpp.o

while using 2GB+ swap.

I wonder if we could compile on more powerful RPI and then transfer the library - would that work you think?

from raspberrypi_webrtc.

TzuHuanTai commented on May 19, 2024

I'm researching how to emulate Raspberry Pi OS on Windows to speed up.
I plan to use QEMU to run an arm-based device with it. But I got stuck on the below command.
I'm trying to figure it out. :(

qemu-system-aarch64 -machine raspi3b -cpu cortex-a72 -dtb "D:\bcm2710-rpi-3-b-plus.dtb" -kernel "D:\kernel8.img" -drive file="D:\2023-10-10-raspios-bookworm-arm64-full.img",if=none,id=drive0 -append "rw earlyprintk loglevel=8 console=ttyAMA0,115200 dwc_otg.lpm_enable=0 root=/dev/mmcblk0p2 rootdelay=1" -serial stdio -usb -device usb-mouse -device usb-kbd -device usb-net,netdev=net0 -netdev user,id=net0,hostfwd=tcp::5555-:22

from raspberrypi_webrtc.

TzuHuanTai commented on May 19, 2024

btw, my libs were built on 3B, the pi zero is too slow.
I did transfer compiled libs to pi zero.
I already tested the project on my pi zero two days ago, and it worked fine.

from raspberrypi_webrtc.

TzuHuanTai commented on May 19, 2024

I recompiled all dependent libs and released a static executable file here. You could take a try.

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024

That's very kind, thank you (I deleted the message because right after I asked that the RPI4 arrived :)

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024

I just tested your binaries and everything is working, thank you!

Back to the original question of this issue, for our purpose, we won't need a browser on the receiving end. We will instead have another arm64 linux machine, which will match images based on their timestamps and then send that to a neural network. So, I am now wondering if we should replace signalr with a different (simpler) communication protocol. In particular, because it probably doesn't make much sense to use signalr server hub, right? Despite that, signalr seems like a very nice library and I wonder if I could easily integrate in my usecase, with no browser.

from raspberrypi_webrtc.

TzuHuanTai commented on May 19, 2024

It's a good question! I agree with your suggestion that the signaling mechanism could be replaced or selected by other protocols. I'll try to decouple and refactor it with an interface in a few days. For the time to implement the regular web socket or other protocols, I have no plan for when I'll finish it. Maybe you can implement your protocol on the interface, and recompile the program by setting customized options like -DUSE_SIGNALR=true, -DUSE_WEBSOCKET=true or so.
I initially chose signalr as a signaling mechanism because I have a project that broadcasts sensor data through signalr. Built on top of it is the most convenient way for me, that's all.

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024

Having an interface would be great! Alright, I'll look into the best suitable signaling protocol for my use case while waiting for the interface then. I'll be happy to act as tester afterwards, maybe that could also be helpful for others if we document it well.

from raspberrypi_webrtc.

TzuHuanTai commented on May 19, 2024

I refactored the signaling service code in #86.
For example, If you'd like to use websocket, the new command to set the signaling service would be "cmake .. -DCMAKE_CXX_COMPILER=/usr/bin/clang++ -DUSE_SIGNALING=WEBSOCKET", the default is SIGNALR. You can implement any protocols by inheriting the class in src/signaling/signaling_service.h, and creating the service in src/main.cpp like below.

RaspberryPi_WebRTC/src/main.cpp

Lines 26 to 34 in 01465be

 auto signal = ([&]() -> std::unique_ptr<SignalingService> { 

 #if USE_WEBSOCKET_SIGNALING 

 return nullptr; 

 #elif USE_SIGNALR_SIGNALING 

 return SignalrService::Create(args.signaling_url, conductor); 

 #else 

 return nullptr; 

 #endif 

 })();

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024

That's great! I will give it a try and ask for feedback as I am not an expert :)

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024

Quick question: which version of ffmpeg are you using on the RPI and was tested with?

from raspberrypi_webrtc.

TzuHuanTai commented on May 19, 2024

I use 4.3.6 version of ffmpeg, the shared libs details show with the command apt list --installed | grep libav

libavc1394-0/oldstable,now 0.5.4-5 arm64 [installed,automatic]
libavcodec-dev/oldstable,now 8:4.3.6-0+deb11u1+rpt5 arm64 [installed]
libavcodec58/oldstable,now 8:4.3.6-0+deb11u1+rpt5 arm64 [installed,automatic]
libavdevice-dev/oldstable,now 8:4.3.6-0+deb11u1+rpt5 arm64 [installed]
libavdevice58/oldstable,now 8:4.3.6-0+deb11u1+rpt5 arm64 [installed,automatic]
libavfilter-dev/oldstable,now 8:4.3.6-0+deb11u1+rpt5 arm64 [installed,automatic]
libavfilter7/oldstable,now 8:4.3.6-0+deb11u1+rpt5 arm64 [installed,automatic]
libavformat-dev/oldstable,now 8:4.3.6-0+deb11u1+rpt5 arm64 [installed]
libavformat58/oldstable,now 8:4.3.6-0+deb11u1+rpt5 arm64 [installed,automatic]
libavresample4/oldstable,now 8:4.3.6-0+deb11u1+rpt5 arm64 [installed,automatic]
libavutil-dev/oldstable,now 8:4.3.6-0+deb11u1+rpt5 arm64 [installed]
libavutil56/oldstable,now 8:4.3.6-0+deb11u1+rpt5 arm64 [installed,automatic]

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024

Thank you!

The receiver I am working on is a rock 5A,

Here my output of apt list --installed | grep libav

libavahi-client3/jammy-security,jammy-updates,now 0.8-5ubuntu5.2 arm64 [installed,automatic]
libavahi-common-data/jammy-security,jammy-updates,now 0.8-5ubuntu5.2 arm64 [installed,automatic]
libavahi-common3/jammy-security,jammy-updates,now 0.8-5ubuntu5.2 arm64 [installed,automatic]
libavahi-core7/jammy-security,jammy-updates,now 0.8-5ubuntu5.2 arm64 [installed,automatic]
libavc1394-0/jammy,now 0.5.4-5build2 arm64 [installed,automatic]
libavcodec-dev/jammy,now 7:6.0-5+git230804.e243e8d001~j1 arm64 [installed]
libavcodec58/jammy,now 7:4.4.2-0ubuntu0.22.04.1+rkmpp20230327 arm64 [installed,automatic]
libavcodec60/jammy,now 7:6.0-5+git230804.e243e8d001~j1 arm64 [installed,automatic]
libavdevice-dev/jammy,now 7:6.0-5+git230804.e243e8d001~j1 arm64 [installed]
libavdevice60/jammy,now 7:6.0-5+git230804.e243e8d001~j1 arm64 [installed,automatic]
libavfilter-dev/jammy,now 7:6.0-5+git230804.e243e8d001~j1 arm64 [installed]
libavfilter9/jammy,now 7:6.0-5+git230804.e243e8d001~j1 arm64 [installed,automatic]
libavformat-dev/jammy,now 7:6.0-5+git230804.e243e8d001~j1 arm64 [installed]
libavformat60/jammy,now 7:6.0-5+git230804.e243e8d001~j1 arm64 [installed,automatic]
libavutil-dev/jammy,now 7:6.0-5+git230804.e243e8d001~j1 arm64 [installed]
libavutil56/jammy,now 7:4.4.2-0ubuntu0.22.04.1+rkmpp20230327 arm64 [installed,automatic]
libavutil58/jammy,now 7:6.0-5+git230804.e243e8d001~j1 arm64 [installed,automatic]

I also need to build a specific ffmpeg version.

Do you think this could cause issues? If I understand correctly, the thing the receiver should do is properly decode and scale the video stream. As far as I can see, these are the dynamic libs I should pay attention to:

set(WEBRTC_LINK_LIBS X11 dl)
set(FFMPEG_LINK_LIBS avformat avcodec avutil swscale avdevice)

from raspberrypi_webrtc.

TzuHuanTai commented on May 19, 2024

It'll be ok. The receiver only needs to decode the video stream.
The different versions of ffmpeg may change the way to call their API, but codec standards are usually the same.

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024

Hello @TzuHuanTai !

It took me some time but I managed to arrange a temporary version RaspberryPi_WebRTC that attempts to use Mosquitto for signaling.

I put the instructions how to run it in the README.md. I apologize for the not yet well integrated code. There is lot to be improved in terms of OOP, but I wanted to first make sure to keep the new files isolated so that they are easier to debug. Once all fixed, I'll be happy to make a PR if you are interested.

So far, I managed to successfully exchange SDP and ICE offers (at least, for what I can tell). In particular:

About SDP

Sender SDP:

[Conductor::CreateAnswer] Successfully set local description. SDP:
v=0
o=- 6939216143307478089 3 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=extmap-allow-mixed
a=msid-semantic: WMS test_stream_id
m=audio 9 UDP/TLS/RTP/SAVPF 111 63 9 102 0 8 13 110 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:S1HD
a=ice-pwd:i94ulUDBazgxV3lArAll5L8Y
a=ice-options:trickle
a=fingerprint:sha-256 F9:1A:78:45:5C:69:43:5D:DC:52:D6:70:A0:90:C8:61:53:A3:86:63:C3:64:3C:74:E7:03:01:50:16:9B:CE:7A
a=setup:passive
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendonly
a=msid:test_stream_id raspberrypi_audio
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:63 red/48000/2
a=fmtp:63 111/111
a=rtpmap:9 G722/8000
a=rtpmap:102 ILBC/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:126 telephone-event/8000
a=ssrc:1334265160 cname:yrCNoLNkYVwUaXi2
a=ssrc:1334265160 msid:test_stream_id raspberrypi_audio
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 39 40 100 101 127
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:S1HD
a=ice-pwd:i94ulUDBazgxV3lArAll5L8Y
a=ice-options:trickle
a=fingerprint:sha-256 F9:1A:78:45:5C:69:43:5D:DC:52:D6:70:A0:90:C8:61:53:A3:86:63:C3:64:3C:74:E7:03:01:50:16:9B:CE:7A
a=setup:passive
a=mid:1
a=extmap:14 urn:ietf:params:rtp-hdrext:toffset
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:13 urn:3gpp:video-orientation
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=sendonly
a=msid:test_stream_id raspberrypi_video
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 profile-id=0
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:39 AV1/90000
a=rtcp-fb:39 goog-remb
a=rtcp-fb:39 transport-cc
a=rtcp-fb:39 ccm fir
a=rtcp-fb:39 nack
a=rtcp-fb:39 nack pli
a=rtpmap:40 rtx/90000
a=fmtp:40 apt=39
a=rtpmap:100 red/90000
a=rtpmap:101 rtx/90000
a=fmtp:101 apt=100
a=rtpmap:127 ulpfec/90000
a=ssrc-group:FID 2712533784 115193243
a=ssrc:2712533784 cname:yrCNoLNkYVwUaXi2
a=ssrc:2712533784 msid:test_stream_id raspberrypi_video
a=ssrc:115193243 cname:yrCNoLNkYVwUaXi2
a=ssrc:115193243 msid:test_stream_id raspberrypi_video
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:S1HD
a=ice-pwd:i94ulUDBazgxV3lArAll5L8Y
a=ice-options:trickle
a=fingerprint:sha-256 F9:1A:78:45:5C:69:43:5D:DC:52:D6:70:A0:90:C8:61:53:A3:86:63:C3:64:3C:74:E7:03:01:50:16:9B:CE:7A
a=setup:passive
a=mid:2
a=sctp-port:5000
a=max-message-size:262144

Audio and Video Tracks: Present and configured as sendonly, which means the sender is transmitting audio and video but not receiving.
Data Channel: Configured for data transmission.
ICE Credentials: Present and unique.
DTLS Setup: Passive. This means the sender expects the receiver to initiate the DTLS handshake.
Codecs and Extensions: Standard codecs and extensions are listed.

Receiver SDP:

[Mosquitto] Invoke AnswerSDP: v=0
o=- 3745984242120899573 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=extmap-allow-mixed
a=msid-semantic: WMS
m=audio 9 UDP/TLS/RTP/SAVPF 111 63 9 102 0 8 13 110 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:zcoW
a=ice-pwd:Z+zmz7XcAgZON7/E+x8SwPY+
a=ice-options:trickle
a=fingerprint:sha-256 19:67:44:D9:D4:AF:12:17:E1:F6:FD:DF:0E:B2:36:BE:56:A1:BA:8F:BE:9B:2E:96:F5:EA:B0:43:F7:F0:E6:E5
a=setup:active
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=recvonly
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:63 red/48000/2
a=fmtp:63 111/111
a=rtpmap:9 G722/8000
a=rtpmap:102 ILBC/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:126 telephone-event/8000
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 39 40 100 101 127
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:zcoW
a=ice-pwd:Z+zmz7XcAgZON7/E+x8SwPY+
a=ice-options:trickle
a=fingerprint:sha-256 19:67:44:D9:D4:AF:12:17:E1:F6:FD:DF:0E:B2:36:BE:56:A1:BA:8F:BE:9B:2E:96:F5:EA:B0:43:F7:F0:E6:E5
a=setup:active
a=mid:1
a=extmap:14 urn:ietf:params:rtp-hdrext:toffset
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:13 urn:3gpp:video-orientation
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=recvonly
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 profile-id=0
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:39 AV1/90000
a=rtcp-fb:39 goog-remb
a=rtcp-fb:39 transport-cc
a=rtcp-fb:39 ccm fir
a=rtcp-fb:39 nack
a=rtcp-fb:39 nack pli
a=rtpmap:40 rtx/90000
a=fmtp:40 apt=39
a=rtpmap:100 red/90000
a=rtpmap:101 rtx/90000
a=fmtp:101 apt=100
a=rtpmap:127 ulpfec/90000
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:zcoW
a=ice-pwd:Z+zmz7XcAgZON7/E+x8SwPY+
a=ice-options:trickle
a=fingerprint:sha-256 19:67:44:D9:D4:AF:12:17:E1:F6:FD:DF:0E:B2:36:BE:56:A1:BA:8F:BE:9B:2E:96:F5:EA:B0:43:F7:F0:E6:E5
a=setup:active
a=mid:2
a=sctp-port:5000
a=max-message-size:262144

Audio and Video Tracks: Present and configured as recvonly, which means the receiver is only expecting to receive audio and video but not send.
Data Channel: Configured and matches the sender's configuration.
ICE Credentials: Present and unique.
DTLS Setup: Active. This complements the sender’s passive setup, indicating the receiver will initiate the DTLS handshake.
Codecs and Extensions: Standard codecs and extensions are listed.

Which means that:

Role Compatibility: The sender is in a passive role for DTLS and sending media (audio, video, and data), while the receiver is in an active role and set to receive media. This is should be correct for a sender-receiver setup.
Codec Compatibility: The codecs and extensions listed in both SDPs seem standard and compatible.
ICE and DTLS: Both ICE and DTLS configurations appear to me to be correctly set up, with unique credentials and complementary roles.

About ICE

sender ICE-related logs:

=> AddIceCandidate: end[Conductor] Inside OnIceGatheringChange 

=> OnIceGatheringChange: gathering
[Conductor] ICE is now gathering
=> OnConnectionChange: connecting
[Conductor] OnConnectionChange - IsConnected: [Mosquitto] OfferICE: Received!
0, StreamingState: 1
[Conductor] Inside OnIceCandidate 
[SignalingService] Invoke AnswerLocalIce 
[Mosquitto] Inside AnswerLocalIce: 
[Mosquitto] Invoke AnswerICE: 0, 0, candidate:3005242381 1 udp 2122260223 192.168.2.104 38430 typ host generation 0 ufrag sSFE network-id 3 network-cost 10
[Mosquitto] OfferICE: 0, 0, candidate:1492520864 1 udp 1686052607 37.153.204.14 53849 typ srflx raddr 10.80.224.56 rport 34972 generation 0 ufrag UzVd network-id 1
=> AddIceCandidate: start creating
MosquittoService::PublishMessage - Message published on topic: SenderAnswerICE
=> main: wait for closing!
Async function started. Initial States - IsReadyForStreaming: 1, IsConnected: 0
=> AddIceCandidate: add candidated
=> AddIceCandidate: end
[Conductor] Inside OnIceGatheringChange 
=> OnIceGatheringChange: complete
[Conductor] ICE gathering is complete
[Mosquitto] OfferICE: Received!
[Mosquitto] OfferICE: 0, 0, candidate:4248489337 1 tcp 1518280447 10.80.224.56 43403 typ host tcptype passive generation 0 ufrag UzVd network-id 1
=> AddIceCandidate: start creating
=> AddIceCandidate: add candidated
=> AddIceCandidate: end
[Mosquitto] OfferICE: Received!
[Mosquitto] OfferICE: 0, 0, candidate:3695579394 1 tcp 1518214911 192.168.2.105 53689 typ host tcptype passive generation 0 ufrag UzVd network-id 4 network-cost 10
=> AddIceCandidate: start creating
=> AddIceCandidate: add candidated
=> AddIceCandidate: end
close (24) dmafd
close (25) dmafd
fd(22) is closed!
Open file /dev/video12 fd(22) success!
driver 'bcm2835-codec' on card 'bcm2835-codec-isp' in mplane mode
V4l2m2m 22 formats: BA81(32x32) -> YU12(1280x720)
driver 'bcm2835-codec' on card 'bcm2835-codec-isp' in mplane mode
V4l2m2m 22 formats: YUYV(32x32) -> YU12(960x540)
(22) 9 export dma fd: (24)
(22) 9 export dma fd: (25)
[V4l2Scaler]: prepare done
unmapped (23) buffers
unmapped (23) buffers
close (26) dmafd
close (27) dmafd
fd(23) is closed!
Open file /dev/video10 fd(23) success!
driver 'bcm2835-codec' on card 'bcm2835-codec-decode' in mplane mode
V4l2m2m 23 formats: MPG4(32x32) -> MJPG(1280x720)
V4l2m2m querying (23) 10 buffer: 0x7f89d9a000 with 524288 length
V4l2m2m querying (23) 10 buffer: 0x7f89d1a000 with 524288 length
driver 'bcm2835-codec' on card 'bcm2835-codec-decode' in mplane mode
V4l2m2m 23 formats: YU12(1280x720) -> YU12(1280x720)
(23) 9 export dma fd: (26)
(23) 9 export dma fd: (27)
[V4l2Decoder]: prepare done
close (24) dmafd
close (25) dmafd
fd(22) is closed!
Open file /dev/video12 fd(22) success!
driver 'bcm2835-codec' on card 'bcm2835-codec-isp' in mplane mode
V4l2m2m 22 formats: BA81(32x32) -> YU12(1280x720)
driver 'bcm2835-codec' on card 'bcm2835-codec-isp' in mplane mode
V4l2m2m 22 formats: YUYV(32x32) -> YU12(640x360)
(22) 9 export dma fd: (24)
(22) 9 export dma fd: (25)
[V4l2Scaler]: prepare done
unmapped (23) buffers
unmapped (23) buffers
close (26) dmafd
close (27) dmafd
fd(23) is closed!
Open file /dev/video10 fd(23) success!
driver 'bcm2835-codec' on card 'bcm2835-codec-decode' in mplane mode
V4l2m2m 23 formats: MPG4(32x32) -> MJPG(1280x720)
V4l2m2m querying (23) 10 buffer: 0x7f89d9a000 with 524288 length
V4l2m2m querying (23) 10 buffer: 0x7f89d1a000 with 524288 length
driver 'bcm2835-codec' on card 'bcm2835-codec-decode' in mplane mode
V4l2m2m 23 formats: YU12(1280x720) -> YU12(1280x720)
(23) 9 export dma fd: (26)
(23) 9 export dma fd: (27)
[V4l2Decoder]: prepare done

receiver ICE-related logs:

=> Set OnSuccess: 
[Receiver] Inside OnIceGatheringChange 
=> OnIceGatheringChange: gathering
[Receiver] ICE is now gathering
[Receiver] Inside OnIceCandidate 
[SignalingService] Invoke AnswerLocalIce: 
[Mosquitto] Inside AnswerLocalIce: 
[Mosquitto] Invoke AnswerICE: 0, => main: wait for closing!
0, candidate:2213883873 1 udp 2122260223 10.80.224.56 34972 typ host generation 0 ufrag UzVd network-id 1
[Receiver] Inside OnIceCandidate 
[SignalingService] Invoke AnswerLocalIce: 
[Mosquitto] Inside AnswerLocalIce: 
[Mosquitto] Invoke AnswerICE: 0, 0, candidate:2726949786 1 udp 2122194687 192.168.2.105 55563 typ host generation 0 ufrag UzVd network-id 4 network-cost 10
[Receiver] Inside OnIceCandidate 
[SignalingService] Invoke AnswerLocalIce: 
[Mosquitto] Inside AnswerLocalIce: 
[Mosquitto] Invoke AnswerICE: 0, 0, candidate:1492520864 1 udp 1686052607 37.153.204.14 53849 typ srflx raddr 10.80.224.56 rport 34972 generation 0 ufrag UzVd network-id 1
=> OnConnectionChange: connecting
[Receiver] OnConnectionChange - IsConnected: 0, StreamingState: 1
[Mosquitto] OfferICE: Received!
[Mosquitto] OfferICE: 0, 0, candidate:3005242381 1 udp 2122260223 192.168.2.104 38430 typ host generation 0 ufrag sSFE network-id 3 network-cost 10
=> AddIceCandidate: start creating
=> AddIceCandidate: add candidated
=> OnIceConnectionChange: Checking
=> AddIceCandidate: end
[Receiver] Inside OnIceCandidate 
[SignalingService] Invoke AnswerLocalIce: 
[Mosquitto] Inside AnswerLocalIce: 
[Mosquitto] Invoke AnswerICE: 0, 0, candidate:4248489337 1 tcp 1518280447 10.80.224.56 43403 typ host tcptype passive generation 0 ufrag UzVd network-id 1
[Receiver] Inside OnIceCandidate 
[SignalingService] Invoke AnswerLocalIce: 
[Mosquitto] Inside AnswerLocalIce: 
[Mosquitto] Invoke AnswerICE: 0, 0, candidate:3695579394 1 tcp 1518214911 192.168.2.105 53689 typ host tcptype passive generation 0 ufrag UzVd network-id 4 network-cost 10

And, for completeness, here is the Mosquitto broker thread:

SenderOfferSDP {"sdp":"v=0\r\no=- 7503600236783864713 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0 1 2\r\na=extmap-allow-mixed\r\na=msid-semantic: WMS test_stream_id\r\nm=audio 9 UDP/TLS/RTP/SAVPF 111 63 9 102 0 8 13 110 126\r\nc=IN IP4 0.0.0.0\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:uy+1\r\na=ice-pwd:hyhzCB+BlP/+ULTWGZDVRoL7\r\na=ice-options:trickle\r\na=fingerprint:sha-256 D1:AC:07:32:C6:44:B4:32:31:DD:25:02:2A:78:93:9C:4B:A5:D9:7E:A0:B2:A9:CC:88:86:D8:ED:45:73:B1:7A\r\na=setup:actpass\r\na=mid:0\r\na=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level\r\na=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\na=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\na=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=sendrecv\r\na=msid:test_stream_id raspberrypi_audio\r\na=rtcp-mux\r\na=rtpmap:111 opus/48000/2\r\na=rtcp-fb:111 transport-cc\r\na=fmtp:111 minptime=10;useinbandfec=1\r\na=rtpmap:63 red/48000/2\r\na=fmtp:63 111/111\r\na=rtpmap:9 G722/8000\r\na=rtpmap:102 ILBC/8000\r\na=rtpmap:0 PCMU/8000\r\na=rtpmap:8 PCMA/8000\r\na=rtpmap:13 CN/8000\r\na=rtpmap:110 telephone-event/48000\r\na=rtpmap:126 telephone-event/8000\r\na=ssrc:1492133001 cname:AK9uDt7RgtfEqZoT\r\na=ssrc:1492133001 msid:test_stream_id raspberrypi_audio\r\nm=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 39 40 100 101 127\r\nc=IN IP4 0.0.0.0\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:uy+1\r\na=ice-pwd:hyhzCB+BlP/+ULTWGZDVRoL7\r\na=ice-options:trickle\r\na=fingerprint:sha-256 D1:AC:07:32:C6:44:B4:32:31:DD:25:02:2A:78:93:9C:4B:A5:D9:7E:A0:B2:A9:CC:88:86:D8:ED:45:73:B1:7A\r\na=setup:actpass\r\na=mid:1\r\na=extmap:14 urn:ietf:params:rtp-hdrext:toffset\r\na=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\na=extmap:13 urn:3gpp:video-orientation\r\na=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\na=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay\r\na=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type\r\na=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing\r\na=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space\r\na=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id\r\na=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id\r\na=sendrecv\r\na=msid:test_stream_id raspberrypi_video\r\na=rtcp-mux\r\na=rtcp-rsize\r\na=rtpmap:96 VP8/90000\r\na=rtcp-fb:96 goog-remb\r\na=rtcp-fb:96 transport-cc\r\na=rtcp-fb:96 ccm fir\r\na=rtcp-fb:96 nack\r\na=rtcp-fb:96 nack pli\r\na=rtpmap:97 rtx/90000\r\na=fmtp:97 apt=96\r\na=rtpmap:98 VP9/90000\r\na=rtcp-fb:98 goog-remb\r\na=rtcp-fb:98 transport-cc\r\na=rtcp-fb:98 ccm fir\r\na=rtcp-fb:98 nack\r\na=rtcp-fb:98 nack pli\r\na=fmtp:98 profile-id=0\r\na=rtpmap:99 rtx/90000\r\na=fmtp:99 apt=98\r\na=rtpmap:39 AV1/90000\r\na=rtcp-fb:39 goog-remb\r\na=rtcp-fb:39 transport-cc\r\na=rtcp-fb:39 ccm fir\r\na=rtcp-fb:39 nack\r\na=rtcp-fb:39 nack pli\r\na=rtpmap:40 rtx/90000\r\na=fmtp:40 apt=39\r\na=rtpmap:100 red/90000\r\na=rtpmap:101 rtx/90000\r\na=fmtp:101 apt=100\r\na=rtpmap:127 ulpfec/90000\r\na=ssrc-group:FID 714481605 3074938706\r\na=ssrc:714481605 cname:AK9uDt7RgtfEqZoT\r\na=ssrc:714481605 msid:test_stream_id raspberrypi_video\r\na=ssrc:3074938706 cname:AK9uDt7RgtfEqZoT\r\na=ssrc:3074938706 msid:test_stream_id raspberrypi_video\r\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 0.0.0.0\r\na=ice-ufrag:uy+1\r\na=ice-pwd:hyhzCB+BlP/+ULTWGZDVRoL7\r\na=ice-options:trickle\r\na=fingerprint:sha-256 D1:AC:07:32:C6:44:B4:32:31:DD:25:02:2A:78:93:9C:4B:A5:D9:7E:A0:B2:A9:CC:88:86:D8:ED:45:73:B1:7A\r\na=setup:actpass\r\na=mid:2\r\na=sctp-port:5000\r\na=max-message-size:262144\r\n"}

ReceiverAnswerSDP {"sdp":"v=0\r\no=- 957605517823660984 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0 1 2\r\na=extmap-allow-mixed\r\na=msid-semantic: WMS\r\nm=audio 9 UDP/TLS/RTP/SAVPF 111 63 9 102 0 8 13 110 126\r\nc=IN IP4 0.0.0.0\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:74sM\r\na=ice-pwd:o5k2dS9ccmjTYohdTjRIzzTu\r\na=ice-options:trickle\r\na=fingerprint:sha-256 76:37:8E:78:9A:7B:F7:54:E0:1A:FF:4E:12:3F:7C:BA:ED:78:23:E0:FC:3E:9F:8C:FD:D6:2F:60:19:2D:5D:AE\r\na=setup:active\r\na=mid:0\r\na=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level\r\na=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\na=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\na=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=recvonly\r\na=rtcp-mux\r\na=rtpmap:111 opus/48000/2\r\na=rtcp-fb:111 transport-cc\r\na=fmtp:111 minptime=10;useinbandfec=1\r\na=rtpmap:63 red/48000/2\r\na=fmtp:63 111/111\r\na=rtpmap:9 G722/8000\r\na=rtpmap:102 ILBC/8000\r\na=rtpmap:0 PCMU/8000\r\na=rtpmap:8 PCMA/8000\r\na=rtpmap:13 CN/8000\r\na=rtpmap:110 telephone-event/48000\r\na=rtpmap:126 telephone-event/8000\r\nm=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 39 40 100 101 127\r\nc=IN IP4 0.0.0.0\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:74sM\r\na=ice-pwd:o5k2dS9ccmjTYohdTjRIzzTu\r\na=ice-options:trickle\r\na=fingerprint:sha-256 76:37:8E:78:9A:7B:F7:54:E0:1A:FF:4E:12:3F:7C:BA:ED:78:23:E0:FC:3E:9F:8C:FD:D6:2F:60:19:2D:5D:AE\r\na=setup:active\r\na=mid:1\r\na=extmap:14 urn:ietf:params:rtp-hdrext:toffset\r\na=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\na=extmap:13 urn:3gpp:video-orientation\r\na=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\na=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay\r\na=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type\r\na=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing\r\na=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space\r\na=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id\r\na=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id\r\na=recvonly\r\na=rtcp-mux\r\na=rtcp-rsize\r\na=rtpmap:96 VP8/90000\r\na=rtcp-fb:96 goog-remb\r\na=rtcp-fb:96 transport-cc\r\na=rtcp-fb:96 ccm fir\r\na=rtcp-fb:96 nack\r\na=rtcp-fb:96 nack pli\r\na=rtpmap:97 rtx/90000\r\na=fmtp:97 apt=96\r\na=rtpmap:98 VP9/90000\r\na=rtcp-fb:98 goog-remb\r\na=rtcp-fb:98 transport-cc\r\na=rtcp-fb:98 ccm fir\r\na=rtcp-fb:98 nack\r\na=rtcp-fb:98 nack pli\r\na=fmtp:98 profile-id=0\r\na=rtpmap:99 rtx/90000\r\na=fmtp:99 apt=98\r\na=rtpmap:39 AV1/90000\r\na=rtcp-fb:39 goog-remb\r\na=rtcp-fb:39 transport-cc\r\na=rtcp-fb:39 ccm fir\r\na=rtcp-fb:39 nack\r\na=rtcp-fb:39 nack pli\r\na=rtpmap:40 rtx/90000\r\na=fmtp:40 apt=39\r\na=rtpmap:100 red/90000\r\na=rtpmap:101 rtx/90000\r\na=fmtp:101 apt=100\r\na=rtpmap:127 ulpfec/90000\r\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 0.0.0.0\r\na=ice-ufrag:74sM\r\na=ice-pwd:o5k2dS9ccmjTYohdTjRIzzTu\r\na=ice-options:trickle\r\na=fingerprint:sha-256 76:37:8E:78:9A:7B:F7:54:E0:1A:FF:4E:12:3F:7C:BA:ED:78:23:E0:FC:3E:9F:8C:FD:D6:2F:60:19:2D:5D:AE\r\na=setup:active\r\na=mid:2\r\na=sctp-port:5000\r\na=max-message-size:262144\r\n","type":"answer"}

ReceiverAnswerICE {"candidate":"candidate:3874208578 1 udp 2122260223 192.168.2.105 60998 typ host generation 0 ufrag 74sM network-id 3 network-cost 10","sdpMLineIndex":0.0,"sdpMid":"0"}

SenderAnswerICE {"candidate":"candidate:2426888493 1 udp 2122260223 192.168.2.104 41080 typ host generation 0 ufrag 1cL+ network-id 3 network-cost 10","sdpMLineIndex":0.0,"sdpMid":"0"}

ReceiverAnswerICE {"candidate":"candidate:2552519130 1 tcp 1518280447 192.168.2.105 52533 typ host tcptype passive generation 0 ufrag 74sM network-id 3 network-cost 10","sdpMLineIndex":0.0,"sdpMid":"0"}

The ICEs seem to be exchanged, however, the peer2peer connection gets stuck and never goes into kConnected status. To debug further, to both conductor and receiver I added the method OnIceConnectionChange. Also there you can see that the connections gets stuck both on sender and receiver side on => OnIceConnectionChange: Checking.

In a first attempt to debug this, I tried to use two different networks, but no success. Note that I haven't yet tested a STUN server, however two remarks on this:

I used webrtc communication on the same network before without STUN server and everything was working fine;
Specifically for my usecase, the setup is the following: receiver creates a hotspot; sender connects to it via regular wifi connection; webrtc communicatino starts. In this setup, internet connection is not even contemplated and therefore a STUN server wouldn't be an option. However, if the peers are on the same local network, WebRTC should, in most cases, be able to establish a direct connection without the need for STUN or TURN servers, right?

Alright, this is it for now. I would be very grateful if could give this setup a try. Of course, I am here to answer any questions!

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024

I added also the dependent libs including mosquitto's .so, .a and header files here: https://github.com/eliabruni/RaspberryPi_WebRTC/releases/tag/m115-v0.1.0

from raspberrypi_webrtc.

TzuHuanTai commented on May 19, 2024

I just finished my vacation. I'll browse/test in these few days.

STUN server is used for going through the NAT setting. If there are no settings in the router related to NAT, I think it is not necessary. As far as I know, webrtc will grab the local IP address info from the internet setting while establishing ICE. It should work without STUN or TURN in your case.

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024

I hope you had nice holidays!

Thanks for confirming about STUN/TURN. And thanks, I'll wait for you to have a look, in the hope I made some obvious, easy-to-catch mistake.

from raspberrypi_webrtc.

TzuHuanTai commented on May 19, 2024

After running and briefly reviewing your codes, I've found a problem.
The client's SDP includes vp8, vp9, av1, but the h264 is missing. The SDP needs to pick up the desired codec and remove others on the client side [ref]. In your reciever.cpp code here seems directly copy the codecs setting from conductor.cpp. You can refer to the webrtc official client example here and left needed codec, the created SDP by webrtc will auto apply the codecs setting then.
Below is my SDP from my receiver(browser) to grab h264 streaming, for your information.

 v=0
 o=- 7592745959975588407 2 IN IP4 127.0.0.1
 s=-
 t=0 0
 a=group:BUNDLE 0 1 2
 a=extmap-allow-mixed
 a=msid-semantic: WMS
 m=video 9 UDP/TLS/RTP/SAVPF 102 103 104 105 106 107 108 109 127 125 39 40 41 42 43 44 112 113 114 115 116 117 118 49
 c=IN IP4 0.0.0.0
 a=rtcp:9 IN IP4 0.0.0.0
 a=ice-ufrag:0XdG
 a=ice-pwd:+5/jP+Rf/MkyNms2/wQQsIx5
 a=ice-options:trickle
 a=fingerprint:sha-256 59:FD:06:A8:32:15:AD:7C:27:1E:C2:9E:CA:18:29:70:6C:03:BA:AB:1A:3B:BF:97:C8:DD:56:44:08:8B:B8:F2
 a=setup:actpass
 a=mid:0
 a=extmap:1 urn:ietf:params:rtp-hdrext:toffset
 a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
 a=extmap:3 urn:3gpp:video-orientation
 a=extmap:4 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
 a=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
 a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
 a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
 a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space
 a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
 a=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
 a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
 a=recvonly
 a=rtcp-mux
 a=rtcp-rsize
 a=rtpmap:102 H264/90000
 a=rtcp-fb:102 goog-remb
 a=rtcp-fb:102 transport-cc
 a=rtcp-fb:102 ccm fir
 a=rtcp-fb:102 nack
 a=rtcp-fb:102 nack pli
 a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f
 a=rtpmap:103 rtx/90000
 a=fmtp:103 apt=102
 a=rtpmap:104 H264/90000
 a=rtcp-fb:104 goog-remb
 a=rtcp-fb:104 transport-cc
 a=rtcp-fb:104 ccm fir
 a=rtcp-fb:104 nack
 a=rtcp-fb:104 nack pli
 a=fmtp:104 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f
 a=rtpmap:105 rtx/90000
 a=fmtp:105 apt=104
 a=rtpmap:106 H264/90000
 a=rtcp-fb:106 goog-remb
 a=rtcp-fb:106 transport-cc
 a=rtcp-fb:106 ccm fir
 a=rtcp-fb:106 nack
 a=rtcp-fb:106 nack pli
 a=fmtp:106 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
 a=rtpmap:107 rtx/90000
 a=fmtp:107 apt=106
 a=rtpmap:108 H264/90000
 a=rtcp-fb:108 goog-remb
 a=rtcp-fb:108 transport-cc
 a=rtcp-fb:108 ccm fir
 a=rtcp-fb:108 nack
 a=rtcp-fb:108 nack pli
 a=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f
 a=rtpmap:109 rtx/90000
 a=fmtp:109 apt=108
 a=rtpmap:127 H264/90000
 a=rtcp-fb:127 goog-remb
 a=rtcp-fb:127 transport-cc
 a=rtcp-fb:127 ccm fir
 a=rtcp-fb:127 nack
 a=rtcp-fb:127 nack pli
 a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d001f
 a=rtpmap:125 rtx/90000
 a=fmtp:125 apt=127
 a=rtpmap:39 H264/90000
 a=rtcp-fb:39 goog-remb
 a=rtcp-fb:39 transport-cc
 a=rtcp-fb:39 ccm fir
 a=rtcp-fb:39 nack
 a=rtcp-fb:39 nack pli
 a=fmtp:39 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=4d001f
 a=rtpmap:40 rtx/90000
 a=fmtp:40 apt=39
 a=rtpmap:41 H264/90000
 a=rtcp-fb:41 goog-remb
 a=rtcp-fb:41 transport-cc
 a=rtcp-fb:41 ccm fir
 a=rtcp-fb:41 nack
 a=rtcp-fb:41 nack pli
 a=fmtp:41 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=f4001f
 a=rtpmap:42 rtx/90000
 a=fmtp:42 apt=41
 a=rtpmap:43 H264/90000
 a=rtcp-fb:43 goog-remb
 a=rtcp-fb:43 transport-cc
 a=rtcp-fb:43 ccm fir
 a=rtcp-fb:43 nack
 a=rtcp-fb:43 nack pli
 a=fmtp:43 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=f4001f
 a=rtpmap:44 rtx/90000
 a=fmtp:44 apt=43
 a=rtpmap:112 H264/90000
 a=rtcp-fb:112 goog-remb
 a=rtcp-fb:112 transport-cc
 a=rtcp-fb:112 ccm fir
 a=rtcp-fb:112 nack
 a=rtcp-fb:112 nack pli
 a=fmtp:112 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=64001f
 a=rtpmap:113 rtx/90000
 a=fmtp:113 apt=112
 a=rtpmap:114 H264/90000
 a=rtcp-fb:114 goog-remb
 a=rtcp-fb:114 transport-cc
 a=rtcp-fb:114 ccm fir
 a=rtcp-fb:114 nack
 a=rtcp-fb:114 nack pli
 a=fmtp:114 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=64001f
 a=rtpmap:115 rtx/90000
 a=fmtp:115 apt=114
 a=rtpmap:116 red/90000
 a=rtpmap:117 rtx/90000
 a=fmtp:117 apt=116
 a=rtpmap:118 ulpfec/90000
 a=rtpmap:49 flexfec-03/90000
 a=rtcp-fb:49 goog-remb
 a=rtcp-fb:49 transport-cc
 a=fmtp:49 repair-window=10000000
 m=audio 9 UDP/TLS/RTP/SAVPF 111 63 9 0 8 13 110 126
 c=IN IP4 0.0.0.0
 a=rtcp:9 IN IP4 0.0.0.0
 a=ice-ufrag:0XdG
 a=ice-pwd:+5/jP+Rf/MkyNms2/wQQsIx5
 a=ice-options:trickle
 a=fingerprint:sha-256 59:FD:06:A8:32:15:AD:7C:27:1E:C2:9E:CA:18:29:70:6C:03:BA:AB:1A:3B:BF:97:C8:DD:56:44:08:8B:B8:F2
 a=setup:actpass
 a=mid:1
 a=extmap:14 urn:ietf:params:rtp-hdrext:ssrc-audio-level
 a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
 a=extmap:4 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
 a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
 a=recvonly
 a=rtcp-mux
 a=rtpmap:111 opus/48000/2
 a=rtcp-fb:111 transport-cc
 a=fmtp:111 minptime=10;useinbandfec=1
 a=rtpmap:63 red/48000/2
 a=fmtp:63 111/111
 a=rtpmap:9 G722/8000
 a=rtpmap:0 PCMU/8000
 a=rtpmap:8 PCMA/8000
 a=rtpmap:13 CN/8000
 a=rtpmap:110 telephone-event/48000
 a=rtpmap:126 telephone-event/8000
 m=application 9 UDP/DTLS/SCTP webrtc-datachannel
 c=IN IP4 0.0.0.0
 a=ice-ufrag:0XdG
 a=ice-pwd:+5/jP+Rf/MkyNms2/wQQsIx5
 a=ice-options:trickle
 a=fingerprint:sha-256 59:FD:06:A8:32:15:AD:7C:27:1E:C2:9E:CA:18:29:70:6C:03:BA:AB:1A:3B:BF:97:C8:DD:56:44:08:8B:B8:F2
 a=setup:actpass
 a=mid:2
 a=sctp-port:5000
 a=max-message-size:262144

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024

Good catch! However, if I try to fix it this way, I get Segmentation fault on the receiver side:

./webrtc_receiver --mqtt_host=192.168.2.1 --mqtt_port=2550
=> peer_connection_factory: success!
[SignalingService] Inside InitIceCallback: 
[Receiver][MQTT][WithReconnect]: Attempting reconnection
iphost192.168.2.1
=> main: wait for signaling!
Successfully subscribed to topic SenderOfferSDP
Successfully subscribed to topic SenderOfferICE
Successfully subscribed to topic SenderAnswerSDP
Successfully subscribed to topic SenderAnswerICE
Connected to MQTT broker.
[Mosquitto] OfferSDP: Received!
=> SetOfferSDP: start
=> OnSignalingChange: have-remote-offer
Segmentation fault

And I suspect that this is due to the sender sending out an SDP offer which doesnt specify H264 as encoding (in this version, I made such that the initial SDP offer is sent by the sender; see here and here):

./webrtc_sender --device=/dev/video0 --fps=30 --width=1280 --height=720 --v4l2_format=h264 --enable_v4l2_dma --mqtt_host=192.168.2.1 --mqtt_port=2550
=> network thread start: success!
=> worker thread start: success!
=> signaling thread start: success!
=> peer_connection_factory: success!
Open file /dev/video0 fd(21) success!
GetDeviceName(0): device_name=mmal service 16.1, unique_name=platform:bcm2835_v4l2-0
driver 'bcm2835 mmal' on card 'mmal service 16.1' in splane mode
  Fps: 30
  Rotation: 0
Use h264 format source in v4l2
V4l2m2m 21 formats: JPEG(1024x768) -> H264(1280x720)
V4l2m2m querying (21) 1 buffer: 0x7f8d27b000 with 921600 length
V4l2m2m querying (21) 1 buffer: 0x7f8d19a000 with 921600 length
Open file /dev/video12 fd(22) success!
driver 'bcm2835-codec' on card 'bcm2835-codec-isp' in mplane mode
V4l2m2m 22 formats: BA81(32x32) -> YU12(1280x720)
Open file /dev/video10 fd(23) success!
driver 'bcm2835-codec' on card 'bcm2835-codec-decode' in mplane mode
V4l2m2m 23 formats: MPG4(32x32) -> H264(1280x720)
driver 'bcm2835-codec' on card 'bcm2835-codec-isp' in mplane mode
V4l2m2m 22 formats: YUYV(32x32) -> YU12(1280x720)
V4l2m2m querying (23) 10 buffer: 0x7f8c017000 with 524288 length
V4l2m2m querying (23) 10 buffer: 0x7f67f80000 with 524288 length
driver 'bcm2835-codec' on card 'bcm2835-codec-decode' in mplane mode
V4l2m2m 23 formats: YU12(1280x720) -> YU12(1280x720)
(22) 9 export dma fd: (24)
(22) 9 export dma fd: (25)
(23) 9 export dma fd: (26)
(23) 9 export dma fd: (27)
[V4l2Scaler]: prepare done
[V4l2Decoder]: prepare done
=> InitializeRecorder: Recorder is not created!
Succeeds to create data channel
[SignalingService] Inside InitIceCallback 
[Receiver][MQTT][WithReconnect]: Attempting reconnection
Successfully subscribed to topic ReceiverOfferSDP
IP 192.168.2.1 Port 2550
Successfully subscribed to topic ReceiverOfferICE
IP 192.168.2.1 Port 2550
Successfully subscribed to topic ReceiverAnswerSDP
IP 192.168.2.1 Port 2550
Successfully subscribed to topic ReceiverAnswerICE
IP 192.168.2.1 Port 2550
MQTT service is ready.
Waiting for MQTT service to be ready...
Conductor::SetSignalingDelegate - Delegate set.
Conductor::CreateOffer - Starting to create offer.
iphost192.168.2.1
=> main: Before wait for signaling - IsReadyForStreaming: 0, IsConnected: 0
=> Set OnSuccess: 
Receiver::CreateOffer - Offer created. SDP: v=0
o=- 7088025240881729248 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=extmap-allow-mixed
a=msid-semantic: WMS test_stream_id
m=audio 9 UDP/TLS/RTP/SAVPF 111 63 9 102 0 8 13 110 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:s0pd
a=ice-pwd:jv6OexQwJLPFfClnc1vriBbP
a=ice-options:trickle
a=fingerprint:sha-256 42:E9:60:DB:EF:83:E2:91:F3:07:CB:F5:5D:A6:6C:B1:15:B5:C3:47:06:B6:16:19:BD:2F:6F:D9:CA:DF:81:89
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendrecv
a=msid:test_stream_id raspberrypi_audio
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:63 red/48000/2
a=fmtp:63 111/111
a=rtpmap:9 G722/8000
a=rtpmap:102 ILBC/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:126 telephone-event/8000
a=ssrc:2234857121 cname:y7jsl6IM+o0oRDQB
a=ssrc:2234857121 msid:test_stream_id raspberrypi_audio
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 39 40 100 101 127
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:s0pd
a=ice-pwd:jv6OexQwJLPFfClnc1vriBbP
a=ice-options:trickle
a=fingerprint:sha-256 42:E9:60:DB:EF:83:E2:91:F3:07:CB:F5:5D:A6:6C:B1:15:B5:C3:47:06:B6:16:19:BD:2F:6F:D9:CA:DF:81:89
a=setup:actpass
a=mid:1
a=extmap:14 urn:ietf:params:rtp-hdrext:toffset
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:13 urn:3gpp:video-orientation
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=sendrecv
a=msid:test_stream_id raspberrypi_video
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 profile-id=0
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:39 AV1/90000
a=rtcp-fb:39 goog-remb
a=rtcp-fb:39 transport-cc
a=rtcp-fb:39 ccm fir
a=rtcp-fb:39 nack
a=rtcp-fb:39 nack pli
a=rtpmap:40 rtx/90000
a=fmtp:40 apt=39
a=rtpmap:100 red/90000
a=rtpmap:101 rtx/90000
a=fmtp:101 apt=100
a=rtpmap:127 ulpfec/90000
a=ssrc-group:FID 2208810632 624887038
a=ssrc:2208810632 cname:y7jsl6IM+o0oRDQB
a=ssrc:2208810632 msid:test_stream_id raspberrypi_video
a=ssrc:624887038 cname:y7jsl6IM+o0oRDQB
a=ssrc:624887038 msid:test_stream_id raspberrypi_video
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:s0pd
a=ice-pwd:jv6OexQwJLPFfClnc1vriBbP
a=ice-options:trickle
a=fingerprint:sha-256 42:E9:60:DB:EF:83:E2:91:F3:07:CB:F5:5D:A6:6C:B1:15:B5:C3:47:06:B6:16:19:BD:2F:6F:D9:CA:DF:81:89
a=setup:actpass
a=mid:2
a=sctp-port:5000
a=max-message-size:262144

MosquittoService::PublishMessage - Message published on topic: SenderOfferSDP
Main - Message published to topic: SenderOfferSDP
Conductor::CreateOffer - Delegate called to send offer.

from raspberrypi_webrtc.

TzuHuanTai commented on May 19, 2024

Do you mind if I refer to (copy) your code and try it out?
I can try mosquito by using "my receiver".

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024

Quite the contrary, I would be grateful if you could give it a try!

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024

I invited you to the repo in case it helps.

from raspberrypi_webrtc.

TzuHuanTai commented on May 19, 2024

I received the video on my web receiver by mqtt signaling successfully. I'll clean, refactor some code, and rewrite docs then merge it into main branch in recent days if I have free time.

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024

That's great news! OK, I'll wait for updates then.

from raspberrypi_webrtc.

TzuHuanTai commented on May 19, 2024

I already merged to main and updated docs.

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024

I noticed, awesome! I am currently on holiday, once back I test it.

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024

Hello @TzuHuanTai, I reviewed the code, which looks very well organized now! Despite that, I am still hitting the same issue though and I am not sure how to proceed. Let me try to explain it to you.

Looking at your code, I suspect that the initiative of triggering the SDP exchange is left to the frontend, correct?

What I am trying to do is the following:

Start the SDP exchange from the RPI client. To do that, I first try to print the SDP and inspect it. I do it with this simple method:

void Conductor::CreateOffer() {
    std::cout << "Conductor::CreateOffer - Starting to create offer." << std::endl;

    auto on_offer_created = [this](webrtc::SessionDescriptionInterface* desc) {
        std::string sdp;
        desc->ToString(&sdp);
        std::cout << "Conductor::CreateOffer - Offer created. SDP: " << sdp << std::endl;
    };

}

The problem with that is that I get the following SDP, which I believe is dictated by these (as you can see, no trace of h264):

SDP: v=0
o=- 3622495593741922904 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=extmap-allow-mixed
a=msid-semantic: WMS test_stream_id
m=audio 9 UDP/TLS/RTP/SAVPF 111 63 9 102 0 8 13 110 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:Qlek
a=ice-pwd:XbCePd/89Kp9QCcjyRr/DXyd
a=ice-options:trickle
a=fingerprint:sha-256 C0:4A:29:57:55:C8:A4:77:83:88:87:0A:1B:18:E1:A6:B8:F4:21:7C:53:C2:E0:B8:F7:11:90:24:94:FF:23:89
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendrecv
a=msid:test_stream_id raspberrypi_audio
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:63 red/48000/2
a=fmtp:63 111/111
a=rtpmap:9 G722/8000
a=rtpmap:102 ILBC/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:126 telephone-event/8000
a=ssrc:493255696 cname:4gyOrv5Ca0nKtKEk
a=ssrc:493255696 msid:test_stream_id raspberrypi_audio
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 39 40 100 101 127
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:Qlek
a=ice-pwd:XbCePd/89Kp9QCcjyRr/DXyd
a=ice-options:trickle
a=fingerprint:sha-256 C0:4A:29:57:55:C8:A4:77:83:88:87:0A:1B:18:E1:A6:B8:F4:21:7C:53:C2:E0:B8:F7:11:90:24:94:FF:23:89
a=setup:actpass
a=mid:1
a=extmap:14 urn:ietf:params:rtp-hdrext:toffset
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:13 urn:3gpp:video-orientation
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=sendrecv
a=msid:test_stream_id raspberrypi_video
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 profile-id=0
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:39 AV1/90000
a=rtcp-fb:39 goog-remb
a=rtcp-fb:39 transport-cc
a=rtcp-fb:39 ccm fir
a=rtcp-fb:39 nack
a=rtcp-fb:39 nack pli
a=rtpmap:40 rtx/90000
a=fmtp:40 apt=39
a=rtpmap:100 red/90000
a=rtpmap:101 rtx/90000
a=fmtp:101 apt=100
a=rtpmap:127 ulpfec/90000
a=ssrc-group:FID 755891420 1925509087
a=ssrc:755891420 cname:4gyOrv5Ca0nKtKEk
a=ssrc:755891420 msid:test_stream_id raspberrypi_video
a=ssrc:1925509087 cname:4gyOrv5Ca0nKtKEk
a=ssrc:1925509087 msid:test_stream_id raspberrypi_video
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:Qlek
a=ice-pwd:XbCePd/89Kp9QCcjyRr/DXyd
a=ice-options:trickle
a=fingerprint:sha-256 C0:4A:29:57:55:C8:A4:77:83:88:87:0A:1B:18:E1:A6:B8:F4:21:7C:53:C2:E0:B8:F7:11:90:24:94:FF:23:89
a=setup:actpass
a=mid:2
a=sctp-port:5000
a=max-message-size:262144

In fact, if I replace those with the following:

          webrtc::OpenH264DecoderTemplateAdapter>>();
        //   webrtc::LibvpxVp8DecoderTemplateAdapter,
        //   webrtc::LibvpxVp9DecoderTemplateAdapter,
        //   webrtc::Dav1dDecoderTemplateAdapter>>();

I get an incomplete offer (no trace of h264 either):

DP: v=0
o=- 65187495654357602 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=extmap-allow-mixed
a=msid-semantic: WMS test_stream_id
m=audio 9 UDP/TLS/RTP/SAVPF 111 63 9 102 0 8 13 110 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:0dRW
a=ice-pwd:fa8Ek9RHvtBjAXQ7U/J8UE8a
a=ice-options:trickle
a=fingerprint:sha-256 AD:CE:58:B3:35:9C:DD:97:61:BD:8C:42:F2:64:8C:0C:6F:BF:0A:C4:DF:2B:BB:03:FB:2B:2D:3D:78:27:77:29
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendrecv
a=msid:test_stream_id raspberrypi_audio
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:63 red/48000/2
a=fmtp:63 111/111
a=rtpmap:9 G722/8000
a=rtpmap:102 ILBC/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:126 telephone-event/8000
a=ssrc:2144757454 cname:F1Y/YlumvstoW2ni
a=ssrc:2144757454 msid:test_stream_id raspberrypi_audio
m=video 9 UDP/TLS/RTP/SAVPF 0
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:0dRW
a=ice-pwd:fa8Ek9RHvtBjAXQ7U/J8UE8a
a=ice-options:trickle
a=fingerprint:sha-256 AD:CE:58:B3:35:9C:DD:97:61:BD:8C:42:F2:64:8C:0C:6F:BF:0A:C4:DF:2B:BB:03:FB:2B:2D:3D:78:27:77:29
a=setup:actpass
a=mid:1
a=extmap:14 urn:ietf:params:rtp-hdrext:toffset
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:13 urn:3gpp:video-orientation
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=sendrecv
a=msid:test_stream_id raspberrypi_video
a=rtcp-mux
a=rtcp-rsize
a=ssrc:309919679 cname:F1Y/YlumvstoW2ni
a=ssrc:309919679 msid:test_stream_id raspberrypi_video
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:0dRW
a=ice-pwd:fa8Ek9RHvtBjAXQ7U/J8UE8a
a=ice-options:trickle
a=fingerprint:sha-256 AD:CE:58:B3:35:9C:DD:97:61:BD:8C:42:F2:64:8C:0C:6F:BF:0A:C4:DF:2B:BB:03:FB:2B:2D:3D:78:27:77:29
a=setup:actpass
a=mid:2
a=sctp-port:5000
a=max-message-size:262144

Now, if I instead try to kickstart the SDP offer from my receiver, I get an even smaller SDP offer and that's because all the RTP capabilities are to be inferred from the conductor (i.e., I dont create any tracks on the receiver side):

SDP: v=0
o=- 234016104920187586 2 IN IP4 127.0.0.1
s=-
t=0 0
a=extmap-allow-mixed
a=msid-semantic: WMS

I am quite puzzled and do not understand what am I missing in this logic.

from raspberrypi_webrtc.

TzuHuanTai commented on May 19, 2024

Yes, my frontend leaves the specific codec info to decide it.
Maybe you need to recompile the libwebrtc.a because rtc_use_h264=false is used in our instructions.
The macro WEBRTC_USE_H264 is assigned false, so the h264 will never appear in SDP. ref

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024

Hello @TzuHuanTai , thanks again for the hint. Let me paste here the progress I made so far (problem not solved yet).

First, I check ffmpeg's h264 availability:

ffmpeg -codecs | grep 'h264_omx\|h264_v4l2m2m'
ffmpeg version 4.3.6-0+deb11u1+rpt5 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/usr --extra-version=0+deb11u1+rpt5 --toolchain=hardened --incdir=/usr/include/aarch64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --disable-mmal --enable-neon --enable-v4l2-request --enable-libudev --enable-epoxy --enable-sand --libdir=/usr/lib/aarch64-linux-gnu --arch=arm64 --enable-pocketsphinx --enable-libdc1394 --enable-libdrm --enable-vout-drm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
 DEV.LS h264                 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (decoders: h264 h264_v4l2m2m ) (encoders: libx264 libx264rgb h264_omx h264_v4l2m2m h264_vaapi )

Them, following this, I recompile libwertc.a with the following command:

gn gen out/Release64 --args='target_os="linux" target_cpu="arm64" rtc_include_tests=false rtc_use_h264=true proprietary_codecs=true ffmpeg_branding="Chrome" use_rtti=true is_component_build=false is_debug=false rtc_build_examples=false use_custom_libcxx=false rtc_build_tools=false rtc_use_pipewire=false clang_base_path="/home/pi/clang+llvm-16.0.6-aarch64-linux-gnu" clang_use_chrome_plugins=false'

Then, I recompile pi_webrtc using the new libwertc.a. Then, I try again the same trick on the conductor:

media_dependencies.video_encoder_factory = CreateCustomizedVideoEncoderFactory(args, data_channel_subject_);
    media_dependencies.video_decoder_factory = std::make_unique<webrtc::VideoDecoderFactoryTemplate<
          webrtc::OpenH264DecoderTemplateAdapter>>();
        //   webrtc::LibvpxVp8DecoderTemplateAdapter,
        //   webrtc::LibvpxVp9DecoderTemplateAdapter,
        //   webrtc::Dav1dDecoderTemplateAdapter>>();

And I print the offer like this, in the main:

 auto conductor = Conductor::Create(args);
      // Conductor-specific logic
      while (true) {
          if (!conductor->CreatePeerConnection()) {
              std::cerr << "Failed to create Peer Connection." << std::endl;
              continue;
          }

          std::cout << "=> main: Peer Connection created." << std::endl;

          conductor->CreateOffer();

and here is CreateOffer:

void Conductor::CreateOffer() {
  std::cout << "Conductor::CreateOffer - Starting to create offer." << std::endl;

  auto on_offer_created = [this](webrtc::SessionDescriptionInterface* desc) {
      std::string sdp;
      desc->ToString(&sdp);
      std::cout << "Conductor::CreateOffer - Offer created. SDP: " << sdp << std::endl;
   }

but that's what I get:

./pi_webrtc --device=/dev/video0 --fps=30 --width=1280 --height=720 --v4l2_format=h264  --mqtt_host=10.80.224.55 --mqtt_port=2550  --enable_v4l2_dma
=> network thread start: success!
=> worker thread start: success!
=> signaling thread start: success!
=> peer_connection_factory: success!
Open file /dev/video0 fd(21) success!
GetDeviceName(0): device_name=mmal service 16.1, unique_name=platform:bcm2835_v4l2-0
driver 'bcm2835 mmal' on card 'mmal service 16.1' in splane mode
Fps: 30
Rotation: 0
Use mjpeg format source in v4l2
V4l2m2m 21 formats: H264(1280x720) -> MJPG(1280x720)
V4l2m2m querying (21) 1 buffer: 0x7f90385000 with 921600 length
V4l2m2m querying (21) 1 buffer: 0x7f902a4000 with 921600 length
Open file /dev/video10 fd(22) success!
driver 'bcm2835-codec' on card 'bcm2835-codec-decode' in mplane mode
V4l2m2m 22 formats: MPG4(32x32)Open file /dev/video12 fd(23) success!
driver 'bcm2835-codec' on card 'bcm2835-codec-isp' in mplane mode
V4l2m2m 23 formats: BA81(32x32) -> MJPG(1280x720)
-> YU12(1280x720)
driver 'bcm2835-codec' on card 'bcm2835-codec-isp' in mplane mode
V4l2m2m 23 formats: YUYV(32x32) -> YU12(1280x720)
V4l2m2m querying (22) 10 buffer: 0x7f90123000 with 524288 length
V4l2m2m querying (22) 10 buffer: 0x7f900a3000 with 524288 length
driver 'bcm2835-codec' on card 'bcm2835-codec-decode' in mplane mode
V4l2m2m 22 formats: YU12(1280x720) -> YU12(1280x720)
(23) 9 export dma fd: (24)
(23) 9 export dma fd: (25)
(22) 9 export dma fd: (26)
(22) 9 export dma fd: (27)
[V4l2Scaler]: prepare done
[V4l2Decoder]: prepare done
[Conductor] Recorder is not created!
Succeeds to create data channel
Successfully subscribed to topic: OfferSDP
Successfully subscribed to topic: OfferICE
MQTT service is ready.
=> main: Peer Connection created.
Conductor::CreateOffer - Starting to create offer.
=> Set OnSuccess: 
=> main: Offer created and sent.
=> main: wait for signaling!
Conductor::CreateOffer - Offer created. SDP: v=0
o=- 3258867279528006577 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=extmap-allow-mixed
a=msid-semantic: WMS test_stream_id
m=audio 9 UDP/TLS/RTP/SAVPF 111 63 9 102 0 8 13 110 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:OFAd
a=ice-pwd:RiIf2yatr/uMhH5VCkBFMq59
a=ice-options:trickle
a=fingerprint:sha-256 10:2E:9E:C5:17:44:73:FD:32:3B:67:37:54:24:9F:EE:3A:11:AF:17:27:40:5E:C3:21:97:8B:EE:17:DD:7D:98
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendrecv
a=msid:test_stream_id raspberrypi_audio
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:63 red/48000/2
a=fmtp:63 111/111
a=rtpmap:9 G722/8000
a=rtpmap:102 ILBC/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:126 telephone-event/8000
a=ssrc:4124206175 cname:dPIg9Nrs2Ke4BkjC
a=ssrc:4124206175 msid:test_stream_id raspberrypi_audio
m=video 9 UDP/TLS/RTP/SAVPF 0
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:OFAd
a=ice-pwd:RiIf2yatr/uMhH5VCkBFMq59
a=ice-options:trickle
a=fingerprint:sha-256 10:2E:9E:C5:17:44:73:FD:32:3B:67:37:54:24:9F:EE:3A:11:AF:17:27:40:5E:C3:21:97:8B:EE:17:DD:7D:98
a=setup:actpass
a=mid:1
a=extmap:14 urn:ietf:params:rtp-hdrext:toffset
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:13 urn:3gpp:video-orientation
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=sendrecv
a=msid:test_stream_id raspberrypi_video
a=rtcp-mux
a=rtcp-rsize
a=ssrc:3237880866 cname:dPIg9Nrs2Ke4BkjC
a=ssrc:3237880866 msid:test_stream_id raspberrypi_video
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:OFAd
a=ice-pwd:RiIf2yatr/uMhH5VCkBFMq59
a=ice-options:trickle
a=fingerprint:sha-256 10:2E:9E:C5:17:44:73:FD:32:3B:67:37:54:24:9F:EE:3A:11:AF:17:27:40:5E:C3:21:97:8B:EE:17:DD:7D:98
a=setup:actpass
a=mid:2
a=sctp-port:5000
a=max-message-size:262144

from raspberrypi_webrtc.

eliabruni commented on May 19, 2024

I might have fixed it. In the project CMakeLists.txt I added WEBRTC_USE_H264 globally:

add_compile_definitions(
    WEBRTC_POSIX
    WEBRTC_LINUX
    USE_CPPRESTSDK
    WEBRTC_USE_H264
)

that's because, despite flagging it during libwertc.a compilation, it was still not set within the pi_webrtc project. Now, using this expanded CreateOffer function:

void Conductor::CreateOffer() {

    #ifdef WEBRTC_USE_H264
        std::cout << "Conductor::CreateOffer - WEBRTC_USE_H264 is defined.\n";
        auto supportedCodecs = webrtc::SupportedH264DecoderCodecs();
        std::cout << "Conductor::CreateOffer - Supported H264 Decoder Codecs: ";
        for (const auto& codec : supportedCodecs) {
            std::cout << codec.name << " ";
        }
        std::cout << std::endl;
    #else
        std::cout << "WEBRTC_USE_H264 is not defined.\n";
    #endif


    std::cout << "Conductor::CreateOffer - Starting to create offer." << std::endl;

    auto on_offer_created = [this](webrtc::SessionDescriptionInterface* desc) {
        std::string sdp;
        desc->ToString(&sdp);
        std::cout << "Conductor::CreateOffer - Offer created. SDP: " << sdp << std::endl;
       }
    }

I get this offer from the conductor:

./pi_webrtc --device=/dev/video0 --fps=30 --width=1280 --height=720 --v4l2_format=h264 --mqtt_host=10.80.224.55 --mqtt_port=2550  --enable_v4l2_dma
=> network thread start: success!
=> worker thread start: success!
=> signaling thread start: success!
=> peer_connection_factory: success!
Open file /dev/video0 fd(21) success!
GetDeviceName(0): device_name=mmal service 16.1, unique_name=platform:bcm2835_v4l2-0
driver 'bcm2835 mmal' on card 'mmal service 16.1' in splane mode
  Fps: 30
  Rotation: 0
Use h264 format source in v4l2
V4l2m2m 21 formats: H264(1280x720) -> H264(1280x720)
V4l2m2m querying (21) 1 buffer: 0x7f81bc0000 with 921600 length
V4l2m2m querying (21) 1 buffer: 0x7f81adf000 with 921600 length
Open file /dev/video12 fd(22) success!
driver 'bcm2835-codec' on card 'bcm2835-codec-isp' in mplane mode
V4l2m2m 22 formats: BA81(32x32) -> YU12(1280x720)
Open file /dev/video10 fd(23) success!
driver 'bcm2835-codec' on card 'bcm2835-codec-decode' in mplane mode
V4l2m2m 23 formats: MPG4(32x32) -> H264(1280x720)
driver 'bcm2835-codec' on card 'bcm2835-codec-isp' in mplane mode
V4l2m2m 22 formats: YUYV(32x32) -> YU12(1280x720)
(22) 9 export dma fd: (24)
(22) 9 export dma fd: (25)
V4l2m2m querying (23) 10 buffer: 0x7f8095c000 with 524288 length
V4l2m2m querying (23) 10 buffer: 0x7f808dc000 with 524288 length
driver 'bcm2835-codec' on card 'bcm2835-codec-decode' in mplane mode
V4l2m2m 23 formats: YU12(1280x720) -> YU12(1280x720)
(23) 9 export dma fd: (26)
(23) 9 export dma fd: (27)
[V4l2Scaler]: prepare done
[V4l2Decoder]: prepare done
[Conductor] Recorder is not created!
Succeeds to create data channel
=> main: Peer Connection created.
Conductor::CreateOffer - WEBRTC_USE_H264 is defined.
Conductor::CreateOffer - Supported H264 Decoder Codecs: H264 H264 H264 H264 H264 H264 H264 H264 
Conductor::CreateOffer - Starting to create offer.
Successfully subscribed to topic: OfferSDP
Successfully subscribed to topic: OfferICE
MQTT service is ready.
=> Set OnSuccess: 
=> main: Offer created and sent.
=> main: wait for signaling!
Conductor::CreateOffer - Offer created. SDP: v=0
o=- 3044645544840007406 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=extmap-allow-mixed
a=msid-semantic: WMS test_stream_id
m=audio 9 UDP/TLS/RTP/SAVPF 111 63 9 102 0 8 13 110 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:JV7p
a=ice-pwd:9QuByHcN30nbuny4SRFpg9l3
a=ice-options:trickle
a=fingerprint:sha-256 A4:1A:F8:F4:BB:7E:95:CC:34:AD:BE:E3:B1:91:57:FB:F7:56:6D:BC:5B:E5:4A:09:1F:25:E7:02:F2:89:8E:B9
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendrecv
a=msid:test_stream_id raspberrypi_audio
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:63 red/48000/2
a=fmtp:63 111/111
a=rtpmap:9 G722/8000
a=rtpmap:102 ILBC/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:126 telephone-event/8000
a=ssrc:3406324158 cname:m4sf6bBvTL1hebl+
a=ssrc:3406324158 msid:test_stream_id raspberrypi_audio
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 106 107 108
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:JV7p
a=ice-pwd:9QuByHcN30nbuny4SRFpg9l3
a=ice-options:trickle
a=fingerprint:sha-256 A4:1A:F8:F4:BB:7E:95:CC:34:AD:BE:E3:B1:91:57:FB:F7:56:6D:BC:5B:E5:4A:09:1F:25:E7:02:F2:89:8E:B9
a=setup:actpass
a=mid:1
a=extmap:14 urn:ietf:params:rtp-hdrext:toffset
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:13 urn:3gpp:video-orientation
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=sendrecv
a=msid:test_stream_id raspberrypi_video
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 H264/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=fmtp:96 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 H264/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:106 red/90000
a=rtpmap:107 rtx/90000
a=fmtp:107 apt=106
a=rtpmap:108 ulpfec/90000
a=ssrc-group:FID 1304973974 3977688616
a=ssrc:1304973974 cname:m4sf6bBvTL1hebl+
a=ssrc:1304973974 msid:test_stream_id raspberrypi_video
a=ssrc:3977688616 cname:m4sf6bBvTL1hebl+
a=ssrc:3977688616 msid:test_stream_id raspberrypi_video
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:JV7p
a=ice-pwd:9QuByHcN30nbuny4SRFpg9l3
a=ice-options:trickle
a=fingerprint:sha-256 A4:1A:F8:F4:BB:7E:95:CC:34:AD:BE:E3:B1:91:57:FB:F7:56:6D:BC:5B:E5:4A:09:1F:25:E7:02:F2:89:8E:B9
a=setup:actpass
a=mid:2
a=sctp-port:5000
a=max-message-size:262144

from raspberrypi_webrtc.

TzuHuanTai commented on May 19, 2024

I'm glad you found the problem. Then use the openh264 with your receiver. I think it'll work.

media_dependencies.video_encoder_factory = std::make_unique<webrtc::VideoEncoderFactoryTemplate<
          webrtc::OpenH264EncoderTemplateAdapter>>();
media_dependencies.video_decoder_factory = std::make_unique<webrtc::VideoDecoderFactoryTemplate<
          webrtc::OpenH264DecoderTemplateAdapter>>();

from raspberrypi_webrtc.

Synchronized Video Streaming with Raspberry PI Zero 2 W about raspberrypi_webrtc HOT 48 OPEN

Comments (48)

About SDP

About ICE

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	OnFrame(webrtc::VideoFrame::Builder()
	.set_video_frame_buffer(dst_buffer)
	.set_rotation(webrtc::kVideoRotation_0)
	.set_timestamp_us(translated_timestamp_us)
	.build());

	auto signal = ([&]() -> std::unique_ptr<SignalingService> {
	#if USE_WEBSOCKET_SIGNALING
	return nullptr;
	#elif USE_SIGNALR_SIGNALING
	return SignalrService::Create(args.signaling_url, conductor);
	#else
	return nullptr;
	#endif
	})();