Comments (53)
As mentioned in the first idea, I vote for Recording streams too.
from mediasoup.
We've been eagerly tracking this work and considering it for use in the future.
Here are a few requirements that we have for our existing SFU (I haven't compared them to what you've built here):
- Data channel support for sending high frequency conference specific data.
- Stream level control for managing individual streams for a bundled peer connection.
- Conference/peer stats for monitoring and health check purposes.
- Trickle Ice for quick peer connection setup times (very important to our use-case.)
While it's a bit more touchy, I'd love to see your Javascript API written in Typescript. It is so great that it even brings Microsoft and Google together.
You can still export an ES6 API with no types for your hardcore JS users that install the package, and you can automatically provide the benefit of enhanced code completion and type safety to Typescript users that consume your library.
I'd be happy to contribute to any Typescript efforts if you are so inclined.
from mediasoup.
Not sure if it's the philosophy of mediasoup: ability to record on server all the streams of a room ? (for playing them after)
from mediasoup.
from mediasoup.
@ibc thanks for the information. After a little RTFM I see that you opt out of the higher level things like signaling. I think that is a really nice decision.
I understand the hesitation with Typescript. There are some lower impact options like crafting a single mediasoup.d.ts
file that describes your library API types. They're easy to produce provided you have a simple API (which you do) and would be easy to maintain. I'll bring it up again after 1.0.0
It seems that with mediasoup will provide most of what we need once you support data channels. I'll stay tuned. Thanks for your effort. 👍
from mediasoup.
After a little RTFM I see that you opt out of the higher level things like signaling. I think that is a really nice decision.
mediasoup is not a "SFU built on Node.js" but a "SFU for Node.js" :)
There are some lower impact options like crafting a single mediasoup.d.ts file that describes your library API types. They're easy to produce provided you have a simple API (which you do) and would be easy to maintain. I'll bring it up again after 1.0.0
Great!
from mediasoup.
What are your thoughts on audio mixing in order to connect to a softswitch like Freeswitch or Asterisk?
Honestly zero thoughts. mediasoup won't mix audio/video streams just to make legacy SIP endpoints happy. This is a SFU. Legacy telephony support is not in the feature list.
from mediasoup.
but receiving all streams gives us this ability. Same goes for detecting who the active speaker is.
That's a (good) use case, but not _The Use Case_©️.
Now imagine a social multiconference application in which anyone can see the video of all the others, but just those who pay some money are able to send and receive the audio of other participants. You don't want to mute the audio in client side.
from mediasoup.
http://mediasoup.org/faq/#is-mediasoup-a-native-addon
- I don't depend on internals of Node.js or V8 (which are not stable as your previous link clearly shows).
- I can run many workers in parallel.
- In the future, if Node.js disappears (ough) I could connect mediasoup-worker to any other high level runtime.
from mediasoup.
there is still no example to test mediasoup, can you add one?
from mediasoup.
@mccob the API already allows handling received RTP packets in JavaScript land:
Whether having the raw RTP packets in JS is enough to playback them later is another topic that should be done by another library. But yes, I will work on that.
@notedit Usage examples will be properly written once the API is finished. But now I'm asking for "features" :)
Thanks a lot.
from mediasoup.
@justindujardin thanks for your feedback. Comments:
Data channel support for sending high frequency conference specific data.
DataChannel will be supported in the next version (after 1.0.0
). In fact, I plan two modes:
- DC messages are directly routed to every room participant with DC support.
- DC messages reach the application JavaScript land, so the app can handle them and decide what to do with them.
Stream level control for managing individual streams for a bundled peer connection.
I don't fully understand what you mean with "stream level control", but take into account that BUNDLE is a SDP artifact because, originally, SDP did not allow two media streams over the same transport.
ORTC, and hence mediasoup, lets the app create as many Transports as desired, and attach them to RtpReceivers also as desired.
Conference/peer stats for monitoring and health check purposes.
Yep, there will be some kind of API for that.
Trickle Ice for quick peer connection setup times (very important to our use-case.)
mediasoup is not SDP based so talking about "Trickle ICE" has no sense here. You create a Transport for a remote Peer and retrieve its local ICE candidates. How you pass them to the remote Peer (browser) is up to you.
Regarding TypeScript, I don't want to force all the library users to deal with TypeScript. But I'm not against providing an extra TypeScript layer (probably within a separate library that would act as a mediasoup TypeScript wrapper). Let's continue with this topic once 1.0.0 is out :)
from mediasoup.
What are your thoughts on audio mixing in order to connect to a softswitch like Freeswitch or Asterisk?
As far as I know they don't support the transport of multiple audio streams on the same peerconnection yet. (Correct me if that's wrong)
from mediasoup.
Some ideas
- Provide client SDK (js / Android / iOS) should simplify adoption of mediasoup
- Provide a plugin system
-> a video recorder plugin will be nice
-> a computer vision plugin based on OpenCV (face recognition, sentiment analysis, image overlays, ...)
from mediasoup.
a computer vision plugin based on OpenCV (face recognition, sentiment analysis, image overlays, ...)
This one would require that MediaSoup decodes the video and analyzes it, which can get really CPU intensive. I'm not sure what the long term goals are, but in the short term @ibc wants to build a SFU, that means no decoding of the video... Maybe it could be added later on though or maybe find a way to integrate it nicely with Kurento which already does some of that IIRC.
from mediasoup.
Provide client SDK (js / Android / iOS) should simplify adoption of mediasoup
This won't happen. mediasoup is just a "piece" of the puzzle. It does not provide a network signaling protocol, so it makes no sense to provide a client SDK. Also, mediasoup is just about WebRTC (media plane). Clients don't need special stuff other than a proper usage of the WebRTC API.
"Products" based on mediasoup may provide a client SDK, but mediasoup itself is "just" a server side component.
a video recorder plugin will be nice -> a computer vision plugin based on OpenCV (face recognition, sentiment analysis, image overlays, ...)
RTP recording is the the readmap (in fact, RTP packets can already be reached at JS layer, so the app can store/process them as desired).
from mediasoup.
Maybe stream manipulation (like for Admin mode). to mute/unmute audio/video streams, kick also, on API requests
from mediasoup.
You mean for avoiding the forwarding of RTP? This could be done in the client, since it has access to all streams.
from mediasoup.
Maybe stream manipulation (like for Admin mode). to mute/unmute audio/video streams, kick also, on API requests
There will be a very simple API to pause/resume streams, on both the receiver and on each sende, something like:
rtpReceiver.pause()
rtpReceiver.resume()
rtpSender.pause()
rtpSender.resume()
There is no "admin mode". The "administrator" is the server developer/application. He decides what mediasoup does.
@saghul said:
You mean for avoiding the forwarding of RTP? This could be done in the client, since it has access to all streams.
Certainly there is good rationale to do it via API in the server rather than relying on each client.
from mediasoup.
Certainly there is good rationale to do it via API in the server rather than relying on each client.
I don't disagree. However, I found that many of the traditional admin-like things get simplified in a SFU scenario. Take for example muting a participant which is making noises. Instead of administratively muting them, each connected client can do it. It doesn't mean there shouldn't be another way to do things, but receiving all streams gives us this ability. Same goes for detecting who the active speaker is.
from mediasoup.
@saghul remember that this is not just about boring enterprise meetings
from mediasoup.
@ibc Sure, that's indeed a great counter example! 👍
remember that this is not just about boring enterprise meetings
🕴 🕴 🕴
from mediasoup.
Would https://github.com/nodejs/abi-stable-node affect your design decision to have subprocesses which handle the media layer?
from mediasoup.
Would https://github.com/nodejs/abi-stable-node affect your design decision to have subprocesses which handle the media layer?
Why? mediasoup is not a Node native module.
from mediasoup.
What is the reason to have separate worker processes as opposed to buliding https://github.com/ibc/mediasoup/tree/master/worker/src into a native module?
from mediasoup.
Nice! Also, if means you could run certain workers at a higher priority (say if certain users pay more in the application, just for example).
from mediasoup.
Excuse me, is The Real Time Streaming Protocol a part of mediasoup?
from mediasoup.
Excuse me, is The Real Time Streaming Protocol a part of mediasoup?
For now not.
from mediasoup.
i am thinking some kind video file streaming or rtp input api.
so we can custom the input media source.
from mediasoup.
i am thinking some kind video file streaming or rtp input api so we can custom the input media source.
Definitely yes. That will be properly implemented.
from mediasoup.
support "Unified Plan" for multistreams when it is completed.
https://bugs.chromium.org/p/chromium/issues/detail?id=465349
from mediasoup.
@notedit mediasoup already supports Unified Plan.
from mediasoup.
Provide a kind of "requestFullFrame" in video rtpReceiver API to send FIR RTCP request to browser.
Use case:
We implement SFU with recording of streams by capturing and storing rawrtp packets into file.
Currently if stream without lots of video changes, we have to wait too long for next full intra frame generated by browser.
By having this kind of API, there will be less waiting time between video recording command received and first frame actually recorded.
Another option is to have full control of sending any RTCP packet from node.js side, but this seems to be more complex task.
from mediasoup.
@kolychev mediasoup really support this . https://groups.google.com/forum/#!topic/mediasoup/XrLFm61ndcE
from mediasoup.
Not exactly. Indeed mediasoup sends a PLI to he corresponding browser/endpoint when an event listener is set for the rtpraw
or rtpobject
event. However is not possible to request sending a PLI via API.
We are working actively on mediasoup 2.0, so will incorporate such a feature in there.
from mediasoup.
+1 to datachannel support.
What I'd really like to see is signaling done via regular old HTTPS (or whatever), and then the room messages handled entirely over in-band data channel. The big issue with having room/stream management happen over the signaling channel is it creates ambiguity when one fails but the other doesn't.
The more I work with SFUs, the more I think it's extremely important to minimize the number of connections established to the media server (i.e. multiplex as much as possible, so bundle/rtcp-mux/datachannels for room signaling, etc.)
from mediasoup.
mediasoup v1 does not provide a signaling mechanism. It's up to the application to decide how to transmit SDPs between browsers and mediasoup.
mediasoup v2 comes with a mediasoup-client SDK library that, again, does not provide a signaling protocol (HTTP, WebSocket or whatever) but just emits events with message payloads that the client app must transmit to the server and vice-versa. You can see an example usage in the README: https://github.com/versatica/mediasoup-client
from mediasoup.
@ibc I'm aware that mediasoup does not come with a signaling mechanism. I'm also aware that mediasoup does not have data channel support. Lack of data channel support precludes the architecture I described in my last post, which is why that feature is important to me.
from mediasoup.
Also, to be clear, I like that media soup doesn't have a signaling layer. But my beef is most MCU/SFU implementations recycle the signaling layer for stuff that should be multiplexed over a data channel (e.g. join room, room events, application messaging, etc.)
Example: licode uses socket-io for both signaling and room/application events.
from mediasoup.
@jnoring still I don't get your point. If licode uses socket.io that means that licode provides a signaling layer. mediasoup does not. But you can use socket.io for both, sending mediasoup API requests/responses/notifications (in mediasoup v2) and sending your custom app messages (such as chat ot whatever you wish).
from mediasoup.
@ibc the whole point is not to recycle a signaling layer for "custom app messages", because that creates duplicate persistent connections. In any event, it isn't important--what's important is data channel support.
from mediasoup.
the whole point is not to recycle a signaling layer for "custom app messages"
In mediasoup v2 it would be the other way around: "recycle the app custom signaling layer for mediasoup API message".
because that creates duplicate persistent connections
If there is a single signaling layer (up to the app) why would it create duplicate persisten connections?
In any event, it isn't important--what's important is data channel support.
It will be implemented when possible. Currently the only SCTP implementation in C (usrsctp) is multi-thread, so no valid within the mediasoup design (which is multi-process but not multi-thread).
from mediasoup.
Please @chetanbnaik, use the mailing list for questions and not a random issue in Github.
from mediasoup.
Ability to chain multiple SFU instances for oneway realtime streaming (low-latency broadcast).
PS: Did not see any new call for features after 2.0.0, hoping this is still a right place to ask for features.
from mediasoup.
Ability to chain multiple SFU instances for oneway realtime streaming (low-latency broadcast).
That's in the radar, but will require some time.
from mediasoup.
Cool looking forward to it.
from mediasoup.
Is there any timelines set (or version) for chaining SFU? Don't mean to demand or push for it just curious.
from mediasoup.
Chaining SFUs is in the radar but not yet planned. It's a MUST and will be done eventually.
from mediasoup.
Thank you so much for your prompt response
from mediasoup.
Just found this other webrtc library in python that allows live video frame manipulation https://github.com/jlaine/aiortc#aiortc
from mediasoup.
Maybe stream manipulation (like for Admin mode). to mute/unmute audio/video streams, kick also, on API requests
There will be a very simple API to pause/resume streams, on both the receiver and on each sende, something like:
rtpReceiver.pause()
rtpReceiver.resume()
rtpSender.pause()
rtpSender.resume()
There is no "admin mode". The "administrator" is the server developer/application. He decides what mediasoup does.
Hello, is it implemented?
Can I mute other user?
I saw there is roomClient.muteMicro(), is there the same function for mute other peers exept of me?
Also i saw audioMuted which depend of state.me, i think that i can set this props for my needs, but may be there is a better solution?
I mean mediasoup-demo
from mediasoup.
I mean mediasoup-demo
This is a "call for feature requests" for mediasoup, and not for mediasoup-demo.
In mediasoup you can do all you asked above. No, I won't implement them in the demo. You can do it in your application.
from mediasoup.
Chaining SFUs is in the radar but not yet planned. It's a MUST and will be done eventually.
Hi,
Is what described in this link, actually the documentation regarding Chaining SFUs? If not, can you please direct me to the corresponding documentation around it?
P.S.
Thank you for such a great library.
from mediasoup.
Related Issues (20)
- Worker crash when closing `WebRtcServer` if there are TCP connections active HOT 2
- Mediasoup worker died, exiting in 2 seconds... HOT 2
- [Rust] Error log “XXX closing failed on drop: Channel already closed” HOT 2
- Add interface for RTC::Shared, Channel::ChannelNotifier and ChannelMessageRegistrator HOT 3
- [Rust] Transport with only audio producer results in server-side transport cc is not created. HOT 2
- Bug Simulcast plainTransport -> webrtcTransport HOT 5
- Video software encoding Quality Limitation vs hardware encoding HOT 1
- Ci tests fail consistently in Windows (Node and Rust) HOT 10
- Failed to run custom build command for `mediasoup-sys v0.9.0` HOT 6
- H264 temporal layers no longer detected due to removal of frame-marking RTP extension in libwebrtc (must implement dependency descriptor RTP extension) HOT 6
- AddressSanitizer issues HOT 6
- Couldn't run on ARM64 platform. HOT 3
- consumer#resume/pause not working as it should HOT 2
- worker process silently exits if TcpConnectionHandle::SetPeerAddress() fails HOT 2
- Simulcast frozen video due to dropping RTP packets HOT 4
- the "PacketRouter* packet_router_" and "PacedSender pacer_" is different class, but the pacer_ is assigned by the "packet_router_".
- Update ESLint and its ecosystem
- Docker and io_uring don't work together - add prebuilt binary without liburing HOT 16
- libsrtp: wraparound with loss decryption failure HOT 2
- Include ubuntu-24.04 in CI HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mediasoup.