SCTP throughput ~8x slower than TCP when the connection has latency about sctp HOT 15 OPEN

With your updated version of go-rudp/sctp-test (passing buffer size into pion/sctp assoc), I see dramatically better throughput.

For the curious:

This is faster than TCP.
To repro these tests, I had to make a small fix to cap conn.SetReadBuffer/SetWriteBuffer to ~5mb for mac (the OS UDP limit), while allowing the sctp buffer to be the specified size

Very excited for the use-cases this unlocks for pion/sctp!

from sctp.

@iamcalledrob Thanks for the detailed report. As I mentioned in #62, a modern TCP implementation automatically adjust receive buffer size, while pion/sctp uses a fixed buffer size, which is currently set to 1024KB. The receive buffer size sets the hard limit for the amount of data inflight (or the un-ack'd segments). It appears the throughput you observed is the result of this limit. I think pion/sctp should implement this auto tuning feature. Let me dig into more information about its algorithm.

from sctp.

@enobufs I'm curious if the commit to exclude the in-flight size is ready to land (or could be!)

I've been testing datachannels with the larger buffer offered through the SettingEngine, but am still seeing very low throughout when there's even a tiny amount of packet loss or latency.

Testing with sctptest over LAN and the network link conditioner looked really positive--near 100% link utilisation.

However in the real world I'm still seeing <20% link utilisation over WiFi or coast-to-coast levels of latency.

For example, I have a TURN server which is 80ms away. iperf with a few TCP streams will get 600+Mbps throughput. PION datachannels with an 8mb buffer will get 20Mbps throughput.

I was hoping the larger buffer would be a decent enough fix by itself in the "real world", but that doesn't seem to be the case!

from sctp.

That indicates there would be another limiting factor. From my quick glance, the sender's congestion window kept growing during the 10 sec period but it appears too slow - smell of a bug... I may be wrong. Let me look at this later more carefully.

from sctp.

After fixing the above in sctptest, I was still seeing the throughput was still capped at 1MB receive buffer size. (increasing the size beyond 1MB had no effect).

sctptest RTT=300ms

From the trance log, I noticed that even though there's more room in the congestion window, the inflightQueue was not fully utilized. Then I realized, this was because sctptest has maxBufferedAmount set to 1MB (threshold being 512KB, both hardcoded), which was effectively limiting the send buffer size to 1MB. This is why the throughput did not grow beyond 1MB of receive buffer size.

There was another related issue. The bufferedAmount in the current pion/sctp includes the number of bytes inflight. Looking at the FireFox's implementation, it does not include the size inflight (the size written to usrsctp).

After fixing the all above, now I see the throughput grows proportionally to the size of buffers as shown below:

Other related metrics:

I will write some unit tests then submit a pull-request.

Once all these fixes land, we should consider whether we should dynamically adjust the receive buffer size.

from sctp.

That's so exciting! Nice work @iamcalledrob and @enobufs :)

We should add this to the SettingEngine today and publish the findings!

from sctp.

@Sean-Der I think it is a brilliant idea to allow changing the send/recv buffer size from the SettingEngine as I think it may take sometime for us to implement auto buffer size tuning.

from sctp.

I found this issue from a few years back which looks potentially related? #62

from sctp.

Fantastic testing @iamcalledrob thanks for reporting!

I don't know the latest but @enobufs mentioned he was looking at adding RACK to the implementation. Maybe some other things exist that we can do in the short term? I defer to him on what the best thing to do is :)

from sctp.

@Sean-Der, @enobufs Thank you both for looking into this!

@enobufs regarding the buffer size, I was curious about that as well and decided to do some more testing!

I benchmarked the effect of changing the buffer size via your sctp-test utility—i.e. sctp-test -network udp -b N. I believe this sets both the read and write buffer sizes to 'N bytes' on the SCTP association as well as the underlying UDP conn.

Here's what I saw with round-trip latency of 300ms

As you can see, the throughput maxes out with a buffer size of 256kb—smaller than the default 1mb! Increasing the buffer size beyond 256kb has no effect.

I'm curious about this—I would have assumed that manually setting a larger buffer would have improved throughput in this particular test, but that has not been the case.

I'm happy to help with getting to the bottom of this, though I have to admit I'm not coming from a place of expertise here so pointers are very helpful :)

I've attached trace logs for both the client and server in case it's helpful context.

client.log
server.log

from sctp.

I just found a bug in sctptest. The value with the -b option for the server does not propagate down to pion/sctp :( .

What's going on is, the client side should have a ssthresh (slow start threshold) which should initially be set to the receive buffer window advertised by the peer. This value did not change even though I was increasing the value with -b, then found this bug. I will fix it shortly.

from sctp.

Following updates have been made:

• go-rudp v0.0.3
• sctp-test v0.0.2

TODO:

• pion/sctp: exclude inflight size from bufferedAmount

from sctp.

Thank you for looking into this! I'm excited to see what the throughput looks like when inflight size is excluded :)

from sctp.

@enobufs

v1.8.2
poor performance than tcp in a tiny amount of packet loss or latency

does v1.8.3 fixed pion/sctp: exclude inflight size from bufferedAmount ?
or improved it .

from sctp.

@Sean-Der Is MaxReceiveBuffer arbitrary or is it supposed to match a UDP socket configuration? What are the tradeoffs if the value is arbitrary?

from sctp.