"Websocket Is Closed Before The Connection Is Established" - Web console error about centrifugo HOT 8 CLOSED

Hello, this could mean that client side timeout was called. You could face ephemeral port problem - https://centrifugal.dev/docs/server/infra_tuning#ephemeral-port-exhaustion . But I can't say for sure.

If you don't observe the same behaviour in local environment – then it's sth infrastructural and most probably network stack related. If you can reproduce this with local Centrifugo – we can take a closer look.

Is the issue related to Centrifugo's configuration or is it just how the Centrifugo JS library works?

In normal situation such errors may happen but should be relatively rare.

from centrifugo.

I should note that the errors are observed while running some kind of synthetic test. It emulates 100 concurrent websocket connections from the client and subscribes to the "presence" channel.

All connections are established eventually, but at some point the aforementioned errors observed. If some delay is added between subscriptions (let's say 300ms) - there is no sign of error in the console.

From the server's side, I can't stress enough how low the load right now. With only one replica of Centrifugo it uses ~ 300 TCP connections when triggering the test. Apart from that, even if I scale to 20 replicas - the picture is the same, but according to the recommendations in the doc scaling should help in cases with port exhaustion.

centrifugo # netstat -tanp | wc -l
309

from centrifugo.

If it's a synthetic test you may hit limitations on client side. Are you opening many connections from a single browser? I suppose the best you can do if you want help with this is to provide reproducer, there are too many things which may be involved.

from centrifugo.

Yes, indeed. I can confirm that the test has been performed from a single browser. Having the TCP dump I observe that some of TCP connections are reset on the client's side.

I may have a better understanding of the situation now. In case the server's websocket connections reset, due to events such as Nginx reload config or rolling restart for Centrifugo containers, the client will attempt to establish new connections. However, it may encounter a limit on the number of concurrent connections allowed by the browser.

I can imagine that on the server's side, the action item is to try to minimize the situations when the connection is reset. What about the client-side? Are there any recommendations on how to handle it correctly?

from centrifugo.

I don't know the real-life valid use case when so many connections should be properly established from one device.

If you need to load test Centrifugo - better using specialized tools outside of browser.

If you have real app when there are so many connections and need to reconnect properly - think about different delays for reconnection (SDKs support such option), as you mentioned it helped you already. But again - sounds rather unusual, I don't understand what you want to achieve/test. Probably consider moving WebSocket connection to Service worker or Shared worker and re-use connection from different tabs.

from centrifugo.

We were trying to reproduce an incident-related case, but as you highlighted we were doing it wrong.

I found a k6 benchmarking test in the project. Would you recommend using it to test the number of clients/ws connections the installation can handle?

from centrifugo.

I found a k6 benchmarking test in the project. Would you recommend using it to test the number of clients/ws connections the installation can handle?

You can, probably adapt it a bit for your needs. Note that you can still hit limitations of client when running benchmark using k6 - like local open file limits, local resource usage. BTW, ChatGPT writes k6 scripts pretty well - was using it to write this script when was testing sth.

from centrifugo.

Closing, the main question was addressed.

from centrifugo.