100% Utilisation (lots of sendmsg) about rtpengine HOT 13 CLOSED

You wouldn't happen to have a core dump from this, would you?

Is the address (178.62.XXX.XXX) the local address of the proxy itself by any chance?

from rtpengine.

Hi rfuchs,

I am sorry I don't have a core dump, the program did not crash just max utilisation (and no longer responding to any requests), (I am not sure how to extract one from a running program / if possible).

The 178.62.XXX.XXX is the IPv4 internet address of the proxy, yes.

from rtpengine.

I guess the proxy ended up talking to itself in a loop then. Perhaps an intentional DoS by one of your users? You should be able to find details about who or when in the log. Look for offers and answers advertising the proxy's own IP address

from rtpengine.

We are running a multi-tenant platform currently on standard RTP Proxy, it would be theoretically possible for one customer to send a call through another customer on our platform. It is possible for the same server to be engaged twice. With standard RTP Proxy this has caused no problems.

Is this what you are seeing here?

from rtpengine.

No, not really. The proxy talking to itself isn't a problem per se, this should work as expected. What I mean is the proxy talking to itself in a loop. For example, the proxy sends packets arriving on port X to itself on port Y, and sends packets arriving on port Y to itself on port X. Something like that.

from rtpengine.

Would this not mean that the potential attacker has access to my control ports, this should not be possible as these are fire-walled.

from rtpengine.

No, this could be done through signalling, ie within the SDP body. But deliberate DoS is just a guess, you should really look in your logs and try to identify the call in question. The local ports involved were 31930 and 31934, you should be able to find those in the log.

from rtpengine.

Hi rfuchs,

I have tracked this down and I can see the two commands, comparing this to other lines in the log file, it does not look unusual, it appears that 2 distinct external IP addresses were provided (which upon further checking are my customer and provider).

However if your suspicion is correct (and I have mis-read the data), how can this be prevented, as surely this is a security vulnerability.

from rtpengine.

It would help to know how exactly this came to be, but without additional details I'm afraid that we'll never know for sure.

Detecting these kind of loops (whether caused intentionally or by accident) is tricky. Obviously you can't just keep the proxy from sending packets to itself, as this is required in certain scenarios. I'll try to come up with something.

from rtpengine.

So this is a somewhat crude and experimental attempt to catch forwarding loops, but it seems to do the trick in my tests.

from rtpengine.

I have returned to my rtpengine servers this morning locked up with 100% utilisation and observed the following errors in the log:

Aug 31 03:46:54 rtp-lon-3 rtpengine[10953]: [[email protected] port 38296] Too many packets in UDP receive queue (more than 50), aborting loop. Dropped packets possible
Aug 31 03:46:54 rtp-lon-3 rtpengine[10953]: [[email protected] port 42782] Too many packets in UDP receive queue (more than 50), aborting loop. Dropped packets possible
Aug 31 03:46:54 rtp-lon-3 rtpengine[10953]: [[email protected] port 30426] Too many packets in UDP receive queue (more than 50), aborting loop. Dropped packets possible
Aug 31 03:46:54 rtp-lon-3 rtpengine[10953]: [[email protected] port 29462] Too many packets in UDP receive queue (more than 50), aborting loop. Dropped packets possible
Aug 31 03:46:54 rtp-lon-3 rtpengine[10953]: [[email protected] port 38296] Too many packets in UDP receive queue (more than 50), aborting loop. Dropped packets possible

This has continues for many hours through the night.

The version of the compiles binaries is 3.3.0.0+0~mr3.5.0.0 git-master-c0f8196

from rtpengine.

In addition I have just noticed that the lo interface has

RX bytes:1014751848427 (1.0 TB)  TX bytes:1014751848427 (1.0 TB)

from rtpengine.

So your proxy is still sending packets to itself. Can you select one of the call-ids that you posted, and then post the complete rtpengine log for that call please.

from rtpengine.