Nodes failing to advertise due to socket leaks... about nodemanager HOT 5 CLOSED

Author: jeffra45
The beta node with IP 200.0.206.203 is having this same issue.
1k+ sockets open via netstat -n all, except 1, in the CLOSE_WAIT status.

from nodemanager.

Author: justinc
I committed r4071 to add a little more logging to help to find this issue. I believe it shouldn't be too chatty for production use.

from nodemanager.

Author: justinc
armon says:

It is the "slow responder" problem. I can open a socket to the NM, and never write to it,
effectively blocking the Worker Thread indefinitely. Then, I can just create more and more
incoming connections until I finally exhaust all the file handles.

I've attached a test file that replicates this problem.

There are a number of solutions, it is kinda up to you as to what you think is the cleanest.
You can either:

• Modify session to call willblock() to avoid blocking a recv indefinitely
• Wrap all the sockets NM works with in a timeout_socket

Other measures I would take:

• Bound the max size of the connection queue, should never get > 100
• Make the Worker Thread heartbeat every once in a while, have the main thread check the heartbeat

from nodemanager.

Author: justinc
Hmm, I'm convinced you found 'a' bug, but I'm not sure this is 'the'
bug. From what I've seen, when running your tests, this doesn't
result in sockets in the CLOSE_WAIT state. The sockets you open do
all get cleaned up by the OS after a few minutes.

I am testing this by running the node manager on a PL node and the
test program on my local machine.

I'm starting to look at fixing the bug you found now...

from nodemanager.

Author: justinc
Fixed in r4108. Lots of new socket weirdness was discovered.

Big thanks to Armon for his help on this!

from nodemanager.