fcntl: too many open files about s5cmd HOT 4 CLOSED

Hey @tommyblue, thanks for filing an issue.

~$ ulimit -H -n
unlimited

~$ ulimit -S -n
256

~$ launchctl limit maxfiles
maxfiles 10240 10240

In addition to that, could you share the output of ulimit -n please.

s5cmd walks given <src> directory and sends each file to an internal queue, where workers receive from. By default, 256 workers are up and running and each one tries to open the file they received from the queue. The limit here is your OSs open file descriptor limit. I suspect your open file descriptor limit is around 256.

Increasing the limit to a high value should work.

from s5cmd.

ulimit -n => 256

from s5cmd.

Your diagnosis and proposed solution are certainly correct, but I think the behaviour of s5cmd under this situation is misleading. Moreover the error message is tagged "VERBOSE" so it's probably hidden without the -vv flag.

Some possible solutions:

• Show this kind of errors also without the verbose flag
• check ulimit against the number of configured workers at startup and notify the user if the values are incompatible
• catch the error and show a suggestion at the end of the output. Something like: "s5cmd found that the configured ulimit is too low for the number of worker you're using. Either increase the ulimit value or decrease the number of parallel workers"

from s5cmd.

@tommyblue You're right. The first one is definitely needed. If there is a non-retriable error, it needs to be shown without a verbose flag.

Others are good suggestions also. Let's keep the issue open till we've a fix.

Thanks again.

from s5cmd.