Option to increase ulimit -n? about goofys HOT 5 OPEN

I noticed that you guys now use a wrapper script to set ulimit. How many files do you guys read in parallel? I wonder if the correct thing to do is actually to cap the number of http connections we make, since having thousands of http connections is unlikely to be helpful.

It doesn't look like it's possible yet to limit the number of total connections: golang/go#6785

from goofys.

seems better to use a wrapper script for this?

from goofys.

We ran into this on a c4.8xlarge, 36 vCPU instance. The script uses Python's multiprocessing default of one process per cpu and each job is a file, so we would have had 36 S3 files open at once (each job reads sequentially through the file a number of times, mdsum, gunzip md5sum, format checking.

Even with the default ulimit of 1024, I'm kinda surprised we ran into the limit as that works out at 28 per file whereas pool.maxBuffersPerHandle is only 20. Could the sockets not be being closed properly?

Sorry, I'm not likely to be able to dig into this as I've currently on holiday between changing jobs.

from goofys.

I'm also getting errors on too many open files on s3

Aug  2 18:25:48 ip-172-31-56-75 /usr/bin/goofys_006[1219]: s3.ERROR code=RequestError msg=send request failed, err=Head https://s3.amazonaws.com/grid-public-lidar/Wyoming_BLM/CIR/41104/4110431_SE_13_2015.laz: dial tcp: lookup s3.amazonaws.com on 172.31.0.2:53: dial udp 172.31.0.2:53: socket: too many open files

from goofys.

I've increased open files with fs.file-max = 100000 in sysctl but no improvement

from goofys.