Comments (5)
I noticed that you guys now use a wrapper script to set ulimit. How many files do you guys read in parallel? I wonder if the correct thing to do is actually to cap the number of http connections we make, since having thousands of http connections is unlikely to be helpful.
It doesn't look like it's possible yet to limit the number of total connections: golang/go#6785
from goofys.
seems better to use a wrapper script for this?
from goofys.
We ran into this on a c4.8xlarge, 36 vCPU instance. The script uses Python's multiprocessing default of one process per cpu and each job is a file, so we would have had 36 S3 files open at once (each job reads sequentially through the file a number of times, mdsum, gunzip md5sum, format checking.
Even with the default ulimit of 1024, I'm kinda surprised we ran into the limit as that works out at 28 per file whereas pool.maxBuffersPerHandle
is only 20. Could the sockets not be being closed properly?
Sorry, I'm not likely to be able to dig into this as I've currently on holiday between changing jobs.
from goofys.
I'm also getting errors on too many open files on s3
Aug 2 18:25:48 ip-172-31-56-75 /usr/bin/goofys_006[1219]: s3.ERROR code=RequestError msg=send request failed, err=Head https://s3.amazonaws.com/grid-public-lidar/Wyoming_BLM/CIR/41104/4110431_SE_13_2015.laz: dial tcp: lookup s3.amazonaws.com on 172.31.0.2:53: dial udp 172.31.0.2:53: socket: too many open files
from goofys.
I've increased open files with fs.file-max = 100000
in sysctl but no improvement
from goofys.
Related Issues (20)
- Can't access subdir from top dir even as a root user
- Deprecated Azure API
- main.ERROR Unable to access '<bucket_name>': permission denied HOT 3
- Feature Request: Support s3 new checksums crc32/crc32c/sha1/sha256
- Goofys work with contabo object-storage HOT 2
- Operation to mounted Goofys S3 does not respond
- ReadDir inode leak causes memory leak
- mount with ceph rgw got error ,ceph version is 18.2.1 HOT 1
- Improvement request: set op.Handle as point to FileHandle/DirHandle
- can not compile with go1.16
- Building leads to [email protected]: retracted by module author
- certificate has expired or is not yet valid HOT 3
- Goofys mount is not reloading inside container
- Disable multipart upload HOT 1
- mounting, listing and reading file works, writing doesn't HOT 1
- In kubernetes when killing pod with goffys container, pod stuck in terminating. exit code 137. HOT 2
- Process exits with error during copy of file under goofys mount point
- Failed deleting files HOT 1
- Any luck building this on mac osx? Brew and go have both moved on HOT 2
- How to mount the Blob/Data Lake containers with different Azure Storage Accounts via /etc/fstab ? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from goofys.