Comments (6)
Hi, thanks for opening this issue. This is expected, as for Mountpoint when you do a list we have to paginate and do multiple listObject requests, getting 1000 objects in each request which can lead to this latency. Using the date as a prefix should help reduce time for listing though, as it will mean that Mountpoint can complete the listing with fewer requests.
from mountpoint-s3.
@ahmarsuhail I have a "folder" with over 96882 "files" and listing takes. I compared different s3 mount implementations s3fs-fuse - about 5 minutes goofys - 35 seconds mountpoint-s3 45 minutes!!!
There's obviously something wrong with the listing implementation in mountpoint-s3
Hey @VladimirPchelko, please take a look at the troubleshooting section for this scenario: https://github.com/awslabs/mountpoint-s3/blob/main/doc/TROUBLESHOOTING.md#slow-metadata-operations
Likely, metadata caching can help you: https://github.com/awslabs/mountpoint-s3/blob/main/doc/CONFIGURATION.md#metadata-cache
If you're still experiencing issues, please do open a new issue and we can take a look. https://github.com/awslabs/mountpoint-s3/issues/new?assignees=&labels=bug&projects=&template=bug-report.yml
from mountpoint-s3.
@ahmarsuhail
I have a "folder" with over 96882 "files" and listing takes.
I compared different s3 mount implementations
s3fs-fuse - about 5 minutes
goofys - 35 seconds
mountpoint-s3 45 minutes!!!
There's obviously something wrong with the listing implementation in mountpoint-s3
from mountpoint-s3.
@dannycjones
I need that when deploying a container there are no delays on the first request! Do you offer caching? What's the point?
I'll check, but I seriously doubt it... wait 45 minutes again.
from mountpoint-s3.
after enabling caching
real 1m27.568s
but still slower than goofys
ok
from mountpoint-s3.
@VladimirPchelko we'll be happy to look into the performance issue you're facing, but we will need more information.
Details about the size of each object, and the structure of the S3 bucket would be helpful. For example, how deeply nested are the objects within prefixes - a/file.txt
or a/b/c/d/e/file.txt
.
Please do open a new bug report to describe the issue and share some logs, and we can investigate.
from mountpoint-s3.
Related Issues (20)
- Support ovhcloud s3 high performance object storage HOT 1
- Support for writing to Google Cloud Storage buckets HOT 12
- I think there should be a cool icon for this product HOT 1
- ERROR awscrt::AuthCredentialsProvider: static: invalid credential_source property: EcsContainer HOT 4
- Assuming role configured in AWS profile does not work outside of `aws` partition (China, US Gov Cloud, etc.) HOT 5
- Regression between 1.4.0 and 1.6.0 when overwriting files with --allow-overwrite enabled HOT 10
- Can't open & append file with path
- Clarify memory usage limiting in case of `--metadata-ttl` enabled HOT 2
- Support executable files HOT 2
- > mountpoint_s3::fuse: open failed: inode error: inode 3 (full key "synmon/test_file.txt") is not writable while being read HOT 1
- Use mount-s3 as a crate dependency HOT 1
- Debug/trace logging that includes the HTTP request headers HOT 3
- support for s3 mount on ecs variant of bottle rocket HOT 5
- support `AWS_ENDPOINT_URL` env variable HOT 2
- Mountpoint-s3 with minimal IAM Permissions HOT 1
- On-premises Direct Connect Integration HOT 2
- mount-s3 doesn't work on arm version on copy or writing (e.g. t4g.micro) HOT 3
- Specifying KMS Key ID (not ARN) or KMS Alias for upload causes Mountpoint to crash HOT 6
- Mountpoint doesn't work with IAM Roles Anywhere on non-EC2 instance HOT 3
- S3 Gateway: extend signature types supported for sigv4a and streaming
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mountpoint-s3.