Full throughput limit is not being used about mountpoint-s3 HOT 7 CLOSED

Thanks for the updates. For your workload, you can try spawning threads to read multiple files at the same time. That should get you close to the instance bandwidth. We will also update documentation on our side.

from mountpoint-s3.

Hey @karthik1900, that throughput seems really low. Can you share more about how you read data from the mounted directory? Based on our benchmark results (https://awslabs.github.io/mountpoint-s3/dev/bench/), increasing number of concurrent reads and opening a file in O_DIRECT mode might help.

from mountpoint-s3.

Hey @monthonk, thanks for the quick response

I am using a simple python open file

with open("myfile", "rb") as f:
    data = f.read()

Will try with the O_DIRECT mode and update here

from mountpoint-s3.

When I use fio to to read the file, I am getting about 22 Gb/s, I guess the issue is with my read

fio --name=read --rw=read --bs=4m --direct=1 --filename="myfile"

from mountpoint-s3.

Hey @monthonk

Is there a way where we can set a configuration which will allow multiple threads to read one large file (as s3 allows reading in chunks)

from mountpoint-s3.

Hey, we already do that in the background. In mountpoint, a file system read request could spawn multiple GetObject requests and download object parts in parallel. The bottleneck is at the FUSE interface when you are reading the file. This PR https://github.com/awslabs/mountpoint-s3/pull/834/files explains a bit about that.

from mountpoint-s3.

Hi! We merged that particular documentation change (#834), so we now have a section in troubleshooting slower throughput than expected: https://github.com/awslabs/mountpoint-s3/blob/main/doc/TROUBLESHOOTING.md#slower-throughput-than-expected

If there's any more questions here, please let us know - or we can close the issue. Thanks!

from mountpoint-s3.