Files unexpectedly released leading to `Bad file descriptor` about mountpoint-s3 HOT 5 CLOSED

Thanks for the details on this, @jchorl. We'll look into this soon.

Quickly linking to another comment we saw a similar symptom in another issue: #706 (comment)

from mountpoint-s3.

Thanks again for the details on this report.
We're working on a fix for this and hope to share more details soon.

The regression was introduced in 0030b0a and it is indeed the flush triggered when closing one of the file descriptors that causes the issue.

We have two simple reproductions (thanks @jamesbornholt and @passaro):

One in Python...

import os
import sys

# open the file and duplicate the file descriptor
path = sys.argv[1]
fd = os.open(path, os.O_RDONLY)
fd2 = os.dup(fd)

# read from the first file descriptor
b = os.read(fd, 10)
print(f"read from fd: {b}")

# close the first file descriptor -- triggers a fuse FLUSH
os.close(fd)

# now try reading from the second file descriptor
os.lseek(fd2, 100000, os.SEEK_SET)
b = os.read(fd2, 10)
print(f"read from fd2: {b}")
os.close(fd2)

And the other in Rust as a new integration test to be added: https://github.com/passaro/mountpoint-s3/blob/17c245dbd527815be6659232af97765c7cee07ea/mountpoint-s3/tests/fuse_tests/read_test.rs#L313-L335

from mountpoint-s3.

Thank you for the speedy reproduction and fix.

I just ran a test that was repeatedly failing and it passed, so I think we're all good here. Thanks again!

from mountpoint-s3.

Thanks @dannycjones, just a +1 in that we are also observing this. I was unable to reproduce it succinctly so kudos to @jchorl. The files in question were also bash scripts and our workload is also very heavy. I circumvented the issue in our case by copying the files from S3 (via mountpoint) to EBS such that I wouldn't have to read them from mountpoint during execution but this remains an open issue for us.

from mountpoint-s3.

We've released v1.4.1 which contains a fix for this: https://github.com/awslabs/mountpoint-s3/releases/tag/mountpoint-s3-1.4.1

Please let us know @jchorl if you're still seeing the issue, otherwise I hope we can resolve it.

from mountpoint-s3.