Comments (10)
ayushr2
commented on June 16, 2024
3
mountpoint_s3::fuse: open failed: inode error: inode 3 (full key "synmon/test_file.txt") is not writable while being read
The issue is that gVisor's gofer client (pkg/fsimpl/gofer/) opens the file read-only first and then opens it for writing, but mountpoint_s3::fuse refuses to support that. The gofer client first caches the dentry for the file being opened. It holds a "control FD" for all the dentries it caches. This control FD is opened with O_RDONLY|O_NONBLOCK:
https://github.com/google/gvisor/blob/f67e10cf3866fe3a809c217ae0f47a9c47854ce1/pkg/sentry/fsimpl/gofer/directfs_dentry.go#L41-L50
I think we should change that logic to open regular files with O_PATH only. Directories and FIFOs can continue to be opened with O_RDONLY.
from mountpoint-s3.
dannycjones
commented on June 16, 2024
1
Hey @thundergolfer, thank you for opening the bug report.
The first thing I'd note is that v1.4.0 is recalled, and we would not recommend using it.
Regarding the error message you received: v1.4.0 introduced file overwrites. One downside with the recall is that the behavior you were able to achieve - opening a file for both reading and writing simultaneously - is reverted with v1.4.1+ and is not permitted. We don't plan to allow that behavior to occur, although if that's important for you then I'd recommend opening a feature request for it so we can learn about your use case.
That being said, I was not able to reproduce the issue you are experiencing with v1.6.0. I was able to run this code fine:
# using Python 3.8.19
from pathlib import Path
TEST_FILE_PATH: Path = Path("/mnt/s3/test.txt")
for i in range(50):
TEST_FILE_PATH.write_text(f"Writing text for iteration {i}")I note in your example that the simple reproduction only appears to write to the file. Is that script definitely able to reproduce the issue for you? Could there be another application that has this file open for reading?
from mountpoint-s3.
luiscape
commented on June 16, 2024
1
Hi @dannycjones, adding context regarding
bug where listing the root of the mount twice failed
We filed an issue here and solved here. 1.6.0 includes a fix for that issue that's why we upgraded.
from mountpoint-s3.
jamesbornholt
commented on June 16, 2024
1
Yeah, the reason we forbid this is that S3 doesn't allow reading from an inflight multi-part upload. So if you try reading concurrently, you'll observe the results of the new writes as long as they're still in the page cache, but then start seeing the old object again if they ever get evicted or dropped from cache.
from mountpoint-s3.
monthonk
commented on June 16, 2024
1
Glad that you have a path forward. I'm closing this issue as it's expected behavior in mountpoint.
from mountpoint-s3.
dannycjones
commented on June 16, 2024
We upgraded to 1.6.0 because 1.4.0 had a bug where listing the root of the mount twice failed.
If this is an issue, please do open a bug report for it with some reproduction steps.
from mountpoint-s3.
thundergolfer
commented on June 16, 2024
Thanks for the response @dannycjones. Interesting that you can't repro. Prompted me to check if it was due to container runtime and it is!
If we run the program with runc it works. With runsc (gVisor) it fails.
Appendix
This is the entirety of the Modal program for posterity.
from pathlib import Path
from uuid import uuid4
import modal
MOUNT_PATH: Path = Path("/mnt/s3_bucket")
S3_BUCKET_TEST_PATH: Path = MOUNT_PATH / "synmon/test_file.txt"
app = modal.App(
volumes={
MOUNT_PATH: modal.CloudBucketMount(
"modal-s3mount-test-bucket",
secret=modal.Secret.from_name("cloud-bucket-mount-secret", environment_name="main"),
)
},
image=modal.Image.debian_slim(python_version="3.11"),
)
@app.function()
def write() -> str:
data = uuid4().hex
S3_BUCKET_TEST_PATH.write_text(data)
return data
if __name__ == "__main__":
with app.run():
write_data = write.remote()from mountpoint-s3.
thundergolfer
commented on June 16, 2024
@dannycjones I've done some more investigation into the behavior difference between gVisor and runc.
A gVisor trace of the reproduction program shows that it's only make 1 openat syscall with the O_WRONLY (write) flag.
I0503 02:35:52.374231 2369120 strace.go:570] [ 2: 2] python E openat(AT_FDCWD /root, 0x7edd9f5aa120 /mnt/s3_bucket/synmon/test_file2.txt, O_WRONLY|O_CLOEXEC|O_CREAT|O_TRUNC, 0o666)
I0503 02:35:52.464482 2369120 strace.go:608] [ 2: 2] python X openat(AT_FDCWD /root, 0x7edd9f5aa120 /mnt/s3_bucket/synmon/test_file2.txt, O_WRONLY|O_CLOEXEC|O_CREAT|O_TRUNC, 0o666) = 0 (0x0) errno=1 (operation not permitted) (90.236869ms)
From mount-s3's debug logs though, I can see:
FUSE( 2) ino 0x0000000000000000 INIT kernel ABI 7.34, capabilities 0x33fffffb, max readahead 131072FUSE( 4) ino 0x0000000000000001 GETATTRFUSE( 6) ino 0x0000000000000001 STATFSFUSE( 8) ino 0x0000000000000001 OPENDIR flags 0x28800FUSE( 10) ino 0x0000000000000001 RELEASEDIR fh FileHandle(1), flags 0x28800, flush false, lock owner NoneFUSE( 12) ino 0x0000000000000001 OPENDIR flags 0x28800FUSE( 14) ino 0x0000000000000001 OPENDIR flags 0x28800FUSE( 16) ino 0x0000000000000001 LOOKUP name "synmon"FUSE( 18) ino 0x0000000000000002 OPENDIR flags 0x28800FUSE( 20) ino 0x0000000000000002 LOOKUP name "test_file2.txt"FUSE( 22) ino 0x0000000000000003 OPEN flags 0x28800(open read-only)FUSE( 24) ino 0x0000000000000003 OPEN flags 0x28201(open write-only)- 😖 failed open
Full logs snippet (gvisor)
2024-05-03T01:41:27.236722Z DEBUG lookup{req=16 ino=1 name="synmon"}:list_objects{id=2 bucket="modal-s3mount-test-bucket" continued=false delimiter="/" max_keys="1" prefix="synmon/"}: mountpoint_s3_client::s3_crt_client: meta request finished duration=33.451751ms
2024-05-03T01:41:27.237142Z DEBUG fuser::request: FUSE( 18) ino 0x0000000000000002 OPENDIR flags 0x28800
2024-05-03T01:41:27.237215Z DEBUG fuser::request: FUSE( 20) ino 0x0000000000000002 LOOKUP name "test_file2.txt"
2024-05-03T01:41:27.238657Z DEBUG lookup{req=20 ino=2 name="test_file2.txt"}:head_object{id=3 bucket="modal-s3mount-test-bucket" key="synmon/test_file2.txt"}: mountpoint_s3_client::s3_crt_client::head_object: new request
2024-05-03T01:41:27.240146Z DEBUG lookup{req=20 ino=2 name="test_file2.txt"}:list_objects{id=4 bucket="modal-s3mount-test-bucket" continued=false delimiter="/" max_keys="1" prefix="synmon/test_file2.txt/"}: mountpoint_s3_client::s3_crt_client::list_objects: new request
2024-05-03T01:41:27.250483Z DEBUG lookup{req=20 ino=2 name="test_file2.txt"}:head_object{id=3 bucket="modal-s3mount-test-bucket" key="synmon/test_file2.txt"}: mountpoint_s3_client::s3_crt_client: S3 request finished request_type=Default crt_error=None http_status=200 range=None duration=11.75266ms ttfb=Some(11.564753ms) request_id=BW8YM2QQETXYRDX8
2024-05-03T01:41:27.250533Z DEBUG lookup{req=20 ino=2 name="test_file2.txt"}:head_object{id=3 bucket="modal-s3mount-test-bucket" key="synmon/test_file2.txt"}: mountpoint_s3_client::s3_crt_client: meta request finished duration=11.862071ms
2024-05-03T01:41:27.267847Z DEBUG lookup{req=20 ino=2 name="test_file2.txt"}:list_objects{id=4 bucket="modal-s3mount-test-bucket" continued=false delimiter="/" max_keys="1" prefix="synmon/test_file2.txt/"}: mountpoint_s3_client::s3_crt_client: S3 request finished request_type=Default crt_error=None http_status=200 range=None duration=27.628343ms ttfb=Some(23.65653ms) request_id=BW8M0Z7SGZE66E76
2024-05-03T01:41:27.267885Z DEBUG lookup{req=20 ino=2 name="test_file2.txt"}:list_objects{id=4 bucket="modal-s3mount-test-bucket" continued=false delimiter="/" max_keys="1" prefix="synmon/test_file2.txt/"}: mountpoint_s3_client::s3_crt_client: meta request finished duration=27.726881ms
2024-05-03T01:41:27.268199Z DEBUG fuser::request: FUSE( 22) ino 0x0000000000000003 OPEN flags 0x28800
2024-05-03T01:41:27.269918Z DEBUG open{req=22 ino=3 pid=2201237 name="test_file2.txt"}:head_object{id=5 bucket="modal-s3mount-test-bucket" key="synmon/test_file2.txt"}: mountpoint_s3_client::s3_crt_client::head_object: new request
2024-05-03T01:41:27.271456Z DEBUG open{req=22 ino=3 pid=2201237 name="test_file2.txt"}:list_objects{id=6 bucket="modal-s3mount-test-bucket" continued=false delimiter="/" max_keys="1" prefix="synmon/test_file2.txt/"}: mountpoint_s3_client::s3_crt_client::list_objects: new request
2024-05-03T01:41:27.280321Z DEBUG open{req=22 ino=3 pid=2201237 name="test_file2.txt"}:head_object{id=5 bucket="modal-s3mount-test-bucket" key="synmon/test_file2.txt"}: mountpoint_s3_client::s3_crt_client: S3 request finished request_type=Default crt_error=None http_status=200 range=None duration=10.324378ms ttfb=Some(10.170868ms) request_id=BW8YG0JPWWGZJ2D5
2024-05-03T01:41:27.280360Z DEBUG open{req=22 ino=3 pid=2201237 name="test_file2.txt"}:head_object{id=5 bucket="modal-s3mount-test-bucket" key="synmon/test_file2.txt"}: mountpoint_s3_client::s3_crt_client: meta request finished duration=10.422985ms
2024-05-03T01:41:27.283902Z DEBUG open{req=22 ino=3 pid=2201237 name="test_file2.txt"}:list_objects{id=6 bucket="modal-s3mount-test-bucket" continued=false delimiter="/" max_keys="1" prefix="synmon/test_file2.txt/"}: mountpoint_s3_client::s3_crt_client: S3 request finished request_type=Default crt_error=None http_status=200 range=None duration=12.393492ms ttfb=Some(11.423544ms) request_id=BW8QRFJM2X8SX9FG
2024-05-03T01:41:27.283927Z DEBUG open{req=22 ino=3 pid=2201237 name="test_file2.txt"}:list_objects{id=6 bucket="modal-s3mount-test-bucket" continued=false delimiter="/" max_keys="1" prefix="synmon/test_file2.txt/"}: mountpoint_s3_client::s3_crt_client: meta request finished duration=12.458821ms
2024-05-03T01:41:27.284211Z DEBUG open{req=22 ino=3 pid=2201237 name="test_file2.txt"}: mountpoint_s3::fs: new file handle created fh=5 ino=3
2024-05-03T01:41:27.284269Z DEBUG fuser::request: FUSE( 24) ino 0x0000000000000003 OPEN flags 0x28201
2024-05-03T01:41:27.285631Z DEBUG open{req=24 ino=3 pid=2201237 name="test_file2.txt"}:head_object{id=7 bucket="modal-s3mount-test-bucket" key="synmon/test_file2.txt"}: mountpoint_s3_client::s3_crt_client::head_object: new request
2024-05-03T01:41:27.287109Z DEBUG open{req=24 ino=3 pid=2201237 name="test_file2.txt"}:list_objects{id=8 bucket="modal-s3mount-test-bucket" continued=false delimiter="/" max_keys="1" prefix="synmon/test_file2.txt/"}: mountpoint_s3_client::s3_crt_client::list_objects: new request
2024-05-03T01:41:27.295750Z DEBUG open{req=24 ino=3 pid=2201237 name="test_file2.txt"}:head_object{id=7 bucket="modal-s3mount-test-bucket" key="synmon/test_file2.txt"}: mountpoint_s3_client::s3_crt_client: S3 request finished request_type=Default crt_error=None http_status=200 range=None duration=10.053739ms ttfb=Some(9.878797ms) request_id=BW8WPBFQNQ4VJBNE
2024-05-03T01:41:27.295791Z DEBUG open{req=24 ino=3 pid=2201237 name="test_file2.txt"}:head_object{id=7 bucket="modal-s3mount-test-bucket" key="synmon/test_file2.txt"}: mountpoint_s3_client::s3_crt_client: meta request finished duration=10.14902ms
2024-05-03T01:41:27.299245Z DEBUG open{req=24 ino=3 pid=2201237 name="test_file2.txt"}:list_objects{id=8 bucket="modal-s3mount-test-bucket" continued=false delimiter="/" max_keys="1" prefix="synmon/test_file2.txt/"}: mountpoint_s3_client::s3_crt_client: S3 request finished request_type=Default crt_error=None http_status=200 range=None duration=12.071358ms ttfb=Some(10.814732ms) request_id=BW8JR54B4ZNZMX2V
2024-05-03T01:41:27.299278Z DEBUG open{req=24 ino=3 pid=2201237 name="test_file2.txt"}:list_objects{id=8 bucket="modal-s3mount-test-bucket" continued=false delimiter="/" max_keys="1" prefix="synmon/test_file2.txt/"}: mountpoint_s3_client::s3_crt_client: meta request finished duration=12.157814ms
2024-05-03T01:41:27.299529Z WARN open{req=24 ino=3 pid=2201237 name="test_file2.txt"}: mountpoint_s3::fuse: open failed: inode error: inode 3 (full key "synmon/test_file2.txt") is not writable while being read
2024-05-03T01:41:27.299583Z DEBUG fuser::request: FUSE( 26) ino 0x0000000000000003 FLUSH fh FileHandle(5), lock owner LockOwner(2081044382602532771)
2024-05-03T01:41:27.299624Z DEBUG fuser::request: FUSE( 28) ino 0x0000000000000003 RELEASE fh FileHandle(5), flags 0x28800, flush false, lock owner None
2024-05-03T01:41:27.299688Z DEBUG fuser::request: FUSE( 30) ino 0x0000000000000002 RELEASEDIR fh FileHandle(4), flags 0x28800, flush false, lock owner None
With runc the debug logs show:
FUSE( 2) ino 0x0000000000000000 INIT kernel ABI 7.34, capabilities 0x33fffffb, max readahead 131072FUSE( 4) ino 0x0000000000000001 GETATTRFUSE( 6) ino 0x0000000000000001 LOOKUP name "synmon"FUSE( 8) ino 0x0000000000000002 LOOKUP name "test_file2.txt"FUSE( 10) ino 0x0000000000000003 OPEN flags 0x8201(open read-only)- ✅ successful open for write
from mountpoint-s3.
thundergolfer
commented on June 16, 2024
@dannycjones Interested to know what the consequences of violating the concurrent read and write restriction would be. Is it just an inconsistent (empty) read?
I've looked through these PRs because didn't gather the consequence:
from mountpoint-s3.
thundergolfer
commented on June 16, 2024
Ok opened google/gvisor#10389 in gvisor repo @ayushr2.
@jamesbornholt happy for this to be closed on your end 🙂 .
from mountpoint-s3.
Related Issues (20)
- Slurm(srun) unable to start mountpoint HOT 11
- Support wasabi s3 storage HOT 1
- Support ovhcloud s3 high performance object storage HOT 1
- Support for writing to Google Cloud Storage buckets HOT 11
- I think there should be a cool icon for this product HOT 1
- ERROR awscrt::AuthCredentialsProvider: static: invalid credential_source property: EcsContainer HOT 4
- Assuming role configured in AWS profile does not work outside of `aws` partition (China, US Gov Cloud, etc.) HOT 5
- Can't open & append file with path
- Clarify memory usage limiting in case of `--metadata-ttl` enabled HOT 2
- Support executable files HOT 2
- > mountpoint_s3::fuse: open failed: inode error: inode 3 (full key "synmon/test_file.txt") is not writable while being read HOT 1
- Use mount-s3 as a crate dependency HOT 1
- Debug/trace logging that includes the HTTP request headers HOT 3
- support for s3 mount on ecs variant of bottle rocket HOT 5
- support `AWS_ENDPOINT_URL` env variable HOT 2
- Mountpoint-s3 with minimal IAM Permissions HOT 1
- On-premises Direct Connect Integration HOT 2
- mount-s3 doesn't work on arm version on copy or writing (e.g. t4g.micro) HOT 3
- Specifying KMS Key ID (not ARN) or KMS Alias for upload causes Mountpoint to crash HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mountpoint-s3.