In an attempt to disable the metadata cache, I set the --met

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Clarify the expected behavior of the `--metadata-ttl` option set to zero about mountpoint-s3 HOT 2 OPEN

mhnap commented on June 23, 2024

Clarify the expected behavior of the `--metadata-ttl` option set to zero

from mountpoint-s3.

Comments (2)

jamesbornholt commented on June 23, 2024

Yes, --metadata-ttl=0 disables metadata caching entirely, which is almost certainly not a good thing, as filesystem performance with zero metadata caching is very bad. For Mountpoint, that means sending many more LIST/HEAD requests than necessary, as you're seeing.

When caching is disabled (i.e. no --cache or --metadata-ttl is set), the actual setting is roughly the same as --metadata-ttl=0.1 (noting that you can't actually set that as a float), as this comment explains:

mountpoint-s3/mountpoint-s3/src/fs.rs

Lines 303 to 313 in 022f915

 // We want to do as little caching as possible by default, 

 // but Linux filesystems behave badly when the TTL is exactly zero. 

 // For example, results from `readdir` will expire immediately, and so 

 // the kernel will immediately re-lookup every entry returned from `readdir`. So we apply 

 // small non-zero TTLs. The goal is to be small enough that the impact on consistency is 

 // minimal, but large enough that a single cache miss doesn't cause a cascading effect where 

 // every other cache entry expires by the time that cache miss is serviced. We also apply a 

 // longer TTL for directories, which are both less likely to change on the S3 side and 

 // checked more often (for directory permissions checks). 

 let file_ttl = Duration::from_millis(100); 

 let dir_ttl = Duration::from_millis(1000);

This 0.1 setting is also the reason our documentation has a caveat around consistency:

However, Mountpoint may return stale metadata for an existing object within 1 second of the object being modified or deleted in your S3 bucket by another client. This occurs only if the object was accessed through Mountpoint immediately before being modified or deleted in your S3 bucket. The stale metadata will only be visible through metadata operations such as stat on individual files. Directory listings will never be stale and always reflect the current metadata. These cases do not apply to newly created objects, which are always immediately visible through Mountpoint. Stale metadata can be refreshed by either opening the file or listing its parent directory.

We probably should just forbid --metadata-ttl=0, or convert it to mean the same thing as the default value, as I can't imagine a setting where it would actually work well. Until then, I think --metadata-ttl=1 is likely the effect you're looking for.

from mountpoint-s3.

mhnap commented on June 23, 2024

@jamesbornholt Thank you for such a detailed explanation. Now it's clear to me why I saw such behavior for the --metadata-ttl=0 case. I just didn't realize that --metadata-ttl=0 would result in different settings than disabled caching.

Now, as I understand that --metadata-ttl=0 is technically valid and produces expected behavior, the only proposition I have is to add more details about the current default TTLs settings to the documentation. Not sure whether any changes are needed on the CLI side.

from mountpoint-s3.

Clarify the expected behavior of the `--metadata-ttl` option set to zero about mountpoint-s3 HOT 2 OPEN

Comments (2)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	// We want to do as little caching as possible by default,
	// but Linux filesystems behave badly when the TTL is exactly zero.
	// For example, results from `readdir` will expire immediately, and so
	// the kernel will immediately re-lookup every entry returned from `readdir`. So we apply
	// small non-zero TTLs. The goal is to be small enough that the impact on consistency is
	// minimal, but large enough that a single cache miss doesn't cause a cascading effect where
	// every other cache entry expires by the time that cache miss is serviced. We also apply a
	// longer TTL for directories, which are both less likely to change on the S3 side and
	// checked more often (for directory permissions checks).
	let file_ttl = Duration::from_millis(100);
	let dir_ttl = Duration::from_millis(1000);