Snapshot delete failing to delete snapshots for very large buckets about astra HOT 2 OPEN

Interestingly, all of the terminated calls appear to be stuck on the isDirectory logic. This may necessitate reworking this logic so we either don't need to check if it's a directory first, or making the directory check faster.

from astra.

This appears to be an issue with the current blobfs design, as it heavily relies on the assumption that the object store functions with directory-style functionality. This is a problem because S3 does have a traditional understanding of "folders" so attempting to operate on "directory" or "folder" level operations gets extremely slow with the larger amount of files stored (due to many ListDirectory calls).

Amazon does recommend using a secondary index if you are attempting to perform operations like this.

Our recommended path is probably to replace the existing blobfs design to one that doesn't rely on directory discovery, storing the specific file assets. This will require moving the SnapshotMetadata to store the exact list of files as well.

from astra.