AWS S3 sync operations do not work with S3 directory buckets (S3 Express One Zone) about aws-cli HOT 9 CLOSED

Hi @jeffgardnerdev, thanks for reaching out. Could you provide debug logs of this behavior? You can get debug logs by adding --debug to your command, and redacting any sensitive information. Logs with --delete and without --delete would both be appreciated. Thanks!

from aws-cli.

Here are the two debug outputs, one without the --delete option and one with the --delete option. In this case the two prefixes are identical but the sync command copies/deletes 4 of the 5 files every time. Bucket names are redacted. ListObjectsV2 responses for the two prefixes return the objects in a different order.
s3-sync-standard-to-directory-with-delete-debug-output.txt
s3-sync-standard-to-directory-debug-output.txt

from aws-cli.

@jeffgardnerdev, thanks for your patience. I was able to reproduce this behavior, and your theory that this is related to ListObjectsV2 not sorting directory buckets is likely correct. We've reached out to the service team, and I'll leave any updates in this issue.

If other people are experiencing this issue, providing details and any impact in this issue would be appreciated.

Ticket # for internal use : P114641353

from aws-cli.

@RyanFitzSimmonsAK Could you share some more information about the decision to disable the sync command for directory buckets entirely? This is useful functionality that would be nice to keep, provided the ordering issue could be fixed.

from aws-cli.

Hello,

Thank you for reporting this issue. We have released AWS CLI v1.32.25 and v2.15.13 and strongly recommend that you upgrade to address this issue. If you are unable to upgrade, do not run the aws s3 sync command with an S3 Express One Zone directory bucket.

from aws-cli.

@jeffgardnerdev I did see your question about further information, which I will post here if I am able.

from aws-cli.

Here are some additional technical details on this issue if you are interested.

The reason we’ve removed the sync command for directory buckets is what @jeffgardnerdev already noticed: there is an incompatibility between the way that the CLI compares a list of objects in a bucket and the results of the ListObjectsv2 API call on directory buckets[1]. Operations with S3 are threaded, and we’re comparing a list of objects as from the S3 API as that list is being populated. This isn’t compatible with directory buckets, because there’s no way to ensure the ordering of objects coming from S3. Because of the incompatibility, the sync command will not work properly on a directory bucket. There is no workaround for directory buckets and sync at this time, except to refrain from using sync and instead use cp or similar.

In the versions of the CLI I noted above, v1.32.25 and v2.15.13 and later, we removed sync with directory bucket destinations or sources to prevent anyone from using this command and getting inconsistent or incorrect results.

1: Specifically, "Sorting order of returned objects" on the linked documentation page.

from aws-cli.

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

from aws-cli.

Isn't it a bit weird to close this issue, @kellertk? I would assume that people would want sync to work with S3 Express One Zone, for natural API compatibility.

from aws-cli.