Add "sync" command about s5cmd HOT 10 CLOSED

Hi, sync is on our roadmap, please follow this issue to be notified about any updates regarding s5cmd sync implementation.

from s5cmd.

A aws cli sync feature would be rally useful, so object are remove at bucket. Any news on a sync feature?

from s5cmd.

Thanks for the feature request. It'll be a great addition to the current feature set.

Even though it's not the same as "sync", you can synchronize two directories (update if a file differs) with s5cmd cp -u -s --parents s3://<bucket>/<folder> .. Please keep in mind that it's not going to remove an existing object/file.

from s5cmd.

Dear developers, do you plan to implement synchronization? For example like https://github.com/sethkor/s3kor (without checking MD5). I cannot use s3kor because I have a non-AWS endpoint. s5cmd there is just an opportunity to use the --endpoint-url flag. that's what i need. orient us please

from s5cmd.

s5cmd v2.0.0-beta is released. It includes sync command. Happy to receive your feedback and contributions!

from s5cmd.

looking forward for a faster version of s3 sync as well

from s5cmd.

just got this project https://github.com/sethkor/s3kor supports fast sync

from s5cmd.

just got this project https://github.com/sethkor/s3kor supports fast sync

Here I quote from the s3kor repository page

Sync only compares mod timestamps and sizes as the only true way to get a MD5 or some other checksum of an S3 object is to download it comletley first. An objects ETag is not the MD5 of the object.

Then it's very similar to s5cmd cp -u -s, except it did one more scan to remove non-existed keys. Sounds like an acceptable workaround, better than nothing.

from s5cmd.

Any update our new release version that includes sync operator? Thanks!

from s5cmd.

Hi, sync command works great with few files. I am trying to sync 2 million files into s3 bucket which has more than 15 million files but I stops after 5 minutes with "killed" error message

from s5cmd.