rsavordelli / s3bro Goto Github PK

boto3 lacks a sync function like the cli has (https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html). It would be really great if s3bro could implement this, useful features:

• multithreading
• multipart upload
• --delete option to ensure files not in the source are removed from the destination

This would make s3bro useful as a library for many cloudformation projects that need to ensure a local working copy is synced to s3 before launching the cfn stack. A good example is taskcat, currently a 35MB working dir (already populated, with small changes to 2 or 3 files) takes over 5 min to be analysed and synced.

Some related issues:
boto/boto#3343
aws-ia/taskcat#68

Hello,
I am getting bellow errors:
File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap self.run() self.run() File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run self.run() File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run self.run() self._target(*self._args, **self._kwargs) File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run File "/usr/lib/python2.7/multiprocessing/pool.py", line 113, in worker self._target(*self._args, **self._kwargs) File "/usr/lib/python2.7/multiprocessing/pool.py", line 113, in worker self._target(*self._args, **self._kwargs) File "/usr/lib/python2.7/multiprocessing/pool.py", line 113, in worker result = (True, func(*args, **kwds)) File "/usr/lib/python2.7/multiprocessing/pool.py", line 113, in worker result = (True, func(*args, **kwds)) File "/usr/lib/python2.7/multiprocessing/pool.py", line 65, in mapstar File "/usr/lib/python2.7/multiprocessing/pool.py", line 65, in mapstar result = (True, func(*args, **kwds)) return map(*args) result = (True, func(*args, **kwds)) return map(*args) File "/usr/lib/python2.7/multiprocessing/pool.py", line 65, in mapstar File "/usr/local/lib/python2.7/dist-packages/s3bro/ratelimit.py", line 25, in rate_limited_function File "/usr/lib/python2.7/multiprocessing/pool.py", line 65, in mapstar Process PoolWorker-8: File "/usr/local/lib/python2.7/dist-packages/s3bro/ratelimit.py", line 25, in rate_limited_function return map(*args) Process PoolWorker-6: time.sleep( left_to_wait ) Process PoolWorker-9: return map(*args) Traceback (most recent call last): time.sleep( left_to_wait ) File "/usr/local/lib/python2.7/dist-packages/s3bro/ratelimit.py", line 29, in rate_limited_function Traceback (most recent call last): KeyboardInterrupt Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/s3bro/ratelimit.py", line 29, in rate_limited_function File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap KeyboardInterrupt File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap ret = func( *args, **kwargs ) File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap ret = func( *args, **kwargs ) self.run() File "/usr/local/lib/python2.7/dist-packages/s3bro/s3_restore.py", line 100, in restore_default self.run() File "/usr/local/lib/python2.7/dist-packages/s3bro/s3_restore.py", line 100, in restore_default self.run() File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run if obj.storage_class == 'GLACIER': File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run if obj.storage_class == 'GLACIER': self._target(*self._args, **self._kwargs) self._target(*self._args, **self._kwargs) File "/usr/local/lib/python2.7/dist-packages/boto3/resources/factory.py", line 339, in property_loader self._target(*self._args, **self._kwargs) File "/usr/local/lib/python2.7/dist-packages/boto3/resources/factory.py", line 339, in property_loader File "/usr/lib/python2.7/multiprocessing/pool.py", line 113, in worker File "/usr/lib/python2.7/multiprocessing/pool.py", line 113, in worker File "/usr/lib/python2.7/multiprocessing/pool.py", line 113, in worker self.load()

Add an -ly to the end of Permanente
https://github.com/rsavordelli/s3bro/blob/master/s3bro/s3_restore.py#L75

The readme currently doesn't provide any information about how to provide the necessary AWS key ID and access key. This may be common knowledge for python devs but should be included in the usage instructions. I found out how to provide it in the boto docs and the environment variables worked just fine for me. I found the ~/.boto config file to be problematic however and I wasn't able to get it to work with that.

Thanks for writing this handy tool!

Would be very useful to be able to tail for read and deny actions as well.