eschwim / s3wipe Goto Github PK

Would be great if it was in PyPi so it can be installed with:

pip install s3wipe

If you need a hand with that, let me know.

In line 208:
listThreads = args.maxthreads / 3

can return a float value which causes an error

For example:

INFO: Starting 66.66666666666666 delete threads...

Modify listThreads to force an integer value and it will work:
listThreads = int(args.maxthreads / 3)

s3wipe has a large number of lines with trailing spaces and a few lines that are only spaces -- this is pretty painful if your text editor is configured to highlight unneeded whitespace. Would a PR to clean this up be accepted?

Avoid having credentials used as cli parameters that will potentially be stored in shell history presenting a security risk.

Have the option to use a config file or fetch creds from any existing aws cli ~/.aws dir configs
https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html

Can we get a little more help for when to modify batchsize and maxqueue from the defaults? When deleting millions of objects I just use the defaults and wait. It seems to dynamically adjust somehow. Can you throw something in the README when you get a chance? Thanks. Awesome script by the way!

Brian

This is a problem if the number of subdirectories is high, exhausting the number of available file descriptors. Suggest setting a maximum, say --max-threads=100.

Read up here: boto/boto#2836

I fixed this by adding this in the headers:

import ssl
if hasattr(ssl, '_create_unverified_context'):
ssl._create_default_https_context = ssl._create_unverified_context

Hey man,

I used your cli to delete 170TB of data and work fine, but the only problem in line 196 you enable the Bucket versioning then, in the end, I don't delete 170TB of that I only added the delete mark in all files.
I'm not sure why you are doing that, because you can't delete bucket with versions file inside.

I was trying to delete a directory called foo, but if I passed s3://bucket/foo/, it would not proceed, and if I passed s3://bucket/foo it would also delete s3://bucket/foo2

I worked around it by deleting s3://bucket/foo/1, s3://bucket/foo/2 and so-forth for every first character.

Two possible solutions: allow trailing slashes, or assuming a trailing slash after the input (i.e. don't delete partial paths).

So that I can declare it as a development dependency using Poetry.

There is an s3wipe package on PyPI, but it's not mentioned in the README, so I don't trust it.

Currently I'm working around it by commiting a copy of the script to my own repo like this:

curl "https://raw.githubusercontent.com/eschwim/s3wipe/80cdb19655a4db48830c50969a43dfa36b657015/s3wipe" > bin/s3wipe
chmod +x bin/s3wipe