eschwim / s3wipe Goto Github PK
View Code? Open in Web Editor NEWRapid AWS S3 bucket delete tool
License: MIT License
Rapid AWS S3 bucket delete tool
License: MIT License
Would be great if it was in PyPi so it can be installed with:
pip install s3wipe
If you need a hand with that, let me know.
In line 208:
listThreads = args.maxthreads / 3
can return a float value which causes an error
For example:
INFO: Starting 66.66666666666666 delete threads...
Modify listThreads to force an integer value and it will work:
listThreads = int(args.maxthreads / 3)
s3wipe
has a large number of lines with trailing spaces and a few lines that are only spaces -- this is pretty painful if your text editor is configured to highlight unneeded whitespace. Would a PR to clean this up be accepted?
Avoid having credentials used as cli parameters that will potentially be stored in shell history presenting a security risk.
Have the option to use a config file or fetch creds from any existing aws cli ~/.aws dir configs
https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
Can we get a little more help for when to modify batchsize and maxqueue from the defaults? When deleting millions of objects I just use the defaults and wait. It seems to dynamically adjust somehow. Can you throw something in the README when you get a chance? Thanks. Awesome script by the way!
Brian
This is a problem if the number of subdirectories is high, exhausting the number of available file descriptors. Suggest setting a maximum, say --max-threads=100.
Read up here: boto/boto#2836
I fixed this by adding this in the headers:
import ssl
if hasattr(ssl, '_create_unverified_context'):
ssl._create_default_https_context = ssl._create_unverified_context
Hey man,
I used your cli to delete 170TB of data and work fine, but the only problem in line 196 you enable the Bucket versioning then, in the end, I don't delete 170TB of that I only added the delete mark in all files.
I'm not sure why you are doing that, because you can't delete bucket with versions file inside.
I was trying to delete a directory called foo
, but if I passed s3://bucket/foo/
, it would not proceed, and if I passed s3://bucket/foo
it would also delete s3://bucket/foo2
I worked around it by deleting s3://bucket/foo/1
, s3://bucket/foo/2
and so-forth for every first character.
Two possible solutions: allow trailing slashes, or assuming a trailing slash after the input (i.e. don't delete partial paths).
So that I can declare it as a development dependency using Poetry.
There is an s3wipe package on PyPI, but it's not mentioned in the README, so I don't trust it.
Currently I'm working around it by commiting a copy of the script to my own repo like this:
curl "https://raw.githubusercontent.com/eschwim/s3wipe/80cdb19655a4db48830c50969a43dfa36b657015/s3wipe" > bin/s3wipe
chmod +x bin/s3wipe
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.