mozilla / build-tooltool Goto Github PK

FYI: The following changes were made to this repository's wiki:

• defacing spam has been removed

• the wiki has been disabled, as it was not used

These were made as the result of a recent automated defacement of publically writeable wikis.

Tooltool admins should be able to do the following through the UI:

• delete files uploaded accidentally (including deleting all instances of the files in all regions)
• change file visibility ("oops, I uploaded that as PUBLIC")
• force revalidation (if a file seems to be corrupted on S3, this will re-validate the hash and delete any non-matching files; replication will then fill in the gaps if possible)

Each file will have a -- probably fairly small -- set of filenames under which it was uploaded. We should show those in the file display.

Per #10, doing otherwise opens us up to race conditions where an attacker could upload an invalid file.

The idea here is that each pending upload corresponds to a specific, unique key, probably something like /uploads/<pending upload id>. The check_pending_upload task will copy that key to /sha512/<digest> if the latter does not exist, and then validate the result of the copy operation. In any case, it should delete the upload key after the URL has expired.

We should probably also have a task that looks for uploaded files that have no matching pending upload, and deletes them.

This results in MySQL servers disappearing - see mozilla/build-relengapi#214.

Via newrelic:

Traceback (most recent call last):
File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request rv = self.dispatch_request()
File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args)
File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/newrelic-2.46.0.37/newrelic/hooks/framework_flask.py", line 40, in _nr_wrapper_handler_ return wrapped(*args, **kwargs)
File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/relengapi/lib/api.py", line 103, in replacement result = wrapped(*args, **kwargs)
File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/relengapi/blueprints/tooltool/__init__.py", line 140, in upload_batch if not p.get('tooltool.upload.{}'.format(v)).can():
AttributeError: 'NoneType' object has no attribute 'can'

All of my tooltool invocations push a single file to tooltool, and I never want the manifest until the upload is complete.

Can we make python tooltool.py upload --file=... Just Do The Right Thing, generating a file-specific manifest and uploading without all the add command staging?

In this situation I think it would make sense to poll the metadata service; if the poll fails, don't pass any region argument.

A JS frontend can do all of the things required to upload a file -- read local files, calculate their sha512 hash, get an upload URL, and PUT to that URL. We should allow users to upload via the UI using those techniques.

Starting with https://bugzilla.mozilla.org/show_bug.cgi?id=1355731 we're starting to use pieces of tooltool in a mach command. And while doing that, we're hitting two inconveniences:

• FileRecord.validate needs the file in the current directory, which means we need to os.chdir first (and os.chdir back to where we were before)
• unpack_file needs the given path to have the file name that appears in the manifest, which prevents from just feeding it with a cache file directly (which is an improvement which would be useful for tooltool itself).