Tasks in the RelEng infrastructure and make use of generic binary artifacts, which are stored in dedicated artifacts repositories (S3 buckets). ToolTool application provides an interface to those artifacts repositories.
License: Mozilla Public License 2.0
Sentry Issue: TOOLTOOL-1A
BadRequestProblem:
File "flask/app.py", line 1823, in full_dispatch_request
rv = self.dispatch_request()
File "flask/app.py", line 1799, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "connexion/decorators/decorator.py", line 68, in wrapper
response = function(request)
File "connexion/decorators/uri_parsing.py", line 149, in wrapper
response = function(request)
File "connexion/decorators/validation.py", line 377, in wrapper
raise BadRequestProblem(detail=error)
TypeError: common_error_handler() missing 1 required positional argument: 'exception'
File "flask/app.py", line 2529, in wsgi_app
response = self.full_dispatch_request()
File "flask/app.py", line 1825, in full_dispatch_request
rv = self.handle_user_exception(e)
File "flask_cors/extension.py", line 165, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "flask/app.py", line 1676, in handle_user_exception
return self.ensure_sync(handler)(e)
TypeError: common_error_handler() missing 1 required positional argument: 'exception'
Sentry Issue: TOOLTOOL-1T
ValueError: A string literal cannot contain NUL (0x00) characters.
(19 additional frame(s) were not displayed)
...
File "sqlalchemy/engine/base.py", line 1846, in _execute_context
return self._exec_single_context(
File "sqlalchemy/engine/base.py", line 1987, in _exec_single_context
self._handle_dbapi_exception(
File "sqlalchemy/engine/base.py", line 2347, in _handle_dbapi_exception
raise exc_info[1].with_traceback(exc_info[2])
File "sqlalchemy/engine/base.py", line 1968, in _exec_single_context
self.dialect.do_execute(
File "sqlalchemy/engine/default.py", line 920, in do_execute
cursor.execute(statement, parameters)
ValueError: A string literal cannot contain NUL (0x00) characters.
The releng_tooltool_files.size column is an integer, meaning it takes values between -2147483648 and +2147483647.
It should probably be a bigint instead.
I'm looking at https://docs.mozilla-releng.net/projects/tooltool.html#request-authentication-credentials-for-the-client-tooltool-py (which I don't know where is now maintained, if anywhere).
The instructions say that people should file a bug, and then we will generate taskcluster credentials for them.
This should be changed to indicate that we will grant their taskcluster role the appropriate scopes, and then they can create a client with the appropriate scopes to use tooltool.
It may also make sense to tooltool to use the standard taskcluster environment variables, rather than having its own file format for them.
All XHR requests fail. Most likely because the base URL is set wrong (it's trying localhost).
I created a PR that should have failed a test case on tox, but all checks passed. It looks like tox is not being run.
Running the delete command with the client fails with this error:
Traceback (most recent call last):
File "/tmp/tooltool/client/tooltool.py", line 1713, in <module>
sys.exit(main(sys.argv))
^^^^^^^^^^^^^^
File "/tmp/tooltool/client/tooltool.py", line 1709, in main
return 0 if process_command(options, args) else 1
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/tooltool/client/tooltool.py", line 1540, in process_command
return delete_instances(
^^^^^^^^^^^^^^^^^
File "/tmp/tooltool/client/tooltool.py", line 1473, in delete_instances
return send_operation_on_file(data, base_urls, digest, auth_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/tooltool/client/tooltool.py", line 1450, in send_operation_on_file
urllib2.urlopen(req)
File "/usr/lib/python3.11/urllib/request.py", line 216, in urlopen
return opener.open(url, data, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/urllib/request.py", line 516, in open
req = meth(req)
^^^^^^^^^
File "/usr/lib/python3.11/urllib/request.py", line 1279, in do_request_
raise TypeError(msg)
That being said, even with that fixed up, delete doesn't seem to work.
Capture some of the discussion on https://bugzilla.mozilla.org/show_bug.cgi?id=1743179 to guide future tooltool travellers.
I see this error when trying to download a manifest file from tooltool
manifest:
https://searchfox.org/mozilla-central/source/testing/mozbase/mozproxy/mozproxy/backends/mitm/mitmproxy-rel-bin-4.0.4-win.manifest
command line: python3.6 tooltool.py fetch -o -m <manifets path>
/mozilla/tooltool/client$ python3.6 tooltool.py fetch -o -m /home/florinstrugariu/mozilla/mozilla-central/testing/mozbase/mozproxy/mozproxy/backends/mitm/mitmproxy-rel-bin-4.0.4-win.manifest
INFO - Attempting to fetch from 'https://tooltool.mozilla-releng.net/'...
INFO - ...failed to fetch 'mitmproxy-4.0.4-windows.zip' from https://tooltool.mozilla-releng.net/
Traceback (most recent call last):
File "tooltool.py", line 741, in fetch_file
indata = to_text(indata)
File "tooltool.py", line 99, in to_text
return val.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xad in position 10: invalid start byte
ERROR - The following files failed: 'mitmproxy-4.0.4-windows.zip'
Here we have 1.3.0: https://github.com/mozilla-releng/tooltool/blob/master/client/version.txt
and here, 1 : https://github.com/mozilla-releng/tooltool/blob/master/client/tooltool.py#L55
let's synchronize both, and change __version__ so it matches version.txt
(I guess in the long term we drop __version__)
In at least two ways:
Uncaught ReferenceError: hawk is not defined after logging inError: hawk is not defined and Error: can't access property "is_restapi_request", response.config is undefined when trying to searchHitting these exceptions:
tooltool_api.lib.pulse: Reconnecting in 10 seconds (exception='Traceback (most recent call last):
File "/appenv/lib/python3.7/site-packages/aioamqp/frame.py", line 78, in read
data = await reader.readexactly(7)
File "/usr/local/lib/python3.7/asyncio/streams.py", line 677, in readexactly
raise IncompleteReadError(incomplete, n)
asyncio.streams.IncompleteReadError: 0 bytes read on a total of 7 expected bytes
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/app/src/tooltool_api/lib/pulse.py", line 107, in create_consumer
return await _create_consumer(user, password, exchange, topic, callback)
File "/app/src/tooltool_api/lib/pulse.py", line 46, in _create_consumer
port=port,
File "/appenv/lib/python3.7/site-packages/aioamqp/__init__.py", line 63, in connect
insist=insist)
File "/appenv/lib/python3.7/site-packages/aioamqp/protocol.py", line 216, in start_connection
await self.dispatch_frame()
File "/appenv/lib/python3.7/site-packages/aioamqp/protocol.py", line 263, in dispatch_frame
frame_channel, frame = await self.get_frame()
File "/appenv/lib/python3.7/site-packages/aioamqp/protocol.py", line 250, in get_frame
return await amqp_frame.read(self._stream_reader)
File "/appenv/lib/python3.7/site-packages/aioamqp/frame.py", line 80, in read
raise exceptions.AmqpClosedConnection() from ex
aioamqp.exceptions.AmqpClosedConnection')
Also have aioamqp.protocol: only PLAIN login_method is supported, falling back to AMQPLAIN in the logs.
We have 3 different copies of the tooltool.py client vendored into the tree:
The last time we tried to update the in-tree copies, we discovered that two of them had differences not found in the other, nor the copy in this repo.
We should add a test here to verify that all copies are the same. Maybe a test in tree as well.
Hey, the doc says
If you want to use the client, just copy out tooltool.py -- it has no dependencies.
I think it's a problem.
Having people copy that script here and there has two major disadvantages:
For example the version in this repo says version == '1' but the one in mozharness
has the same version but is a different module.
We bumped into the issue because we started to work on https://phabricator.services.mozilla.com/D54090 to add a small retry mechanism to try
to reduce intermittents on Raptor, and decided to use mozbuild's tooltool version as
the source of truth, but it turns out this repo is the one :)
there's also https://github.com/mozilla/build-relengapi ?
I would like to make this proposal:
Even after #9 is fixed, login will be broken because the /auth0 endpoints don't exist. We probably need to change auth to talk directly to Taskcluster, although this needs a bit more investigation first.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.