mozilla-releng / buildhub2 Goto Github PK
View Code? Open in Web Editor NEWA database index of buildhub.json files built by Mozilla
Home Page: https://buildhub2.readthedocs.io/
License: Mozilla Public License 2.0
A database index of buildhub.json files built by Mozilla
Home Page: https://buildhub2.readthedocs.io/
License: Mozilla Public License 2.0
The issue can be reproduced in both Firefox and Chrome browser. I describe the steps to reproduce it in Chrome browser first, and in Firefox browser later.
Developer Tools
pannel in the window and click on the network
tab so you can watch the networking activities.https://buildhub2.stage.mozaws.net
console
tab)The issue can be re-produced in Firefox browser as well. The only difference is that if using a Firefox browser, between the above step 4 and 5, you need to wait for more than 24 hours (presumably, that is the amount of time specified in the response cache-control
header, where it demands max-age=86400, public
for the root page.
In the step 5 above, when clicking the reload button, the following two requests were received at the server end.
"GET /favicon.ico HTTP/1.1" 200 34885 ...
"GET /service-worker.js HTTP/1.1" 200 1383 ...
In both browsers, running the following code in the javascript console will allow the web site to reload again.
navigator.serviceWorker.ready.then(registration => { registration.unregister(); });
Serve it like we do in Tecken.
Because that's what we're going to deploy in Stage and Prod.
Here's now missioncontrol queries buildhub
(and settings)
Once we have a new URL for Buildhub2 (and we're confident we have all old and new builds) we should help them migrate.
At the moment, kinto-migration relies on HTTP. We need to make it be able to draw straight from a kinto Postgres database.
We've discovered that if you run ./manage.py kinto-database-migration --skip-validation
there are some really really broken records going into the JSONField. They're so broken that it breaks running ./manage.py reindex-elasticsearch
because it assumes that every JSON blob has the keys build
, target
, source
, and download
. If any of them are missing, reindex-elasticsearch
will crash.
If you run DJANGO_ES_REFRESH_INTERVAL=-1 ./manage.py reindex-elasticsearch
the refresh_interval
gets "stuck". Meaning, the next time you start Elasticsearch it's still set to -1
.
See http://localhost:9400/buildhub2 (look at settings.refresh_interval
). Need a way to custom change this when running reindex-elasticsearch
.
Need to be aware that the check might fail when you start docker-compose
or honcho
.
Once moved to mozilla set up Pyup.
Our current unit tests actively use Postgres. We should equally use Elasticsearch but we should not pollute the index we use for running the dev server.
For example, a crash like this links to https://mozilla-services.github.io/buildhub/?q=20180807170231 which is going to have to change in Crash-Stats but it should still make sure the right buildid search works and highlights.
Especially configuration and how to run it locally.
At the moment, "build" is not a required key.
This is rather odd. All valid records on Buildhub Prod (the legacy one) have a "build" key.
docker_layer_caching: true
only works if it's enabled on the account.
Because of the way releases are all put into the same directory, it doesn't make sense to look for a file called exactly buildhub.json
.
For example, a directory might contain: Firefox Setup 61.0.exe
, Firefox Setup 60.0.exe
and Firefox Setup 62.0.exe
. In that case the buildhub.json file is going to be called 61.buildhub.json
(or was it 61-buildhub.json
).
We decided that the minimum is that the file will always end on buildhub.json
.
Therefore this or this won't work.
URLs like / and /static/main.css should be served from disk with Whitenoise.
I think there is a typo on this line. It probably should be DJANGO_ES_URLS
.
The UI uses the /search
API endpoint but it doesn't pass any of the request.POST to Elasticsearch.
https://pypi.org/project/black/ is awesome but still alpha. Are we ready to adopt?
In Buildhub1 there used to a section of example queries
Some of them are Kinto specific and some of them are Elasticsearch.
We should have list like this too somewhere for Buildhub2.
This is useful to see how each record was created.
That makes it easier to organize settings and rock solid solution to make sure we use settings specifically for tests.
If you use the ./manage.py reindex-elasticsearch
it warns about setting ES_REFRESH_INTERVAL
If you do that you should run _forcemerge
afterwards. The command is this:
'curl -XPOST http://localhost:9200/{}/_forcemerge?max_num_segments=5'.format(index_name)
When running ./manage.py reindex-elasticsearch
to create the buildhub2
index for the first time, we ran into the following warning message.
$ ./manage.py reindex-elasticsearch
{"Timestamp": 1536193993444640512, "Type": "markus.backends.datadog", "Logger": "buildhub", "Hostname": "ip-172-31-2-154", "EnvVersion": "2.0", "Severity": 6, "Pid": 8, "Fields": {"msg": "DatadogMetrics configured: localhost:8125 "}}
System check identified some issues:
WARNINGS:
main.Build.metadata: (postgres.E003) JSONField default should be a callable instead of an instance so that it's not shared between all field instances.
HINT: Use a callable instead, e.g., use `dict` instead of `{}`.
Force merged Elasticsearch index after bulk insert
Fwiw, despite of the above warning, the buildhub2
index was created successfully (it seems like).
Not sure how serious the warning message is or if you want to fix it or not.
This way we only need to redirect the domain or simply as people to change domain name.
After we have a Dockerfile we should test in Circle with
Use Markus and put incrs, timers and gauges everywhere where it makes sense.
We need to mock SQS and S3 but there should be tests.
Resources for -stage environment (database, elasticsearch, application stack and etc.) are set up and the application is able to start up.
Trying to run the migrate
command for the first time to bootstrap the database in this environment.
Here is the command we ran:
$ docker run --rm --net host --env-file /etc/dockerflow/buildhub2.txt --log-driver=syslog hub.prod.mozaws.net/pipelines/buildhub2/buildhub2:latest migrate
And here is the error:
Not waiting for any services
{"Timestamp": 1533331050280736768, "Type": "markus.backends.datadog", "Logger": "buildhub", "Hostname": "ip-172-31-41-251.ec2.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 6, "Fields": {"msg": "DatadogMetrics configured: localhost:8125 "}}
Traceback (most recent call last):
File "manage.py", line 9, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 365, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 288, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 332, in execute
self.check()
File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 364, in check
include_deployment_checks=include_deployment_checks,
File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 58, in _run_checks
issues.extend(super()._run_checks(**kwargs))
File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 351, in _run_checks
return checks.run_checks(**kwargs)
File "/usr/local/lib/python3.6/site-packages/django/core/checks/registry.py", line 73, in run_checks
new_errors = check(app_configs=app_configs)
File "/app/buildhub/dockerflow_extra.py", line 41, in check_elasticsearch
health = fetch(url)["status"]
File "/usr/local/lib/python3.6/site-packages/backoff/_sync.py", line 99, in retry
ret = target(*args, **kwargs)
File "/app/buildhub/dockerflow_extra.py", line 33, in fetch
response.raise_for_status()
File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 939, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 408 Client Error: Request Timeout for url: http://internal-buildhub2-es-s-elb-1qf0a53abi35c-1034878185.us-east-1.elb.amazonaws.com:9200/_cluster/health/buildhub2
So it looks like there is some kinda of chicken-egg problem. Because it tries to check the health of a non-existing index before running the database migration.
Is it possible to document the steps of how to bootstrap a brand new environment? That would be ideal if we can follow those steps exactly to set this up.
A less ideal solution would be let us know how to manually work around it. But that means we'll have to play the same "manual tricks" when we deploy -prod environment.
Error shows
Jul 12 03:22:07 ip-172-31-27-217 systemd[1]: Started Daemon for buildhub2.
Jul 12 03:22:08 ip-172-31-27-217 docker-buildhub2[6141]: Not waiting for any services
Jul 12 03:22:08 ip-172-31-27-217 docker-buildhub2[6141]: python: can't open file 'gunicorn': [Errno 2] No such file or directory
gunicorn doesn't seem to be installed in the docker image.
$ sudo docker run -it --rm hub.prod.mozaws.net/pipelines/buildhub2/buildhub2 pip list |egrep -c gunicorn
0
Resources for -stage environment (database, elasticsearch, application stack and etc.) are set up and the application is able to start up.
Trying to run the migrate
command for the first time to bootstrap the database in this environment.
Here is the command we ran:
$ docker run --rm --net host --env-file /etc/dockerflow/buildhub2.txt --log-driver=syslog hub.prod.mozaws.net/pipelines/buildhub2/buildhub2:latest migrate
And here is the error:
Not waiting for any services
Traceback (most recent call last):
File "manage.py", line 8, in <module>
from configurations.management import execute_from_command_line
File "/usr/local/lib/python3.6/site-packages/configurations/management.py", line 3, in <module>
importer.install(check_options=True)
File "/usr/local/lib/python3.6/site-packages/configurations/importer.py", line 49, in install
importer = ConfigurationImporter(check_options=check_options)
File "/usr/local/lib/python3.6/site-packages/configurations/importer.py", line 67, in __init__
self.check_options()
File "/usr/local/lib/python3.6/site-packages/configurations/importer.py", line 87, in check_options
add_help=False)
TypeError: __init__() takes 1 positional argument but 2 were given
Please note that the user scenario is the same as #187, but the error this time is different.
When we upgraded SearchKit we lost the ability to custom sort the versions.
We should set all good-for-security headers with config. I.e. nginx through cloudops-deployment.
When we have this set up we don't need to set these things in Django additionally.
Title says it all.
Most important; a note in README or dev.rst about how to use it :)
If this prototype is ever going to run in CloudOps it needs to have a Dockerfile. And we need to make sure it works as a docker file locally so we need to write a docker-compose.yaml file.
It's currently in peterbe.
Need to mock boto like crazy
We're sure to make mistakes with SQS (code or operational) so a cron job backfill script would be nice to suck up all buildhub.json files that exist.
DB migration is done. App is able to start up successfully.
The app instances can't register with the ELB because the app returns 400 for /__lbheartbeat__
endpoint.
Here is the app log:
Aug 10 19:18:28 ip-172-31-21-40.ec2.internal systemd[1]: Started Daemon for buildhub2.
Aug 10 19:18:29 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: Not waiting for any services
Aug 10 19:18:30 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: [2018-08-10 19:18:30 +0000] [6] [INFO] Starting gunicorn 19.9.0
Aug 10 19:18:30 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: [2018-08-10 19:18:30 +0000] [6] [INFO] Listening at: http://0.0.0.0:8000 (6)
Aug 10 19:18:30 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: [2018-08-10 19:18:30 +0000] [6] [INFO] Using worker: sync
Aug 10 19:18:30 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: [2018-08-10 19:18:30 +0000] [9] [INFO] Booting worker with pid: 9
Aug 10 19:18:30 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: [2018-08-10 19:18:30 +0000] [10] [INFO] Booting worker with pid: 10
Aug 10 19:18:30 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: [2018-08-10 19:18:30 +0000] [11] [INFO] Booting worker with pid: 11
Aug 10 19:18:30 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: [2018-08-10 19:18:30 +0000] [12] [INFO] Booting worker with pid: 12
Aug 10 19:18:30 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: [2018-08-10 19:18:30 +0000] [13] [INFO] Booting worker with pid: 13
Aug 10 19:18:33 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: {"Timestamp": 1533928713849477376, "Type": "markus.backends.datadog", "Logger": "buildhub", "Hostname": "ip-172-31-21-40.ec2.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 10, "Fields": {"msg": "DatadogMetrics configured: localhost:8125 "}}
Aug 10 19:18:33 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: {"Timestamp": 1533928713855258112, "Type": "markus.backends.datadog", "Logger": "buildhub", "Hostname": "ip-172-31-21-40.ec2.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 11, "Fields": {"msg": "DatadogMetrics configured: localhost:8125 "}}
Aug 10 19:18:33 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: {"Timestamp": 1533928713858804992, "Type": "markus.backends.datadog", "Logger": "buildhub", "Hostname": "ip-172-31-21-40.ec2.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 12, "Fields": {"msg": "DatadogMetrics configured: localhost:8125 "}}
Aug 10 19:18:33 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: {"Timestamp": 1533928713859929856, "Type": "markus.backends.datadog", "Logger": "buildhub", "Hostname": "ip-172-31-21-40.ec2.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 13, "Fields": {"msg": "DatadogMetrics configured: localhost:8125 "}}
Aug 10 19:18:33 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: {"Timestamp": 1533928713860281344, "Type": "markus.backends.datadog", "Logger": "buildhub", "Hostname": "ip-172-31-21-40.ec2.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 9, "Fields": {"msg": "DatadogMetrics configured: localhost:8125 "}}
Aug 10 19:19:40 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: 127.0.0.1 - - [10/Aug/2018:19:19:40 +0000] "GET /v1/__lbheartbeat__ HTTP/1.0" 400 26 "-" "ELB-HealthChecker/1.0"
Aug 10 19:19:54 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: 127.0.0.1 - - [10/Aug/2018:19:19:54 +0000] "GET /v1/__lbheartbeat__ HTTP/1.0" 400 26 "-" "ELB-HealthChecker/1.0"
Aug 10 19:20:09 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: 127.0.0.1 - - [10/Aug/2018:19:20:09 +0000] "GET /v1/__lbheartbeat__ HTTP/1.0" 400 26 "-" "ELB-HealthChecker/1.0"
Aug 10 19:20:24 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: 127.0.0.1 - - [10/Aug/2018:19:20:24 +0000] "GET /v1/__lbheartbeat__ HTTP/1.0" 400 26 "-" "ELB-HealthChecker/1.0"
Aug 10 19:20:39 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: 127.0.0.1 - - [10/Aug/2018:19:20:39 +0000] "GET /v1/__lbheartbeat__ HTTP/1.0" 400 26 "-" "ELB-HealthChecker/1.0"
Aug 10 19:20:47 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: 127.0.0.1 - - [10/Aug/2018:19:20:47 +0000] "GET /v1/__lbheartbeat__ HTTP/1.0" 400 26 "-" "ELB-HealthChecker/1.0"
Aug 10 19:20:54 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: 127.0.0.1 - - [10/Aug/2018:19:20:54 +0000] "GET /v1/__lbheartbeat__ HTTP/1.0" 400 26 "-" "ELB-HealthChecker/1.0"
Aug 10 19:21:02 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: 127.0.0.1 - - [10/Aug/2018:19:21:02 +0000] "GET /v1/__lbheartbeat__ HTTP/1.0" 400 26 "-" "ELB-HealthChecker/1.0"
Aug 10 19:21:09 ip-172-31-21-40.ec2.internal docker-buildhub2[2655]: 127.0.0.1 - - [10/Aug/2018:19:21:09 +0000] "GET /v1/__lbheartbeat__ HTTP/1.0" 400 26 "-" "ELB-HealthChecker/1.0"
There is no further info in the logs to tell why it returns 400, unfortunately.
The app also returns 400 for other standard endpoints, such as /__version__
, /__heartbeat__
, /__lbheartbeat__
, /v1/__version__
, /v1/__heartbeat__
and etc.
To be run once to fill up the database with legacy builds that weren't made as buildhub.json
files.
When you start from absolutely scratch, you have nothing the database nor in Elasticsearch. If you try to use the website from that point you'll get an error like this from ES:
{
"status": 404,
"error": {
"index_uuid": "_na_",
"index": "buildhub2",
"resource.type": "index_or_alias",
"root_cause": [
{
"index_uuid": "_na_",
"index": "buildhub2",
"resource.type": "index_or_alias",
"resource.id": "buildhub2",
"reason": "no such index",
"type": "index_not_found_exception"
}
],
"reason": "no such index",
"type": "index_not_found_exception",
"resource.id": "buildhub2"
}
}
But that error isn't important arguably because you shouldn't be hitting the web API URLs until you have bootstrapped.
The question is; how do you bootstrap. It's not documented. Documentation on this should help both local dev coders and fresh deployments.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.