In the 1082 records currently in Kinto, there's 98 with a version parsed as 05-03-00-40-06-comm-aurora-l10n/thunderbird-54.0a2.

If it helps, their archive folder is https://archive.mozilla.org/pub/thunderbird/nightly/2017/05/2017-05-03-00-40-06-comm-aurora-l10n/ (and an example file is https://archive.mozilla.org/pub/thunderbird/nightly/2017/05/2017-05-03-00-40-06-comm-aurora-l10n/thunderbird-54.0a2.is.win32.zip

Currently with the async producer/consumer, the 4XX errors when pushing the records are not well emphasized.

Maybe we could accumulate errors in a queue and output them all at once when the scripts terminates.

Related Kinto/kinto-http.py#153 & Kinto/kinto-http.py#154

Build info are published as JSON in the candidates builds folder and for nightlies, but only for en-US.

https://archive.mozilla.org/pub/firefox/candidates/51.0b6-candidates/build1/linux-x86_64/en-US/firefox-51.0b6.json

https://archive.mozilla.org/pub/firefox/nightly/2017/05/2017-05-01-10-01-39-mozilla-central/firefox-55.0a1.en-US.linux-i686.json

Investigate:

• what info we can reuse for other locales?
• how the Socorro scrapper uses the list of candidates builds?
• is the list of builds fixed or has a chance to be updated?

See #18

Once #40 is landed, we'll have a Route stored in the model, which holds all the information for the filters, thus making the individual *Filter fields redundant.

$ tree -L 2 pub/firefox/releases/partners
├── fujitsu-siemens
│   ├── 2.0.0.6
├── google
│   ├── 1.5.0.12
│   ├── 2.0.0.6
├── packardbell
│   ├── 2.0.0.6
├── seznam
│   ├── 2.0.0.6
├── yahoo
│   ├── 1.5.0.6
│   ├── 1.5.0.7
└── yahoo-japan
    ├── 2.0.0.6

Should we handle them or just ignore them?

• Previous/next beta
• Previous/next nightly
• Previous ESR
• Previous stable
  ...

ultimate goal: diff between releases :)

It would be nice to make sure the UI can build when we do PR on the web ui.

If we want to leverage what we receive in Pulse, we'll have to migrate it to Python 3

Rework first version of patch here: https://bugzilla.mozilla.org/show_bug.cgi?id=1294230

Somewhere (ShipIt at least), there is the mercurial changeset of the locale that was used to build the release. Find out how to get that :)

The scraping picks us stuff like https://archive.mozilla.org/pub/mobile/releases/21.0b1/android-armv6/multi/fennec-21.0b1.multi.android-arm-armv6.tests.zip and https://archive.mozilla.org/pub/mobile/releases/24.0b1/android/en-US/fennec-24.0b1.en-US.android-arm.crashreporter-symbols.zip which leads to record ids conflicts

Currently we don't have clear semantics about tree/channel etc.

I propose that we keep:

• repository: full repository URL (eg. https://hg.mozilla.org/releases/mozilla-beta)
• channel: update channel (for AUS, beta, nightly, release)
• tree: path of repository relative to root (eg. releases/mozilla-beta)→ useful?

Related #17

https://archive.mozilla.org/pub/thunderbird/candidates/52.0.1-candidates/build3/linux-x86_64/en-US/thunderbird-52.0.1.json

When the filter choices were built dynamically, it made sense to "pin" the chosen value (and display it as a button). Now that the values are hardcoded (see #72), it doesn't make sense anymore.

When we select a new filter, it fires a new request to kinto with the updated filters, but also a "url change" (which ultimately sends a request to Kinto).

This results in two requests to Kinto, we should fix that.

As you can see it says, 10 build displayed (0 total)

I was experiencing very slow loading times from the UI, but thought it was because of my old computer but @Natim and @n1k0 were having the same issues, and sometimes even worse with error messages in the console like http://i.imgur.com/OO36Ymf.png or http://i.imgur.com/cdbaxRF.png

After some investigation, it seems we were hitting elm/compiler#1521

The workaround is simple: split long lists and concatenate them.

At the moment, the error message is just a log in the console.

Whenever there's a string entered in the build id search box, two things happen:
1/ you can't filter on any other filter (selecting any other filter should reset the build id search box)
2/ it doesn't filter on the build id at all

Right now generated app build is 500k. With uglifyjs, we could strip than down to ~150k.

Currently we let the server choose the records ids (using POST).

@Natim suggested that we could have a relation between a release and its record.

Idea: We could use the download url (which is unique) to compute a uuid for example.

34443782 bytes is quite hard for me to parse.
After some calculation I understood it was 32,85 MB I think that's the value I would like to have in the UI directly. We could however display the bytes value on mouseover (title?acronym?)

S3 has the ability to post events when objects are uploaded and deleted. Since archive.mozilla.org is an S3 bucket, whenever a new build is uploaded we can be notified and buildhub updated in real time.

For the flow:

upload -> s3 -> SNS -> SQS <- buildhub daemon -> buildhub kinto

or

upload -> s3 -> lambda -> buildub kinto

What do you guys think?

I suppose ShipIt or Balrog will be the source of this.
How do we pull it into Build Hub?

It's redundant with the "channels" filter

We used to build the list of values for filters dynamically, using the list of build records we had at any one point. This made sure we had no missing value, and also that we'd always present a relevant list of filter values to the user, given a list of build records.

However, this isn't scalable: we can't load the full list of build records in memory on the client, and then go through them to build the list of filter values.

A first step towards being able to navigate through the whole data set is to harcode the filter values.

We're experimenting with using elasticsearch to do the indexing/faceting (see https://github.com/Kinto/kinto-elasticsearch) which might replace those hardcoded values in the future.

So it seems that filtering on a build id is also impacted by Kinto/kinto#1217 :

I added quotes around the build id, but then it seems only searches that are numbers give results.
Searching for "2017" gives a few results, including 2017-06-01T20:32:00.

But searching for "2017-06-01T20:32:00" gives no results. Same for "2017-" or even just "T".

@Natim any clue what's happening?

It seems that pulse does not meet our requirements. For posterity, @leplatrem could you describe the reasons that we decided against using it? Also describe what we prefer to do instead.

Last month (Apr 24) I talked to @willkg about Socorro's (Crashstats) usage of build info. Socorro currently has an ftpscraper.py that indexes archive.mozilla.org. This populates a database table that's used by various tools for lookup.

BuildHub should be able to replace their scraper and their custom database table.

Of the 1082 records at the time of this writing, there's exactly 0 with a specified "channel" field. Either it's not an information we get back from pulse on the messages we're interested in, either there's an issue when storing those in kinto?

This issue is to discuss how we want to structure the project to make it easy to configure and deploy for the operations team. Currently there are are server side and client side components. The server side components are daemons and cron job compatible scrapers. The client side is a static HTML website.

Data is stored in a kinto instance which is backed by Postgresql RDS. This is relatively slow moving and shouldn't require much operational work other than standing it up and keeping it updated.

Server side code

• ship a container with all server based code
• follow dockerflow
• Build/test container in circle.ci
• Publish container to hub.docker.com/mozilla/buildhub
• A single container with all of the scripts / workers / scrapers / etc
• version.json written into the container, accessible by run.sh version
• a run.sh file as the container's ENTRYPOINT.

run.sh

Buildhub's docker container is a toolbox and run.sh provides an abstraction to the tools inside. Here are an examples of how run.sh works as the ENTRYPOINT:

Self documenting...
$ docker run buildhub:master help


On the CLI or as cronjob: 
$ docker run buildhub:master scrape_archives --auth user:password


As a daemon: 
$ docker run --envfile /etc/dockerflow/buildhub-pulse.txt \
    buildhub:master pulse_listener

Web UI

The WebUI (SearchKit: #127) also needs to be deployed.

• Hosted in S3 fronted by Cloudfront. On the https://buildhub.services.mozilla.com domain?
• Automatic deployment from Circle => Stage|Prod would be nice to have

At the moment, a request is sent to Kinto on each keypress. We should rather debounce it (only send a request whenever no keypress was detected for 200ms or more, or add a "search" button).

The URI address in the browser should update when we change filters. This would make it easier for other web tools (mozreview, bugs, phrabricator, etc) to link back to build/release information.

It seems that scrape_archives is using android-arm as the platform.

However is there two platform for android? android-api-9 and android-api-15? Can we have the same release compatible with both?

Since the platform is used in the ID of the record, we need to make sure that android-api-9 releases won't be overridden by android-api-15 ones and vice et versa.

Trying to filter on certain version numbers results in the following error message in the console:

An error occured while fetching the build records: KintoError 503 "Service Unavailable" { errno = 201, message = "Service temporary unavailable due to overloading or maintenance, please retry later.", code = 503, error = "Service Unavailable" }

The request sent to Kinto is: https://kinto-ota.dev.mozaws.net/v1/buckets/build-hub/collections/releases/records?_limit=10&_sort=-build.date&target.version=0.8

If however you filter on version 53.0b1 there's no problem: https://kinto-ota.dev.mozaws.net/v1/buckets/build-hub/collections/releases/records?_limit=10&_sort=-build.date&target.version=53.0b1

https://archive.mozilla.org/pub/mobile/nightly/2017/05/2017-05-30-10-01-27-mozilla-central-android-api-15-old-id/fennec-55.0a1.multi.android-arm.apk

The generation of a unique ID for this URL is not unique.

It set the following ID: fennec_2017-05-30-10-01-27_55-0a1_android-arm_multi while it should set fennec_2017-05-30-10-01-27_55-0a1_android-arm_multi

We have other example of another URL that returns the exact same ID, leading to missing builds:

• https://archive.mozilla.org/pub/mobile/nightly/2017/05/2017-05-30-10-01-27-mozilla-central-android-x86-old-id/fennec-55.0a1.multi.android-i386.apk

The latter should set: fennec_2017-05-30-10-01-27_55-0a1_android-i386_multi

So that searching for Elm bits is easier.

In the Wednesday meeting @magopian mentioned a new structure/design for the WebUI. Can we document the design / decisions for it on GH?

Currently, we take the highest version on the remote storage to resume a previous run.

It should work ok as long as things go well. But if a scraping task crashes, it can leave some records out for specific locales, platforms or whatever.

We should elaborate a clear strategy to allow idempotence/resuming efficient :)

In order to show the possibility of the underlying APIs, we could have a static page that:

• Shows a curl example to retrieve the revision of a buildid (for example)
• Shows kinto-http.(py|js) snippets to query the API (simple stuff like list of locales for a version)
• Links to Kinto API docs
• Info to contact us (IRC,

Thoughts?

#32 (comment)

We sometimes use build, archive, release... We should take some time to define a consistent vocabulary for our documentation, UI, variables....

Last month (apr 17) I discussed with @mreid-moz for telemetry requirements. Telemetry's main requirement is being able to cross reference the build id/release channel in a telemetry ping back to a commit in mercurial.

Summary of requirements:

• A single source of truth to identify a build. They often get data from non official build and release version of firefox.
• With a build id/release channel be able to look up the commit hash for a release

The blue print contains the detailed meeting notes.