mozilla-releng / bouncerscript Goto Github PK

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please reach out to [email protected].

(Message COC001)

We need to make sure that all possible behavior tasks handled by bouncerscript are being covered in integration tests.

Something similar to what @JohanLorenzo did under https://github.com/mozilla-releng/shipitscript/blob/master/shipitscript/test/integration/test_integration_script.py

In https://github.com/mozilla-releng/bouncerscript/blob/master/bouncerscript/task.py#L81 we're checking against cross-channel updates. We should also include some versioning checks. The existing vs the one about-to-be-pushed.

One scenario that I'm thinking is accidentally push (e.g.) Firefox 52.0b12 when 62.0b12 is expected. I could use @JohanLorenzo's https://github.com/mozilla-releng/mozilla-version/ for making these checks.

Including button and reference in README as scriptworker has.

The version bump and changelog entry should (probably) be included to the PR for a change, now that the master branch is protected and changes require a review. Or I don't have sufficient rights.

Sort of redundant. The implementation is not too nice right now. We need to address this later on to improve it. More details here #12

More here

Particularly useful in cases where we have two dot releases X.0.1 and X.0.2 triggered and shipped in consecutive days and than potentially rerun the bouncer aliases from X.0.1 that would overwrite the latest ones.

Callek> mtaba.ra: [defer until after deved b1 is done] -- is 2018-03-02 19:16:02,528 - bouncerscript.utils - INFO - Performing a GET request to https://bounceradmin.mozilla.com//product_show?product=Devedition-60.0b1-Partial-59.0b13 with kwargs {'timeout': 60} at the start of the bouncer sub log merely to check for a 404? if so, wouldn't a HEAD request be better?
13:12:03 [so we can avoid the html output in log, if nothing else]
13:13:01 sure, good improvement there if we want that. was GET in the old behavior so didn't improve that but sounds like a good issue

@JohanLorenzo fixed mozilla-releng/mozilla-version#62 so we can now leverage some of that in bouncerscript to reuse the code, to check if two versions are successive, before we bump the nighty-latest locations (both in Firefox for central and Fennec for esr)

This it to track that.

• Use Docker & Docker Compose for running a local dev environment, based on the python:3 image
• Use pip-compile-multi, and Pyup.io for dependency management
  • See https://github.com/mozilla-releng/releng-rfcs/blob/master/rfcs/0022-app-best-practices.md#dependency-management for requirements file details.
• Use tox as a test runner
• Use Taskcluster for CI
• Ensure production pushes happen manually, in response to bugs filed
• Ensure stage pushes happen in response to balrog_stage tags on Dockerhub
• Eliminate the development environment, if it exists

(Remove items that are not applicable to this project).

More of this conversation here: #16 (diff)

Basically, once we have TB added in bouncer, we'll need to perform additional checks to make sure we're not mistakenly updateing cross-product aliases (TB to update firefox aliases via bad payload and vice-versa). Can do this via scopes parsing + checks like in scriptworker.

mozilla-releng/mozilla-version#61 will add support in mozilla-version for nightly regex so that we can trim some of the code duplication in this project. This is to track that.

While shipping TB60, bouncerscript sanity caught some error that thunderbird-esr-next-latest didn't exist on bouncer. Which was true. We don't have that neither in bouncerscript regexes, nor in bouncer.
While this issue alone is dealt with in separate bugs, I did want to take a moment and double-check we're up-to-date with entries in bouncer aliases.

Ran a quick script to double-check what we currently have under https://github.com/mozilla-releng/bouncerscript/blob/master/bouncerscript/constants.py#L1 against official bouncer entries and I got:

(Pdb) pp official_set.difference(our_set)
{'firefox-esr52-latest-ssl', 'firefox-esr52-latest'}
(Pdb) pp our_set.difference(official_set)
{'thunderbird-latest-ssl',
 'thunderbird-next-latest',
 'thunderbird-next-latest-ssl'}

So:

1. We don't have the esr52 aliases included here. That's expected since that's not subject to bouncerscript these days.
2. We have three additional aliases defined in our regexes that are nowhere to be found in bouncer, so most likely never used by other pieces of automation.

@tomprince: are we okay with removing these three aliases below from our sanity validation check?
CCing @JohanLorenzo

{'thunderbird-latest-ssl',
 'thunderbird-next-latest',
 'thunderbird-next-latest-ssl'}

Original conversation here.

What @escapewindow said lastly there is as following:

1. We can put the regexes directly into bouncer_scriptworker/templates/script_config.json.erb, which means it's closer to developer -> json -> python regex. If there's another format that works better, e.g. yaml, we could switch script_config.json.erb to script_config.yml.erb. scriptworker::instance takes a $task_script_config argument that defaults to ${basedir}/script_config.json but can be any file path. In this example, it would be more developer -> yaml -> python regex

   • We can even create a python file in puppet, not a template. As long as we point to and use that file's regexes, it then becomes developer -> python regex, although we use puppet to deploy that python regex.

2. For the [now-obsolete] scriptworker docker sha allowlists, we had them in scriptworker.constants.DEFAULT_CONFIG, allowing for testing in unit tests. We also overrode them in puppet's config. And whenever we updated the puppet config, we also submitted a PR to update scriptworker.constants.DEFAULT_CONFIG to match.

   • downside: this means 2x as many PRs to deploy
   • upside: we can local unittest and end-to-end test
   • upside: when a new docker sha busted the trees, we could deploy the fix quickly via puppet, and were not required to deploy the scriptworker release immediately. The scriptworker patch could ride along with the next scheduled scriptworker release.

Having said that, I don't expect these bouncer regexes to change as often as the scriptworker docker shas did. I just want to point out that we still have options here.

In https://tools.taskcluster.net/groups/SdybP-zoSbaWp1x6FdHW1A/tasks/Y86FlD3cSxqo8HiRSvoUdQ/details we actually failed to submit but task was green. This is so wrong. We need to proper test the return status code or something. Or process the input and validate better to prevent this in the future.