It's pretty common for a few different parts of the generation process to want to edit the same roles. This is especially common with user-related roles such as github-team:.... For example, a resouce generator for github projects might want to add github-related scopes to a role github-team:taskcluster/core, and another resource generator might want to add hook-related scopes to that same group.

Right now, handling this is tricky because a role can only be added to a Resources object once, so it's up to you to determine all of the required scopes before creating that resource.

Let's find a more convenient way to just "add" a scope to a role, creating that role if it doesn't already exist.

Move development from the master branch to main.

This is probably due to the version of diff I'm using which is relatively new since I'm on archlinux. On tc-admin diff I'm getting the following:

  File "/home/eijebong/.local/lib/python3.10/site-packages/tcadmin/util/sessions.py", line 26, in wrap
    return await fn(*args, **kwargs)
  File "/home/eijebong/.local/lib/python3.10/site-packages/tcadmin/main.py", line 86, in diffCommand
    different = diff.show_diff(expected, actual)
  File "/home/eijebong/.local/lib/python3.10/site-packages/tcadmin/options.py", line 61, in wrap
    return fn(*args, **kwargs)
  File "/home/eijebong/.local/lib/python3.10/site-packages/tcadmin/diff.py", line 156, in show_diff
    result = textual_diff(generated, current, context)
  File "/home/eijebong/.local/lib/python3.10/site-packages/tcadmin/diff.py", line 132, in textual_diff
    return "\n".join(lines)
  File "/home/eijebong/.local/lib/python3.10/site-packages/tcadmin/diff.py", line 130, in <genexpr>
    colors[l[0]](l).rstrip() for l in (line if line else " " for line in lines)
KeyError: '\\'

A quick look shows that my diff tool outputs \ No newline at end of file.
Version of diff: /usr/bin/diff is owned by diffutils 3.8-1.

A quick workaround: Add "\\": lambda s: s, to

Resolver runs everything 100x to try to get better benchmark numbers. That's not necessary!

For renovate to merge a PR, there must be no pending reviews, so let's exempt changes to setup.py from CODEOWNERS.

Black is too opinionated, IMHO :)

Also, its definitions evolve and that seems to cause errors on PRs.

ci-admin defines some pytest fixtures to get the current and generated resources. Those will be universally necessary for writing checks, so let's include that functionality in tcadmin directly.

tc-admin diff just produced unexpectedly large output. Here is one chunk of it:

@@ -421,34 +421,34 @@ WorkerPool=proj-servo/win2016-staging:
                       ],
                       "SubnetId": "subnet-036a0812d241733ef"
                   },
                   "region": "us-west-2",
                   "workerConfig": {
                       "genericWorker": {
                           "config": {
                               "checkForNewDeploymentEverySecs": 600,
-                              "deploymentId": "9a2aa17aa9128db5ee8c14878e668c800e4e1ccdb746fadbf58ea9e854b19e5a",
-                              "ed25519SigningKeyLocation": "C:\\generic-worker\\generic-worker-ed25519-signing-key.key",
-                              "idleTimeoutSecs": 14400,
-                              "livelogExecutable": "C:\\generic-worker\\livelog.exe",
-                              "sentryProject": "generic-worker",
-                              "shutdownMachineOnIdle": true,
-                              "taskclusterProxyExecutable": "C:\\generic-worker\\taskcluster-proxy.exe",
-                              "workerTypeMetadata": {},
-                              "wstAudience": "communitytc",
-                              "wstServerURL": "https://community-websocktunnel.services.mozilla.com"
-                          }
-                      }
-                  }
-              },
-              {
-                  "capacityPerInstance": 1,
-                  "launchConfig": {
-                      "ImageId": "ami-040db55391fa01dd7",
+                              "deploymentId": "6a4410a78638956b1b041e28563d7edab1fb3126700736915672aa35114538e3",
+                              "ed25519SigningKeyLocation": "C:\\generic-worker\\generic-worker-ed25519-signing-key.key",
+                              "idleTimeoutSecs": 14400,
+                              "livelogExecutable": "C:\\generic-worker\\livelog.exe",
+                              "sentryProject": "generic-worker",
+                              "shutdownMachineOnIdle": true,
+                              "taskclusterProxyExecutable": "C:\\generic-worker\\taskcluster-proxy.exe",
+                              "workerTypeMetadata": {},
+                              "wstAudience": "communitytc",
+                              "wstServerURL": "https://community-websocktunnel.services.mozilla.com"
+                          }
+                      }
+                  }
+              },
+              {
+                  "capacityPerInstance": 1,
+                  "launchConfig": {
+                      "ImageId": "ami-0707fecfddd0c8421",
                       "InstanceMarketOptions": {
                           "MarketType": "spot"
                       },
                       "InstanceType": "c5.xlarge",
                       "Placement": {
                           "AvailabilityZone": "us-west-2d"
                       },
                       "SecurityGroupIds": [

As far as I can tell, this is equivalent:

                       ],
                       "SubnetId": "subnet-036a0812d241733ef"
                   },
                   "region": "us-west-2",
                   "workerConfig": {
                       "genericWorker": {
                           "config": {
                               "checkForNewDeploymentEverySecs": 600,
-                              "deploymentId": "9a2aa17aa9128db5ee8c14878e668c800e4e1ccdb746fadbf58ea9e854b19e5a",
+                              "deploymentId": "6a4410a78638956b1b041e28563d7edab1fb3126700736915672aa35114538e3",
                               "ed25519SigningKeyLocation": "C:\\generic-worker\\generic-worker-ed25519-signing-key.key",
                               "idleTimeoutSecs": 14400,
                               "livelogExecutable": "C:\\generic-worker\\livelog.exe",
                               "sentryProject": "generic-worker",
                               "shutdownMachineOnIdle": true,
                               "taskclusterProxyExecutable": "C:\\generic-worker\\taskcluster-proxy.exe",
                               "workerTypeMetadata": {},
                               "wstAudience": "communitytc",
                               "wstServerURL": "https://community-websocktunnel.services.mozilla.com"
                           }
                       }
                   }
               },
               {
                   "capacityPerInstance": 1,
                   "launchConfig": {
-                      "ImageId": "ami-040db55391fa01dd7",
+                      "ImageId": "ami-0707fecfddd0c8421",
                       "InstanceMarketOptions": {
                           "MarketType": "spot"
                       },
                       "InstanceType": "c5.xlarge",
                       "Placement": {
                           "AvailabilityZone": "us-west-2d"
                       },
                       "SecurityGroupIds": [

… which I would much prefer to have. It looks like this output is produced by difflib, which unfortunately doesn’t have any parameter that tweaks the algorithm. Compare with git diff which has --indent-heuristic, --diff-algorithm={patience|minimal|histogram|myers}, etc.

Would it make sense to use something other than difflib?

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

This repository currently has no open or pending branches.

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

In

The current release (v2.6.0) requires PyYAML ~= 5.3.1, which means 5.4 is not accepted. Because PyYAML 5.4 is a security release, there should be a tc-admin update released to include it.

Let's make sure that the various booting options supported by tc-admin all work.

https://phabricator.services.mozilla.com/D115536

fx-ci uses tc-admin to access configuration and other stuff, and does so using some "hidden" features of AppConfig. Let's make a public API that will allow this.

That might be custom subcommand support, or perhaps it's just making AppConfig.as_current a public API and making a public means to parse command-line options.

tc-admin needs a release cut with taskcluster v39 to pick up taskcluster/taskcluster#4109

A project specifying:

taskcluster~=39.2.0
tc-admin~=2.5.1

(which creates an impossible dependency on taskcluster==39.* and taskcluster==30.*) takes a long time to fail due to ongoing issues with the new pip dependency solver. I opened a pip issue for this at pypa/pip#9346.

See mozilla/community-tc-config#384 where a Hook resource had a routes property in its input. Hooks don't have a routes property, so this should have been flagged as an error.

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: Cannot find preset's package (pinDependencies)

The fuzzing-tc project needs to run some actions before or after changes have been applied with tc-admin:

1. cancel running tasks in a worker pool if it will be modified
2. trigger hooks once they are created / modified
3. ... maybe more actions later ?

We could implement that around tc-admin, but i think it could also be done by adding some callbacks functionality in the apply workflow.

An external repository would add some declarations in its local tc-admin.py :

appconfig.callbacks.before_apply.add(clear_tasks, actions=["update", "delete"], resources=[WorkerPool])
appconfig.callbacks.after_apply.add(trigger_hooks, actions=["update", "create"], resources=[Hook])

Each callback function would then receive a list of filtered resources if any matches the actions & resources type filter during the apply execution.

The tcadmin/update.py file seem like the obvious choice to trigger those callbacks.

@djmitche Would you be OK with a PR for this behaviour ?

https://hg.mozilla.org/ci/ci-admin/file/tip/README.md sets the root URL based on --environment, meaning that TASKCLUSTER_ROOT_URL doesn't need to be set for the read-only case. Probably most uses of tc-admin will only talk to a single deployment, and may want to just hard-code the root URL.

Let's allow registration of a callback to execute when the command line has been parsed, but before anything has started running. And let's add a way to set the root URL, overriding the default get-it-from-the-environment behavior.

tc-admin diff is extremely slow. It's not uncommon for it to take 20-30min on the Firefox CI cluster. Back when ci-admin had its own tc-admin code, it had some optimizations for this. Most notably, fast_diff: https://hg.mozilla.org/ci/ci-admin/file/949eacedadb887cc3ec16b6d42c5217504e9fb40/src/tcadmin/diff.py#l79

We should pull those into tc-admin to make things faster for everyone.

The most recent versions are susceptible to errors when a rootUrl has a / at the end. Technically rootUrls should not have a trailing /, but in practice we should be resilient to the presence of /.

Currently, every renovate PR requires a run of ./pipupdate to be pushed before it succeeds.

Since tc-admin is used as a library, does it even make sense to pin requirements? Another package that depends on tc-admin won't even see requirements.txt.

We pinned versions in https://bugzilla.mozilla.org/show_bug.cgi?id=1381870 but I think at that point it wasn't clear that tc-admin would be a library.

Abstract out some of the ciconfig support for loading data used to generate resources, in the form of some utility classes capable of loading things from another repo or local files, and of parsing YAML and creating attrs-based objects.