nix-community / buildbot-nix Goto Github PK

Right now, CI runs can only be triggered via push by configuring web hooks or similiar integrations.
With pull based runs buildbot checks a repo periodically for changes.

upsides of this:

• can work with any git server without doing integration work.
• doesn't need to have a publicly reachable server for webhook handling.

Started in #68

Reported by @nikstur in nix-community

We should document what minimal permissions a token needs:

Requests in buildbot-nix:

• GET "https://api.github.com/repos/{owner}/{repo}/hooks?per_page=100"
• GET "https://api.github.com/user/repos?per_page=100"
• POST "https://api.github.com/repos/{owner}/{repo}/hooks"

Requests in buildbot:

GitHubStatusPush:

• POST "https://api.github.com/repos/{owner}/{repo}/statuses"

AvatarGitHub:

• GET "https://api.github.com/users/{username}'
• GET "https://api.github.com/search/users"
• GET "https://api.github.com/search/commits"

GitHubEventHandler:

• GET "https://api.github.com/repos/{repo}/pulls/{number}/files"
• GET "https://api.github.com/repos/{repo}/commits/{sha}"

ngi-nix/ngipkgs will migrate to buildbot ngi-nix/ngipkgs#200. It's a monorepo that contains derivations for various NGI Projects. They all somehow consume nixpkgs, but it is rare that NGI Projects depend on each other. Thus, when viewed as a DAG of derivations, it looks roughly like this:

Ignoring references to nixpkgs (in the build context: assume it's cached), the repo decomposes into multiple weakly connected components. Now, my thesis is that (especially if you have multiple workers on multiple machines), it'd be beneficial not to create one build per derivation, but rather one build per weakly connected component of derivations. With the above diagram, you would not get 9 builds but 3.

I have implemented such an analysis in Python using NetworkX on top of nix-eval-jobs that generates a build matrix for GitHub Actions, see .../build_matrix/main.py. This results in workflows like the following:

• Run 8926962816: With cache, just builds one package (and sanity checks).
• Run 8744345769: Without cache (for testing purposes), fans out to many jobs, but note that they are not strictly one per derivation, e.g. " build kbin, kbin-frontend-0.0.1, kbin-0.0.1" is a build for 3 derivations.

Since I am not (yet) a user of buildbot-nix, I might be missing something, but...

• AFAIK buildbot-nix does not make such an analysis. Is that on purpose, i.e. do you think there's no benefit in doing that? If yes, I'd be curious to learn why?
• Is this something you would like to see upstreamed to buildbot-nix? If yes, should it be hidden behind some configuration? Which?

We currently do one retry, which is fine.
However if a future push shows that the derivation failed in the past, we should cache the failure.
If possible we should add a retry button to retry the derivation anyway.

It looks like github sometimes takes a while to report us the merge branch.
So far we work around by retrying on network failure. However this will always
fail for merge conflicts. In those case it's better to manually merge in the CI
and report proper errors why a pull request cannot be build.

Hi there

I was hoping for an alternative to Hydra that connects to something other than Github and even to be able to use private repos?

You mention plans to support Gitea, do you still have those plans at some stage?

Thanks.

I've observed random failures, where GitHub would deny the JWT or installation tokens, due to them being expired or being improperly generated. @Mic92 has ran into it too.

The code needs a checkup.

It's a lot less friction to just be able to create a github app and install it in an organization.
A github app can also request a time-bound github token like this: https://github.com/NixOS/nixpkgs-merge-bot/blob/cae47034242b6604db5199a73e5f4ce5121ffa74/nixpkgs_merge_bot/github/GitHubClient.py#L249
It needs to be regularly refreshed

That's one feature that is nice in Hydra.

When building reports and things like that, it's nice to display them in the UI directly (less pleasant: all of the CSRF issues that it implies).

Something is not happening for me with a private Gitea instance, and I am not sure if it's a config issue.

I have temporarily created a token with ALL access to test this.

I get the following error when buildbot attempts to access my repo (buildbot-mannuk is an ORG);

buildbot_nix.common.HttpError: Request for GET https://git.domain.com/api/v1/repos/buildbot-mannuk/nix-config/topics failed with 403 Forbidden: {"message":"Only signed in user is allowed to call APIs."}

When I ssh into the node running buildbot and curl the Gitea API it succeeds, and I am unclear why;

~~> curl -X 'GET' \
          'https://git.domain.com/api/v1/repos/buildbot-mannuk/nix-config/topics?access_token=< tokenFile contents >' \
          -H 'accept: application/json'

Response below which is correct;

{"topics":["buildbot-mannuk"]}

Buildbot service snippet;

    admins = [
      "mannuk-buildbot"
    ];
    authBackend = "gitea";
    gitea = {
      enable = true;
      instanceUrl = "https://git.domain.com";
      topic = "buildbot-mannuk";
      tokenFile = config.sops.secrets."buildbot/gitea-token".path;
      webhookSecretFile = config.sops.secrets."buildbot/gitea-webhook-secret".path;
      oauthSecretFile = config.sops.secrets."buildbot/gitea-oauth-secret".path;
      oauthId = "redacted";
    };

Using the gitea swagger api, I note that the api only works if the repo owner is specified. Does this mean that the buildbot user has to own the repo being built?

Any thoughts as to what's going on? Thanks.

It would be nice to display warnings so they stand out in the UI.

Right now if the user doesn't look at the logs, they might miss deprecation issues for example.

Some generic nix copy would be great We might need some secret support for things like s3 credentials or ssh keys.

In some settings, we only want to show the builds to users with access to the repos.

Is it possible to map the repo access control to the authenticated users?

If not, create a "private" mode where only logged-in users can see the builds and builders.

If GitHub is unavailable on boot, buildbot will fail ungracefully right now.

Add more coverage to detect regressions.

When clicking around in buildbot, the user wants to be able to navigate back to the repository where the builds have been triggered from.

Buildbot is hitting some rate limit. I noticed that as treefmt-nix grew, it started having issues reporting build statuses.

Eg: numtide/treefmt-nix#185 (comment)

Jun 11 20:10:59 bld2 twistd[2847473]: 2024-06-11T20:10:59+0000 [HTTP11ClientProtocol (BufferingTLSTransport),client] Failed to update "success" for numtide/treefmt-nix at b303c743d853af36b8bd75b4d83b48ed7ab0747a, context "buildbot/nix-build .#checks.aarch64-linux.formatter-rufo", issue 185. http 403, b'{\n  "documentation_url": "https://docs.github.com/free-pro-team@latest/rest/overview/rate-limits-for-the-rest-api#about-secondary-rate-limits",\n  "message": "You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID 9748:27DC3C:961101D:9705126:6668AF52."\n}'
Jun 11 20:10:59 bld2 twistd[2847473]:         Traceback (most recent call last):
Jun 11 20:10:59 bld2 twistd[2847473]:           File "/nix/store/w869q04ijifzfax4hysjvnall3kgah6d-python3-3.11.9-env/lib/python3.11/site-packages/twisted/internet/defer.py", line 1078, in _runCallbacks
Jun 11 20:10:59 bld2 twistd[2847473]:             current.result = callback(  # type: ignore[misc]
Jun 11 20:10:59 bld2 twistd[2847473]:           File "/nix/store/w869q04ijifzfax4hysjvnall3kgah6d-python3-3.11.9-env/lib/python3.11/site-packages/twisted/internet/defer.py", line 1949, in _gotResultInlineCallbacks
Jun 11 20:10:59 bld2 twistd[2847473]:             _inlineCallbacks(r, gen, status, context)
Jun 11 20:10:59 bld2 twistd[2847473]:           File "/nix/store/w869q04ijifzfax4hysjvnall3kgah6d-python3-3.11.9-env/lib/python3.11/site-packages/twisted/internet/defer.py", line 2003, in _inlineCallbacks
Jun 11 20:10:59 bld2 twistd[2847473]:             result = context.run(gen.send, result)
Jun 11 20:10:59 bld2 twistd[2847473]:           File "/nix/store/w869q04ijifzfax4hysjvnall3kgah6d-python3-3.11.9-env/lib/python3.11/site-packages/buildbot/reporters/github.py", line 234, in sendMessage
Jun 11 20:10:59 bld2 twistd[2847473]:             log.err(
Jun 11 20:10:59 bld2 twistd[2847473]:         --- <exception caught here> ---
Jun 11 20:10:59 bld2 twistd[2847473]:           File "/nix/store/w869q04ijifzfax4hysjvnall3kgah6d-python3-3.11.9-env/lib/python3.11/site-packages/buildbot/reporters/github.py", line 222, in sendMessage
Jun 11 20:10:59 bld2 twistd[2847473]:             raise RuntimeError()
Jun 11 20:10:59 bld2 twistd[2847473]:         builtins.RuntimeError:

Right now buildbot's npm frontend mess is getting copied directly from the release. This makes it harder for us to experiment with UI changes.

README.md assumes "classic" PATs. During set up of buildbot-nix, @Erethon and me noticed that it would probably be enough to create a Fine-Grained Personal Access Token with the following permissions for the buildbot GitHub Account:

• Repository Permissions for "Webhooks" (read and write).
• Repository Permissions for "Commit Statuses" (read and write).
• Repository Permissions for "Contents" (read). We are configuring a public repo, so did not have to set this, but I guess that this is what would be needed for a private repo. StackOverflow seems to agree.

The account also needs "Write" Permissions on the repository. This seems to work fine. What's preventing their use is an explicit check in buildbot-nix that guards repository discovery:

https://github.com/Mic92/buildbot-nix/blob/077a60a5d040f7161c50a83d827db77a2a68ed9f/buildbot_nix/github_projects.py#L270-L274

This checking method is incompatible with Fine-Grained PATs. The response looks like this:

{ "permissions": { "admin": false, "maintain": false, "push": false, "triage": false, "pull": true } }

One downside of Fine-Grained PATs is that such a check apparently cannot be implemented.

However, in case buildbot runs into a HTTP 401, it cloud also expose the contents of the response header X-Accepted-GitHub-Permissions in the logs:

To help you choose the correct permissions, you will receive the X-Accepted-GitHub-Permissions header in the REST API response. The header will tell you what permissions are required in order to access the endpoint.

A benefit is that, well, it's true that permissions can be set in a much more fine grained way. It avoids having to give the bot user account admin permissions on the repo(s).

Our workaround was to switch buildbot from a "classic" PAT to a Fine-Grained PAT after it had installed the Webhook. Another option would of course be to patch away the above check.

Here's how that looks in Org settings at https://github.com/organizations/${NAME}/settings/personal-access-tokens/:

I noticed the following interaction that causes attrs to not get gcroot-ed when you would expect them to be:

1. Open a pull request bumping any check attribute's dependencies
2. buildbot-nix runs and builds the attr. It doesn't register a gcroot because the build runs on the pull request's branch, which isn't the repo's default branch
3. Merge the pull request into the default branch, while the check attr remains unchanged.
4. buildot-nix runs, but notices that the attr is cached; it skips the nix build of the attr, and doesn't gcroot the out_path for the attr (because the skipped builds don't have the gcroot-adding step).

This interaction has resulted in my check attrs getting new revisions several times, while the corresponding gcroot still points to versions multiple days out of date.

I think one way to potentially fix this would be to have the gcroot step run on the skipped builder also: It is slightly risky (the attr might be "unchanged" according to the scheduler but not be present in the store due to a mistimed gc run)... but this is already happening, I think.

https://docs.buildbot.net/current/manual/configuration/www.html#badges