We're supposed to be using https://docs.netlify.com/routing/redirects/ to route all URLs to the home page but looking at the analytics (https://app.netlify.com/sites/pytorch-ci-hud/analytics) and monitoring the site itself when it loads it's clear that it's still hitting the 404 page -> redirect pathway. This isn't a big deal but fixing this will remove a round trip (the one to the 404 page that redirects to the relevant page) and make load times better.

Currently, PyTorch has two different GitHub Actions workflows containing a job named test:

• Test tools
• Linux CI (pytorch-linux-xenial-py3.6-gcc5.4)
  

Since the HUD uses only the job name, these two workflows both get put into the same column, which in practice seems to mean that only the signal for the latter is shown, while the former gets ignored.

It would be better if GitHub Actions job names were prefixed by the workflow name as they are on GitHub itself, so those columns would instead be called "Test tools / test" and "Linux CI (pytorch-linux-xenial-py3.6-gcc5.4) / test" respectively.

cc @seemethere @malfet

We still rely a bunch on CircleCI and will for at least the coming weeks. It'd be nice if it was added to the list of runs alongside GHA. CircleCI is more liberal with un-auth'ed API requests than GitHub so we may be able to just query it directly without having the user authenticate. If that doesn't work, we could have a AWS lambda that generates the data for each commit, but this would be tricky and it's not clear if this would allow downloading logs.

https://hud.pytorch.org/commit/e69d82a9cc8dcbff92b838c1afca11e4b1f9fe95

Here is the console error:

These duplicate a lot of code and we could combine them

e.g., https://hud.pytorch.org/commit/3b4cb9ddca1cc2b92277541ff781730042bee3c5

Feature Request

I'd like it if we could have functionality for HUD to render test reports that are uploaded to Github artifacts instead of S3, we have some set of jobs that won't have access to S3 to upload to so it'd be nice to have the functionality for both

Feature Request

For each repo we should generate a status badge image that can be used in the HUD itself for a quick view on how the latest commit is faring. We could also embed this in the README

The badge itself would probably have to be generated in GHA and stored in S3 as an image, then the HUD / README can just download it as necessary.

Visiting a commit specific hud page like https://hud.pytorch.org/commit/945d40dca6e6fc98ccbc209d7d19e99ac8d4badd doesn't show the HUD for the specific commit but actually ends up showing the latest commit on the default branch of pytorch/pytorch

Right now groups are just a bolded version of whatever their status is which is not always easy to identify in context. We also now have overloaded behavior for clicking, where in some cases it goes to a new webpage and in other it expands the group (which doesn't respond immediately either).

We should add something to groups to make them stand out more

Indicate at the top of the page somehow the aggregate status of workflows

Right now we list jobs then artifacts as separate steps, but all artifacts come from a specific job, so we should try to match these up.

e.g. on https://github.com/pytorch/pytorch/pull/64303/checks?check_run_id=3562967932, the workflow is Lint, the job is mypy, and the step is Run mypy. Right now the logs stop at jobs, but we should let users view at the step level since we often finish each job with some cleanup that creates lots of log spew.

We might need to fetch the entire log archive for the workflow which may be prohibitively large, or we could manually parse out the ##[group] and ##[endgroup] sigils that GitHub adds to logs (this is probably best if JS can handle it)

it redirects straight to /build2/pytorch-master when dialed directly, but when clicking on "status" from that page it loads correctly.

https://hud.pytorch.org/status

semi-related: the JS for the page isn't updating for me on non-incognito windows (i.e. I don't see any of the charts I do when I visit the same page in incognito on Chrome)

We should put the failing jobs first, followed by successful ones, then skipped/grouped jobs

Currently if I click on the tests button on https://hud.pytorch.org/pr/65178, it crashes the site:

Was removed by bc70547 which makes it impossible to see docker-build jobs.

The status page doesn’t render and just shows a white screen, we’re probably missing some guards somewhere for browser APIs that aren’t on mobile (like Notification)

Using iOS + macbook safari to debug should be pretty simple, maybe we could also mock out the lack of APIs in our tests so too

We should add a button for GHA workflows on PRs that:

1. Adds the with-ssh label to the associated PR
2. Triggers a re-run of the workflow
3. Monitors workflow annotations and reports the SSH command when its available

For example, for https://hud.pytorch.org/pr/57331 it now shows both original and re-run of the same workflow, see screenshot below:

First link points to https://github.com/pytorch/pytorch/actions/runs/1284854980, 2nd link to https://github.com/pytorch/pytorch/actions/runs/1284845754

Appears as though artifacts are not showing up on the commit/pr page where they previously were showing up

We should highlight if there is a test suite / name combo that occurs more than once in the results for a specific job (across shards)

Right now it goes way off my laptop's screen and has come up a few times (see #68), a couple of solutions we came up with internally:

• freeze PR number column so you don't have to scroll around to get it
• add option to hide columns where all builds are both present + passed

Feature Request

Currently it is hard to discover where is the doc error: https://hud.pytorch.org/commit/pytorch/pytorch/364c4959c3e0d0f226958e43fcbe55f758c0f4d4

It would be really helpful to extract the test error in build-docs (python) or Doc Build and Push so it's easier to discover.

Sorting should help with this, but a background color or something could make it easier to see

Was trying to see an overview of all recent PRs' CI status, but the following url is pulling too many requests

https://hud.pytorch.org/build1/pytorch-pull-request

Infra failures, for example, could be marked as something other than red X's, to make it easier for people to focus their attention on new/unknown/genuine failures.

Another possibility (as opposed to identifying these automatically) would be to just let people update the HUD UX to mark failures as erroneous or reported/mitigated to reduce redundant work.

Test shards are not a logical distinction in CI jobs, they're an artifact of us increasing parallelization in our workflows. We could de-duplicate these on the HUD to compress some info:

• aggregate status
• merge test reports

Access to PyTorch build artifacts from hud.pytorch.org/pr/${PR_NUM} is not working after GitHub login and authentication. Page displayed is the same after refreshing except for the authorize_github code. This occurs on incognito/anonymous browser modes on different browsers such as Chrome, Edge and Firefox.

The build1 page nicely shows all the commit corresponding to a CI run. But build2 doesn't.
It would be helpful to at least get a visual marker on build2 indicating that this is a bundle of commits (so people should check build1 for other commits) or even better have the same multi-line visualization to know what are the other commits.

This wouldn't replace pytorch's status as the default page, but it would be +1 in the hierarchy. #152 adds support for domain libraries, so there should be a way that these are all discoverable:

• use S3 api to list repos + branches
• provide links to each

see https://hud.pytorch.org/pr/63641, this job shows it's pending but it's completed in GHA: https://github.com/pytorch/pytorch/runs/3416072438

🐛 Bug

Groupings for pytorch/pytorch appear to be broken

Feature Request

Right now the same failures get listed multiple times, it would be nice to consolidate these in some way if the text of the report is exactly the same:

ex. https://hud.pytorch.org/commit/pytorch/pytorch/89b02fc70b6a279b602545fddbcd56b4248cee4f

🐛 Bug

For Group jobs, ci/circle: binary_windows in hud.pytorch.org.
The aggregate status are always red, but all sub jobs are green.

To Reproduce

From the hud.pytorch.org, select and set the Name filter as Windows

Expected behavior

If all sub jobs are green, the aggregate status should be green.

Additional details / screenshot

It looks the some circle jobs are still included. For example, https://circleci.com/gh/pytorch/pytorch/17013182?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link, their status are canceled though they're not shown in the UI.
Since all nightly jobs are transferred to Github actions, the backend should not return circleci jobs.

We could leverage Dr. CI's ruleset / identification to locate existing failures. We can then highlight these on the HUD, use them in PRs to auto-scroll logs to the relevant line, and mark certain failures as already broken / flaky. We'd need some way to extract that information from Dr. CI which we can probably grab from the GitHub comment (and if we're reporting it on the HUD we can hide it behind a <details> tag in the comment.

Currently probot adds a comment (example to all (non opted-out) PyTorch PRs with information about CIFlow. We should integrate this into the PR HUD and use the authorization token that we already have to control the labels on a PR. In order, we can:

• implement the same view as the comment but with click-to-add-a-label functionality as its own React component
• compress the view with some interactive features
• answer questions like: what labels do I need to add X jobs

See https://hud.pytorch.org/ci/PyTorchLightning/pytorch-lightning/master
Because url can be null here

We can iframe these in from metrics.pytorch.org's default dashboard, we should stick to metrics that general PyTorch developers would care about when developing a PR:

• master failure % (a, b)
• queue time
• average TTS per build

as well as a link to a wiki page on how to use metrics.pytorch.org for PyTorch developers

We could at the very least query PR commits -> runs on those refs, but this misses job re-runs on a specific commit. These would be useful to revisit historic failures (especially if we do a better job encoding things in the URL these can be easily shared / referenced)

They should either expand the group or go straight to one of the failing jobs in the group

All user intractable elements should encode their state in the URL so pages can be shared, for example:

PR Page

• current workflow
• expanded test results
• expanded log scroll position

HUD

• name filter
• group jobs checkbox

🐛 Bug

Wrong pull request number is rendered in a HUD

To Reproduce

Go to https://hud.pytorch.org/ci/pytorch/pytorch/master?name_filter=macos_10_15 and notice pull request with issue#2

Expected behavior

HUD should have picked the right issue number

Additional details / screenshot

Feature Request

Add a toggle-able dark mode. It should respect the OS preference first but be toggle-able and saved in localStorage. We should start using SCSS too to make this easier. For color scheme-ing we can probably just copy all the colors from GitHub's dark/light mode