Much of our infra here points to master, but as we're transitioning to use main, we should support both for a time for a smooth transition.

This includes some lambdas as well as torchci/HUD2/flaky bot code.

@albanD (and others) feel that the padding is too small between columns
@suo Has said in the past they prefer to have smaller padding because it's the only way to see all the jobs on one page.

Solution: Create a setting for a compact mode

Ubuntu 18.04 ships with python 3.6, which does not have PyConfig defined. However, PyTorch uses this definition in torch/csrc/deploy/interpreter/interpreter_impl.cpp. Relevant CI failure

The actual build is able to run on python 3.6 enabled machines because the build step checks out python 3.8 and runs against those headers. CMake file

Since we don't actually want to build PyTorch when running the clang-tidyjob, the workaround would be to bump our Docker base to use Cuda 11.2 with ubuntu 20.04

@malfet has wanted the grouped view setting to be persisted.

it'd be nice to have groupings in the timeline view similar to what hud.pytorch.org already has

Since we already have the ELB metrics, we may want to add more logging related to the squid egress proxy for more visibility and spot potential bottlenecks.

Tracker issue for implementing with-ssh for windows

• Manually experiment with AMI to enable ssh for runneruser built from #91
• Create sshd_config that only disables password auth and tells sshd only to look for keys in ~/.ssh/authorized_keys
• Deploy AMI to seemethere/test-repo and test with seemethere/test-repo#35
• Deploy AMI to pytorch/pytorch, see https://github.com/fairinternal/pytorch-gha-infra/commit/6ecc9ebd1ad4f64195e2fd53a6d780e40bc0e7bc
• Verify that seemethere/add-github-ssh-key works with the windows runners
• Land PR to pytorch/pytorch to enable with-ssh for windows see pytorch/pytorch#63440

See pytorch/pytorch#72372 (comment)

We could probably add functionality to probot to actually allow people to use chatops to apply release notes labels.

Similar to

@pytorchbot label {label_name}

from @malfet

Considering that Windows build in GHA very often fails with CUDA installation, why can't we pre-bake toolkit installation into the Packer script?

As we're migrating pytorch/pytorch to main, we should retire and rename "master" to main in our Grafana queries.

As we go at it, we should remove any outdated/unused tables

Should be one per line instead of one long line

From pytorch/pytorch#73811 (comment)

Potentially related to adding mandatory checks?

• investigate if we can reduce deployment complexity by consolidating on Terraform + AWS instead of (terraform -> ansible -> docker swarm -> nginx -> grafana)
• move logging from json-file to cloudwatch
• add auth via GitHub logins gated on the PyTorch org

For each of the follow query lambdas, change references in name, default value, and within the query to main instead of master. Be careful not to remove any versioning, but try to create a new version for each change so that our infra doesn't break.

commons

• cancelled_jobs
• master_commits
• original_pr_hud_query

metrics

• last_branch_push
• last_successful_workflow
• master_commit_red
• master_commit_red_avg
• master_jobs_red
• master_jobs_red_avg
• reverts
• top_failures

Build MacOS binaries for clang-tidy and upload them to S3. This will require making changes to the setup script since lld is not supported on Macs. Creating a tracking issue since naively opting in to using ld doesn't seem to work.

As described here: #205 (comment)

• We are currently showing git commit summary in the summary view. Probably we should show PR body instead, since it not always the same as the git commit.
• We don't currently have a link to go to the GitHub PR page anywhere. The commit summary has a "GitHub" link that goes to the commit itself, which is generally not what people want.
• We don't currently have a way to get to the PR page from anywhere except a direct link
• We should follow github's path structure for commits and PRs (e.g. /[repoOwner]/[repoName]/pull/[prNumber]). That way someone can change github.com to torch-ci.com and be taken directly to the correct page.

As noted on #198, currently grouping breaks name filters. For example, if I link to any name filter that doesn't match the default grouped view, it will break, e.g. https://hud2.pytorch.org/hud/pytorch/pytorch/master/0?name_filter=docs

This is relatively commonly used functionality, so we should fix it.

Removing a dashboard in Grafana doesn't get reflected in the repo, so the removed dashboard is re-added on the next deploy run.

This dashboard is old and we can remove it, but we need support from the updating script first. To do this, make your changes to https://github.com/pytorch/test-infra/blob/main/aws/websites/metrics.pytorch.org/update_dashboards.py.

1. Get the credentials from secrets_tool and set them in your terminal

export GRAFANA_USER=abc
export GRAFANA_PASSWORD=abc

2. Edit the script to recognize when a dashboard is missing in the API response from Grafana but present in the local filesystem, and delete the local file

3. Verify the correctness by checking git status and seeing that the file was deleted and that the script will notify the GitHub Actions runner by printing ::set-output name=UPDATED_DASHBOARDS::yes

4. Land the code, delete the dashboard for real and watch https://github.com/pytorch/test-infra/actions/workflows/generated-update-grafana-dashboards.yml to make sure it's working correctly (GitHub_Actions_Status.json should be deleted from https://github.com/pytorch/test-infra/tree/main/aws/websites/metrics.pytorch.org/files/dashboards)

T103505698

@malfet has wanted the author in the HUD view.

Implement the automatic refreshing of certificate as per following document:

https://docs.google.com/document/d/1nq3dx-_8wasii1koCkXJDSo3uz_0Ee8DzIS2-j2TOpA/edit#

Possible solution 1:

1. run the metrics github action install periodically (once a month)
2. regenerate new certificate during install
3. disable and remove overwriting of certificates from secrets

Possible solution 2:

1. create monthly cron job to take down the metrics server update certificates and restart
2. disable overwriting of certificates from secrets

The dashboards are pretty unorganized right now and it's not clear what is supposed to be used vs experimental. Some dashboards (e.g. gcc5.4 job times) also have outdated ways to gather metrics. We should use folders and re-name things to be more accurate

Flaky tests bot no longer works as expected after workflow consolidation as the platforms are based on workflow names.

The fix is to use the job name instead

Would be nice to consolidate them all in one place and deploy them centrally. We have lambdas sprawled over a bunch of places:

• ossci-log-analyzer
• the three lambdas to populate job information from GHA, circleci, jenkins
• pytorch/probot is deployed a lambda
• All the lambdas in the aws/ folder

This is currently blocked on our usage of Vercel, which has a 15s timeout for serverless functions. Some of our functions take longer than that, to download big logs and/or analyze them.

To fix:

• We can move to netlify, which had background functions that adhere to the underlying lambda restrictions (15m)
• We can wait; I spoke to a Vercel support person who said they are planning similar functionality to background functions (although no ETA was given)

Currently, some commits are hidden due to the fact that GitHub and ShipIt batch commits per push. The way the current HUD works is by taking the head commit of every push and not mentioning the tag-along commits that don't have CI run for them.

This is misleading for diagnosing issues as one of the tag-along commits could be the culprit, e.g., pytorch/pytorch#71611.

We should agree on a way to visualize these batched commits and implement it into the new HUD.

I think the steps of this task

1. Also return 'commits' as a result of the master_commits query on Rockset
2. Use the new version of the query near

   test-infra/torchci/lib/fetchHud.ts

   Line 51 in 68043cb

   const commitsBySha = _.keyBy(commitQuery.results, "sha");

3. Figure out how we want to display these commits

This URL doesn't actually show results: https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0

@zhouzhuojie

We would need to create alerts to metrics.pytorch.org when the server becomes unreachable for X (5?) minutes or more.
Alerts should be sent to pytorch_dev_infra oncall

Currently, the workflows that automatically update slow tests and disabled tests statistics commit the changes to main. To minimize noise, it would be useful to migrate these to push to a different branch, like maybe branch stats-update or something. We can later merge stats-update into main if desired.

Steps:

• Update the workflow files https://github.com/pytorch/test-infra/blob/main/.github/workflows/update_disabled_tests.yml and https://github.com/pytorch/test-infra/blob/main/.github/workflows/update_slow_tests.yml to push to new branch
• Update the references in pytorch/pytorch to read from the new branch instead of main
• Stretch: add a workflow to automatically merge the new branch into main.

I'd like for us to automate the packer builds for our windows images through Github Actions:

• Getting packer build policies into our AWS account for use with Managed IAM (Meta Employees: see internal wiki on how to generate these credentials)
• Write Github Actions Workflow with credentials from Managed IAM to facilitate building packer images in CI (will only really work on non-forked PRs) (code for building packer images can be found here, https://github.com/pytorch/test-infra/tree/main/aws/ami/windows)

This is needed in order to remove manual step in CUDA upgrade. 6. Generate new Windows AMI, test and deploy to canary and prod.

If every job in a group doesn't exist, it should aggregate to a grey o, not a green o. Since we skip all jobs for stacked diffs, we frequently are showing misleading green signal. For example, in this image the linux jobs have been failing consistently, but it looks like there is green interspersed with red because of this bug:

We have too many refs and branches and checking out pytorch takes 7min, which is very slow. Try to not do this in CI.

The installer script in pytorch/pytorch uses a sha256sum hash of the binary for verification. Each time we build a new binary, however, that hash gets invalidated. This causes our CI to fail.

We need to implement versioned binaries which provides us with a window of time to update the hash while still keeping CI active.

https://github.com/pytorch/test-infra/runs/2768685177

the red revert button links to bouncycastle. Now with our migration to GitHub first, there are commits that don't have a corresponding diff to revert, the link is broken (points to https://www.internalfb.com/intern/test/bouncycastle/?arcanist_name=fbsource&revision_or_diff_id=null). We should probably detect if a commit came from GitHub-only and have a different revert flow.

The disabled tests json is updated through a GHA workflow that runs every 15 min. This causes many commits to the test-infra repo, which adds unnecessary noise + bloats our commit history.

We could remedy this by:

1. Hosting the disabled tests elsewhere, like on S3. This would effectively remove the noise in our commit history, but we would lose the previous benefits of having it on a GitHub repo. For example, it would be harder to modify and update the file manually, and there would be no review process when that occurs. Secondly, version control would be harder to visualize for an S3 bucket compared to with Git, which is a system built for versioning.

2. Looking back on a few commits, such as d18b417, it seems as if the contents for the disabled tests have not actually changed, but it's been the order of the json arrays. If there's an easy way to sort the json before we commit, we could also reduce noise by only making commits when the actual contents change.

Please comment which alternative you think is better!

This is a lacking area as we currently have no easy way to access test history without sorting through the data ourselves.

Some things we may want to be able to know from a test view:

• When was the last time this test case was run and on what platforms?
• What branch/PR was the test case was run on?
• How often does this test pass?
• Is this test flaky?
• How many tests are we running per commit on trunk?

Other requirements for the test view:

• Publicly accessible
• Easy to navigate for someone without SQL knowledge.

As a software developer I would like to make sure grafana server logs are forwarded to host machine. This way in case docker image is killed or we need redeployment of docker image, the logs are not lost.

Metrics add usage statistics about metrics website use ultimately we want to display the usage statistics per dashboard.

Signed In users vs not signed , which dashboards are viewed and number of page views. Very similar to google analytics

Have a look at this one for example

test (distributed, 1, 1, linux.2xlarge) has completed, and is marked as such in this commit pytorch/pytorch@73c4bfc
but according to https://s3.amazonaws.com/ossci-job-status/master/73c4bfc30aa9a425fb5b565839f96779bd0aae6a.json it is still pending

It would be good to write something like "Caught exception while trying to fetch job name for ..." in

In these logs, there are several failing tests, but the classification only records the last. This sometimes causes confusion in the current test failure historical view as certain instances will not be captured. For example, the test_Sparse_to_Sparse_copy__cuda_bfloat16 failure is not captured in https://www.torch-ci.com/failure/test_Sparse_to_Sparse_copy__cuda_bfloat16%2C%20TestSparseCUDA.