There are two different methods that enumerate open issues. One is the shared library and one in the Quest2GitHub project.

They are different in that these methods return different fields in a GitHub issue object.

These should be refactored into a shared library.

It might be possible to use some kind of config or options object to control which fields of an issue are returned.

Associated WorkItem - 166826

Add build verification workflow for all tools

Associated WorkItem - 59877

The current code uses the System.DisplayName when setting the Assigned to field. That usually works, but relies on each user using the same display name in both Azure Dev Ops systems being queried.

Instead, retrieve the MS email from the OSPO DevOps board, then query the msft-skilling board for that email. If it's successful, use the retrieved ID to assign the issue.

Related to #145.

Even if no code files in a project are referenced, the project shouldn't be deleted if it's part of a solution where one or more of its projects are referenced.

Associated WorkItem - 69087

When a GitHub issue is imported into AzDo or the AzDo work item is re-synced, update the Size project property and the current iteration.

If an issue is in multiple projects, use the data from the latest (most recent) iteration.

In order to read the V2 project data, Sequester must be be registered as an org-wide Action, and use a different authentication procedure. Those tasks are included in the list that follows.

There are several tasks that are part of this work. Those requiring help from the dotnet OSSadmins have already been completed:

• Create org-wide Sequester GitHub app.
• Install it in the docs related repos
  • dotnet/docs
  • dotnet/dotnet-api-docs
  • dotnet/csharpstandard
  • dotnet/ASPNETCore.Docs
  • dotnet/ASPNETCore.Docs.Samples
  • dotnet/docs-maui
• Run the Sequester app using the new authentication policy to have an org enabled key based on the private key secret.
• Update application to read project data associated with the issue. Make this code resilient enough to recover when run with the existing authentication strategies. Data should include:
  • Project name
  • Size field
• Write the updated Size and iteration data in AzDo when an issue is updated.
• Create the live private key and install into each repository as a repo secret.

Associated WorkItem - 71067

There are a number of tools our team has used that were initially maintained in an internal Azure DevOps repository. We'll move the current version of each of these tools into this public repository.

As part of this process, we'll remove any code not currently in use. These tools rely on libraries. This is a good opportunity to remove code from the libraries that isn't being used. Some of the projects also include test projects. These tests may not be rigorous, but are a reasonable starting point.

A couple projects aren't needed in the future, and will be archived elsewhere.

Projects to move:

• What's new application:
  • What's new infrastructure class library
  • Infrastructure test project
• RepoMan (label utility.)
  • RepoMan config test project
• LocateProjects (used by Snippets 5000) Addressed in #100
• IssueCloser (Used when we declare issue bankruptcy and want to close issues in bulk)
• GitHubRepositoryExplorer: These projects support the Dotnet/Docs GitHub Issue metrics website.
  • GitHubRepositoryExplorer.Client: Blazor client.
  • GitHub.RepositoryExplorer.Models: Data model class library
  • GitHub.RepositoryExplorer.WebAPI: WebApi endpoint to consume time series data.
  • GitHub.RepositoryExplorer.Services: Services library for the WebAPI endpoint
  • GitHub.RepositoryExplorer.Functions: Azure Function that retrieves issue data on a daily basis and stores the history in Azure storage.

The following projects aren't going to be migrated to this repository:

• LabelUpdater: change labels on issues or PRs in bulk. This uses object Ids for the labels so it will work even when moving from one label to another existing label.
• GitHub.Repository.Explorer.Spike: Console application test application to consume the query data. Used for initial testing.

Finally, the library DotnetDocsTools will be migrated, along with its associated test project. Only code in use in the library will be migrated, and only the tests that exercise those APIs will be migrated.

Improve reading of headings in preview table. Occasionally the h1 fails to be read. Still not sure why, but logs indicate that the files do not exist, instead, let's just fail to read them instead of trying to check if they exist before reading them. Also, add coalesce to title.

Associated WorkItem - 91659

Hi @Youssef1313

We should add an ignore rule that keys on the "What's new" config file... any deletions in this folders should be allow-listed.

See the false negative here: https://github.com/dotnet/docs/pull/24937/checks?check_run_id=2963505022

You might want to read / check for a .whatnews.json file at the repo root. If it's there, any deletions without redirect that are part of that folder should be ignored. See https://github.com/dotnet/docs/blob/main/.whatsnew.json#L9-L15

Originally posted by @IEvangelist in dotnet/docs#24937 (comment)

The Quest action currently runs on triggers when someone edits an issue or PR. That's caused race conditions because we've found that it's often the case that we make multiple changes to an issue that triggers multiple runs of the quest action. There's a window of time where the first run may be processing the issue and a subsequent run reads the earlier state. That results in creating two different AzDo work items.

Instead, run Quest overnight as a cron job to update the AzDo board with all the updates made during the last day.

Associated WorkItem - 67482

We've often added new labels, or important comments to an issue when the issue is closed. When an issue is closed, update the internal description field with all updated labels and contents.

Associated WorkItem - 67483

It doesn't seem like the inline error reporting is working: https://github.com/dotnet/docs/actions/runs/4812097336/jobs/8566903949?pr=35112

Also, the log didn't upload. Thinking more about it, is it required anymore? We had it originally because it was the output file that was created from script1 and sent to script2.

Associated WorkItem - 87258

The current code is here: https://github.com/dotnet/docs-tools/blob/main/actions/sequester/Quest2GitHub/GitHubClientServices/AddOrRemoveLabelMutation.cs

It's been modified to fix some issues with race conditions, and the names, structures, and flow are not clear. Some of it is a naming issue, and some is that the flow will only work well in the use cases already in place.

We should:

• Add unit tests on the structure and form of each mutation query
• Rename types and APIs to reflect what they actually do.
• Consider a builder type that builds a mutation with multiple actions. This would really need tests.

Associated WorkItem - 157255

There's an issue with how the action.yml file is constructed, specifically see error:

Warning
version-sweep
dotnet/docs-tools/main/actions/dotnet-version-updater/action.yml (Line: 32, Col: 23): Unrecognized named-value: 'secrets'. Located at position 1 within expression: secrets.GITHUB_TOKEN

Associated WorkItem - 157257

See #21 (comment)

Example reference style link: [1118]: syslib1118.md

Both ReplaceRedirectedLinks() and CheckFileLinks() need to be updated to account for more link types, including reference links.

Associated WorkItem - 83518

See the warnings in 4 files in https://buildapi.docs.microsoft.com/Output/PullRequest/a3bda507-3390-de91-8f7f-26f90f4e5fc8/202303170422279441-34630/BuildReport?accessString=fdfb71e3a9c8ff44fc6e7deccc0be44402296fd2bc498391f841901f393681fa.

The LocateProjects app is used in the Snippets5000 CI build.

Instead of running that from a hand-generated NuGet package in each repo, we should run it directly from the source in this repo.

/cc @IEvangelist @adegeo @gewarren

Additionally, the Snippets5000 PowerShell logic should merge in with LocateProjects and have a single code base that a single maintainer can work on.

Associated WorkItem - 60144

@IEvangelist @gewarren @BillWagner

Could take the basic format as currently done in dotnet/docs#31543 and replace the few repos that have copies of this similar files.

• Create reference job here. In what folder?
• Create problem match file here
• Replace downstream jobs and problem matcher files
  • https://github.com/dotnet/runtime/tree/main/.github/workflows
  • https://github.com/dotnet/docs/tree/main/.github/workflows

/cc @BillWagner I think this is the repo that I was thinking would be dotnet/workflows

Currently there is a limitation where the preview table generation only accounts for the first: 100 files. Instead, it should evaluate all files.

Associated WorkItem - 96855

In the sprint meeting today, we discussed a tool that automatically pulls F1 keyword metadata from each linked article in a toc.yml file into the toc.yml file as displayName keywords (used in TOC filtering).

• If there are already displayName keywords, use the union of those and the F1 metadata, and select distinct items.
• If there are no F1 keywords in the linked article, skip that TOC entry.

The preview link in this PR: dotnet/docs#34942 was created with the wrong link text.
It interpreted as an H1 a line that begins with a single hashtag inside a code block where it was only a comment.

Associated WorkItem - 87018

1. Needs to account for possibility of multiple redirection files. All redirection files (if there is more than one) will be listed in the .openpublishing.publish.config.json file like so:

"docsets_to_publish": [
    {
      ...
      "build_entry_point": "docs",
      "redirection_files": [
        ".openpublishing.redirection.json",
        ".openpublishing.redirection.architecture.json",
        ".openpublishing.redirection.azure.json",
        ".openpublishing.redirection.core.json"
      ],
      ...
    }

2. For redirects that use source_path (as opposed to source_path_from_root), the path to the file is relative to the location of the redirection file that contains it. So we need to check the path a little differently for this one.

Associated WorkItem - 132723

Discovery

I was investigating why four recent dependabot automated PRs were failing. Here is an example failed build from one of the: https://github.com/dotnet/docs/actions/runs/4355198732/jobs/7611491747.

I immediately found it odd that there were missing source code files. I discovered that these files and projects were deleted.

Realization

The LocateProjects app needs to account for project deletions.

Recently, there was a significant deletion PR in docs: dotnet/docs#34146. This PR deleted several projects that were part of a solution. However, the snippets 5000 build validation didn't validate that the solution still compiled because the solution (*.sln) wasn't returned as part of the LocateProjects subroutine.

In other words, LocateProjects needs to be updated to return solutions when a project is deleted.

Proposal

When a project is deleted, check to see if the said project was part of a solution. If it was, return the solution that it was a part of.

Associated WorkItem - 85827

We are seeing false positives, resulting in empty tables. This is an example:

Associated WorkItem - 97767

Also, add new options for different start directories for articles, media, snippets, and include files.

Associated WorkItem - 69088

The config object has grown slowly with each new feature and changed behavior in the tool. At this point, a number of config options could be flattened or removed. Some currently required options could be made optional.

This should be one update for config, because it may include breaking changes in repos where the tool is installed or used. While we should minimize breaks, they may be the right choice in this instance.

See #254 (comment)

If a site-relative link is found in a toc.yml file, the "relative links" functionality doesn't convert it to file-relative even when that's possible.

For the MS Docs verifier action, if a PR removes a link to an article from a toc.yml, toc.md, index.yml, or index.md file, but doesn't delete that article, the action could check if that was the last reference to that article and it should be deleted and redirected.

Instead, we'd like the table generated regardless of build warnings.

Associated WorkItem - 83887

Associated WorkItem - 147376

Expose the CleanRepo tool as a GitHub Action, such that it may be consumed by GitHub Action workflows in various repos. Ideally, this will run as a CRON job, automatically creating PRs monthly.

Associated WorkItem - 64348

See dotnet/docs#33826 (comment)

In the .NET config, we have some folders where subfolders are in a different section. As a result, the articles are duplicated.

We need to support that scenario. The rules should be that the "longest" path should win. Namely, a more specific subfolder should be where an article should be located when that matches more than one area.

Associated WorkItem - 62196

The actual folder name doesn't have to be "includes" as long as a parent directory is named "includes".

See https://teams.microsoft.com/l/message/19:[email protected]/1681411296058?tenantId=72f988bf-86f1-41af-91ab-2d7cd011db47&groupId=de9ddba4-2574-4830-87ed-41668c07a1ca&parentMessageId=1680212408946&teamName=Learn%20(Docs)%20Platform%20Support&channelName=General&createdTime=1681411296058&allowXTenantAccess=false.

Associated WorkItem - 86271

Debugging issues in redirection verifier can be painful without the ability of unit testing. This should also ensure that we don't regress the feature at some point.

This will help remove the unused code files.

While issues are how we log most of our work items, sometimes we are assigned as reviewer or editor on large PRs. Those should be tracked as well. It is more efficient to generate the internal work item from the PR than to create a separate issue to track

Associated WorkItem - 161878

A number of our tools create reports that track issues / PRs created or assigned. They often check if the author or assignee is a Microsoft FTE using the OSPO REST API.

The results should be cached in memory to save bandwidth and be a good citizen toward that API.

In addition, while working on this one, default to "community member" if the REST API fails. We can filter out FTEs by inspection, when necessary.

Currently there is a bug where README.md files are included in the preview table, these should be ignored. See dotnet/docs#35539 for example issue.

Associated WorkItem - 96049

• Test case for Edit - Solution found, no project fails, need to fix it.

  This test case edits a code file in a folder under the solution, but there isn't a project with that file. The solution is detected and passes testing. It should fail with NO PROJECT FOUND. This is a bit tricky because the code was modified to allow a solution to be discovered when a project under it is deleted so NO PROJECT FOUND yet the solution was found and that's ok in that scenario. The next TODO item may solve this problem.

• Possibly rewrite the entire file detection logic.

  Right now it uses a loop to go down from the root repo to the files being edited. If a solution file is found, it's marked. We used to require a project being found under it for the solution to be valid, but this isn't really a good assumption. If we were removing files in folders below the solution, such as a project in folderB\me.csproj, we would have no project to discover and would fail. There could have been folderA\me.csproj under the solution, making the solution valid.

  I've removed this logic to just assume that if we find a solution, we'll just try to compile it and see if it's OK.

  We may need to redesign the logic to instead build a list of folders of files that have hit by the PR. Then folders are used to build out a tree structure from that folder (if it exists) down.

  • If any code fragment is found, a project must be found from that spot and above.
  • If any project is found, it must be the only project from that spot and above.
  • If any project is found, a solution may be found above it but the project must be in the solution file.

Associated WorkItem - 84404

Now that the source to the What's new in docs app is in this repo, we should add the necessary configuration / docker file to run the tool directly from this source rather than running it from a hand-generated NuGet package in each of our repos.

/cc @Rick-Anderson

Associated WorkItem - 60143

Example from azure-docs-pr repo:

[!code-gradleAzureSpatialAnchorsScript]

Currently the code thinks that INCLUDE file is orphaned, but it's not.

We should add an action that runs on a commit in a PR to add a link to the preview site for updated articles.

We still need to clarify exactly how we want this to work:

• should it update the PR description, or add a comment?
• Should it add a link to every preview article, or only "significant" changes? If the latter, how do we define "significant"?

The right location might be to add this functionality to the action that checks for OPS build warnings. That runs when the build completes, which is a good time to update the links.

Associated WorkItem - 69260

Example:

- name: "wfc.exe Workflow Compiler Tool"
  href: "wfc-exe-workflow-compiler-tool.md" 

Depends on whether internal feature request will be implemented or not (I only have the link without access to it, so the status is unknown to me)

Changing file name from .md to .yml doesn't require adding a redirection.

The set of strings for ignored PRs should be configurable.

There are other repos that have requested to add variations of the following text strings in PR repos:

• "repo_sync_working_branch"
• "main" or "merge release"
• "(Internal)" or "[Internal]" - case insensitive.
• "cruft"

In addition, the "learn-build-service-prod" bot should be excluded.