Build https://github.com/kubernetes-sigs/sig-windows-tools/tree/master/cmd/wincat and add it to pause image

• Upstream PR https://github.com/kubernetes-sigs/sig-windows-tools/pull/3/files
• ACR task PR https://github.com/Azure/container-compute-upstream-private/pull/112/files
• Update aks-engine Azure/aks-engine#1543
• Update docs with new pause image version kubernetes/website#15345

We should find out the best price/perf point for the most common VM sizes. Variables to test include standard/premium disks, and size since the IOPS quota is related to disk size.

Ephemeral disks can also give a good improvement and might be the best price/perf default. This work item is needed to enable ephemeral disks and has a partially completed PR attached: Azure/aks-engine#1287

This should yield improvements in:

• Time to deploy nodes - internal data is aggregated from AKS-Engine test passes
• Time to deploy first workload. This could probably be a new case added to the E2E test pass that we could aggregate data from. The image pull has a disk-bound phase that increases based on the size of the image. For Windows images where 1GB+ is typical, that's a big difference.

• Add Hashicorp vault as a provider
• Kubecon talk
• Add inline volume support 1.15.0+

Speed up Windows deployments and enable testing new Windows, ContainerD, and Kubernetes builds.

• Design Draft Azure/aks-engine#1386
• Propose a Windows Node resource config API with AKS which could meet needs for Cluster API

I started to play with downloading & caching files here: PatrickLang/aks-engine@c255df0

details: kubernetes-sigs/azuredisk-csi-driver#55

Tracking kubernetes/test-infra#11260

Acceptance Criteria:

• e2e tests are run on clean clusters each time
• e2e tests build out necessary resources for tests to pass

Container will have Windows support in version 1.3.
This work stream is about moving moby off of Windows HCS and rely on containerd.

This is needed rather imminently for the upcoming Docker release.

This is primarily driven by John Howard, but requires review upstream: moby/moby#38541

• Review 38541
• Get 38541 merged
• [ ] CI/build (upstream)

applicable only on Standard LoadBalancer

Implement the proposal outline here: Azure/aad-pod-identity#153

For 1.15:

• Get plan reviewed with SIG-Test, wg-k8s-infra
  • Short term goal - have manual build & image promotion, plan to address and mitigate test failures
• Review and make sure this doesn't break builds if windows build host is not specified kubernetes/kubernetes#75618

For 1.15 or later if needed:

• Get all test images merged from Kubernetes-sigs/windows-testing/images to Kubernetes/Kubernetes/test/images
• Set up Windows build workers on Azure or GCE

For full context, see April 9 notes in https://docs.google.com/document/d/1W31nXh9RYAb_VaYkwuPLd1hFxuRX3iU0DmaQ4lkCsX8/edit# and the notes from the breakout with dims, Claudiu and Patrick.

https://github.com/Azure/kubernetes-kms

• support key rotation

• support bring your own key vault/key

• add kms feature to AKS

Upstream doc PR moby/moby#38620

Primary goal here is better stability in the upstream engine.

https://github.com/kubernetes/enhancements/pull/808/files

https://github.com/kubernetes/cloud-provider-azure/blob/master/docs/azurefile-issues.md

kubernetes/kubernetes#75198

• dry run
• mutation
• tests
• policy library

Since contained 1.0, containerd now handles full container and image management lifecycle.

Much of this is duplicated in Moby (older codebase). The work item here is to migrate Moby to use containerd's image management.
Exercising containerd's full capabilities through Moby is a huge step towards ensuring stability of containerd which is at the core of basically all container workloads.

Work stream is happening here: moby/moby#38738
This is mainly being driven by Docker, but there are many TODO's in the change and needs help to get in.

I do not believe there is enough time to get this in for the next Docker release, however.

Tracked https://github.com/open-policy-agent/gatekeeper/projects/10

Tracking issue for kubernetes/website#12426

https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes-sigs/sig-windows/sig-windows-config.yaml

• Update Windows SecurityContext KEP to include user name
• Rebase kubernetes/kubernetes#73609 on top of kubernetes/kubernetes#75459
• tests kubernetes/kubernetes#79539
• docs kubernetes/website#15378

• collect any existing documents or ideas for redesign of the Azure cloudprovider in cluster-autoscaler (or larger scope?)
• coalesce this into a set of goals ranked by priority we can implement

Please help me clarify this issue!

As a developer for Kubernetes/Kubernetes, I want to run e2e tests against a Kubernetes cluster built with my changes on Azure to make sure my PR did not break anything.

kubernetes-sigs/cloud-provider-azure#4

• Get a current state of coverage
• Increase coverage %!

PR: kubernetes/kubernetes#73977

User namepsaces is an important security feature that almost no one uses because it's really difficult to do.
Just using user namespaces would have mitigated CVE-2019-5736, among others.

Making user namepsaces easier to use and even enabled by default will help the whole industry.

moby/moby#38795

Move Windows containers towards parity with Linux containers, and enable new Windows features to be exposed in K8s tracking issue

Github project: https://github.com/orgs/kubernetes/projects/34

• Plan complete
• KEP approved: https://github.com/kubernetes/enhancements/blob/master/keps/sig-windows/20190424-windows-cri-containerd.md
• Implemetation
• non-VHD implementation Azure/aks-engine#1322
• Beta in 1.19

let's get rid of azure.json and use configmaps instead

Tracking: ?

Done: https://github.com/Azure/aks-engine/blob/master/examples/kubernetes-zones/README.md

https://github.com/kubernetes/cloud-provider-azure/blob/master/docs/azuredisk-issues.md

Tracking issue for the KEP and associated work tracked in the SIG-Windows project board

https://docs.google.com/document/d/1EPb3zg-hknAK7WqYh96XIXCEXG9mQqr_Cqn8VuEGoLI/edit#heading=h.6c6ba7tmtfdm

Minimum Feature Set
We believe the following minimum feature set will be best to have:

• Kubernetes integration
  • Validate admission control
  • Audit
  • CRD representation of library and instances
  • Replicate k8s data into OPA
  • Cache data provided to the system where the caller can provide hints on how to interpret the data (e.g. Kubernetes API Server, GCP)
    • only one data source per target necessary for MVP
• Library API on OPA
  • Ability to write ConstraintTemplates/Constraints
  • Support of multiple targets, where a target:
    • Scopes the behavior of matching_constraints
    • Scopes the behavior of iterator helpers that enable audit
    • Allows for some variance in ConstraintTemplate/Constraint schema that can be defined by the caller (see our documentation on TargetHandler for what we think this might look like)
  • A library of say 20 Kubernetes templates covering Workloads, Networks, Configuration, and Storage
    • The libraries Forseti would use to integrate would also be useful for writing the Admission Controller and writing them will make sure we don't code ourselves into a k8s-specific corner
• Upgrade path for existing users

details: kubernetes-sigs/azuredisk-csi-driver#46

Work on kubernetes/kubernetes#54090 for Windows

details: kubernetes-sigs/azuredisk-csi-driver#90

Tracking issue for VK 1.0 virtual-kubelet/virtual-kubelet#413

kubernetes/enhancements#689

As an Application Operator, I need to specify the operating system and architecture of the images I pull from a registry so that I am sure the right images are going to the right machines
Note: the Application Operator is specified as a new persona as identified in the Snowball working group

Tracking: craiglpeters/asp.net-k8s-lift-and-shift-demo#1

Problem Statement

Windows Update's automatic behavior doesn't match what's needed for AKS-Engine or AKS in a few ways:

• Some optional updates may need to be applied, but not others
• Some patches may need to be avoided if we find failures in testing. For example, the April & May 2019 cumulative updates had changes that intermittently broke Kubernetes service connectivity and DNS

It would need to be configurable so that we can use it to patch VHDs with the known-good list of patches from Windows Update services, but also used for private testing to apply new patches in testing that may be in private storage blobs.

Related solutions

Service Fabric Patch Orchestration Application

doc, github
can orchestrate upgrades across multiple nodes, and handles stopping the Service Fabric service cleanly before reboot and starts it after upgrades complete. It also has a central service to make sure nodes are not taken down simultaneously. That would work for "hotpatching" scenarios if adapted to work with Kubernetes.

As a standalone service, it may still be possible to use this to control what updates are installed. This config would let you include/exclude specific patches using a query language.

https://github.com/microsoft/Service-Fabric-POA/blob/0accfa6d7b6218e1b9543e7403bd7a34905906f0/src/PatchOrchestrationApplication/PatchOrchestrationApplication/ApplicationPackageRoot/ApplicationManifest.xml#L27-L28

windows-patching extension

https://github.com/Azure/aks-engine/tree/master/extensions/windows-patches

This uses the AKS-Engine extension support which calls an extra script as a VM extension. It has a few downsides:

1. No VM Scale Set Support. This means it can't work in AKS
2. It requires full paths to MSU or EXE files. It can't find them using Windows Update

Proposal:
Azure/aks-engine#443

Implementation:
Azure/aks-engine#479

In addition to the Azure-CNI support for Advanced networking in AKS, Windows containers need to support kubenetI for basic networking.

TODO: document definition of done
TODO: Link to issue for aks-engine implementation and testing