We need the ability to configure portieries (https://github.com/IBM/portieris) in hypershift clusters. This system is used to validate signatures of containers running in a given cluster and is important from a compliance perspective for IBM Cloud

Configuration of Portieris in IBM Roks Toolkit is shown here:
https://github.com/openshift/ibm-roks-toolkit/blob/ec875b0243ac58d095a3eec840ee4d09c24995c2/assets/kube-apiserver/kube-apiserver-deployment.yaml#L319

Acceptance Criteria:

• Ability to configure Portieris appropriately in hypershift clusters.

We need the ability to specify priority classes on critical master deployment manifests in order to ensure these manifests do not get evicted when the workers they are on enter disk pressure, memory pressure, etc. This allows lower priority workload to get evicted first before the master components are considered for eviction.

IBM Roks configuration: https://github.com/openshift/ibm-roks-toolkit/blob/ec875b0243ac58d095a3eec840ee4d09c24995c2/assets/kube-apiserver/kube-apiserver-deployment.yaml#L59

Acceptance Criteria:

• Ability to specify priority classes on master component manifests

I was curious if that's still planned in hypershift. Didn't see it getting deployed yet so will change this issue into tracking deploying that if we plan on still using it

We need the ability to deploy/configure the appropriate cluster-api machine controller depending on the cloud provider. This involves deploying the machine controller and the appropriate CRDs/config that this controller will read.

Acceptance Criteria:

• Support deploying the IBM Cloud VPC Gen 2 cluster-api machine controller when deploying into IBM Cloud VPC Gen 2 environments.
• Ensure dependent config/CRDs also configured and deployed as well to support the machine controller.
• Define pluggable architecture that can be extended for future cloud providers.

We need a way to configure audit webhooks for the kube-apiserver:
https://kubernetes.io/docs/tasks/debug-application-cluster/audit/

This is important for compliance purposes for IBM Cloud clusters. This functionality is utilized to forward audit events from the Kube APIServer for compliance purposes.

Acceptance Critieria:

• Enable configuration of API Server audit webhooks. Example in ROKS toolkit where that information is configured today:
  https://github.com/openshift/ibm-roks-toolkit/blob/master/assets/kube-apiserver/config.yaml#L41

We need to be able to specify pod affinity rules based off a given clusterID to ensure various master components prefer to run colocated on a given node in a given zone. This helps isolate the affect that a given cluster can have on other components running on separate nodes. It also helps optimize the communication path between cluster components since they prefer to be colocated on a given node.

Configuration in ibm-roks-toolkit:
https://github.com/openshift/ibm-roks-toolkit/blob/ec875b0243ac58d095a3eec840ee4d09c24995c2/assets/kube-apiserver/kube-apiserver-deployment.yaml#L34

Acceptance Criteria:

• Ability to specify affinity rules for manifests of hypershift clusters.

We need the support to specify tolerations on the master component manifests. This allows taints to be used in the management cluster to isolate specific types of workload to help prevent certain noisy neighbor problems and allow for special use cases where larger clusters need dedicated machines to support the amount of traffic/workload being processed in the cluster.

The configuration in ROKS toolkit is here: https://github.com/openshift/ibm-roks-toolkit/blob/ec875b0243ac58d095a3eec840ee4d09c24995c2/assets/kube-apiserver/kube-apiserver-deployment.yaml#L28

Acceptance Criteria:

• Ability to specify tolerations on master component manifests.

Unnecessary transitive dependencies have crept in and are now limiting our ability to move independently of unrelated projects.

HyperShift depends on several upstream libraries using Go modules, including:

• k8s.io/client-go
• sigs.k8s.io/controller-runtime
• k8s.io/api

HyperShift also depends on the following Go modules for the purpose of having Go types representing the third-party CRD APIs we leverage:

• sigs.k8s.io/cluster-api
• sigs.k8s.io/cluster-api-provider-aws

It's important to note the only reason we declare these API dependencies is to gain use of the Go types in our codebase. This is convenient but is in no way a technical constraint. For example, if we really wanted to, we could use unstructured types to achieve the same effect.

The CRD API modules we depend on are not structured as Go submodules; therefore, when HyperShift declares a Go dependency on these API modules, HyperShift inherits those projects' own internal dependencies transitively. As a result, we currently have irreconcilable conflicting versions of the machinery libraries listed above.

The net effect is our application has imposed upon it serious limitations by unrelated third-party applications. For example, the version of the Kubernetes API client or controller-runtime we use in HyperShift should not be in any way dictated by the fact that an unrelated third-party application (e.g. cluster-api-provider-aws) also uses a conflicting version controller-runtime.

This coupling of application dependencies is unacceptable and must be eliminated. The upstream projects which seek to provide publicly consumable API types as Go modules need to make changes to separate these very different concerns:

1. Providing public API types
2. Providing an application that is a consumer of the API types

Options

We have options for regaining control of our dependency graph. Here are some I thought of:

1. Use unstructured types instead of the upstream Go types.
2. Convince the upstream projects to extract their APIs into new Go modules which themselves declare no unreasonable external dependencies.
   • Example: Separate API repositories
   • Example: Nested Go modules in the same repository
3. Copy and keep in sync the Go API types we want from their upstream sources.
   • Some of the existing upstream API types are also doing anti-consumer things like declaring dependencies on k8s machinery and controller-runtime interfaces, so a degree of post-processing is inevitable right now with this approach.
4. Generate our own internal versions of the Go API types from the OpenAPI schemas provided by the upstream projects.
   • https://github.com/OpenAPITools/openapi-generator
   • https://github.com/deepmap/oapi-codegen
     • https://github.com/deepmap/oapi-codegen#whats-missing-or-incomplete
   • Others?

Beyond

After the immediate problems are resolved, we must remain dilligent with our dependencies going forward, taking care not to stumble back into a similar situation. Unfortunately, Go modules and the communities that use them make it very easy to inadvertantly end up in a web of unintended transitive dependencies. One mitigation is to simply agree on a principle that all changes that affect the dependency graph should be carefully scrutinized and viewed through a skeptical lens (i.e. "do we really understand the implications of accepting this dependency?")

Postscript: sigs.k8s.io/cluster-api/util/patch

The use of the sigs.k8s.io/cluster-api/util/patch package has found its way into the codebase, and for understandable reasons (it seems to be useful in some contexts, DRY, etc.) However, regardless of how useful it is, the fact remains it is published as part of the sigs.k8s.io/cluster-api module and so brings along all the problems described above with the API types. If this utility is truly desirable for controller authors generally, the code should be packaged and published with public library consumers in mind: in a separate Go submodule with a minimal/narrow set of dependencies, in a separate repo, or incorporated into an existing upstream project focused on enabling controller authors (e.g. controller-runtime).

Until then, the sigs.k8s.io/cluster-api/util/patch dependency must be eliminated one way or another, e.g. copying the code, replacing it with controller-runtime equivalents, persuading upstream to change how it's published, etc.

The e2e suite needs audited to ensure it properly response to SIGINT and SIGTERM by cleaning up any cluster resources created during the test. This finalization already occurs in response to errors during the normal course of the process, but I think signal handling may be deficient.

Related to #63 in that we want to ensure that if we are timed out or interrupted by the CI system we don't leak cloud resources.

Ensure there is a path where a user can provide the certs they want to use in the master components deployed by Hypershift. When these certs are updated: the system should reconcile the appropriate secrets.

Acceptance Criteria:

• Ability to provide certs utilized in the downstream master components. Currently can be done through the pki secret if the proper variables are defined in this secret:

  hypershift/control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go

  Line 618 in 73da1bc

  Name: "pki",

• Ensure downstream secrets are reconciled when the certs/keys are updated

This is used to match the signing duration of our certs to the maximum value allowed by CISO policies. This ultimately turns out to be one parameter in the kube controller manager that manages how long CSRs are valid for

Acceptance Criteria:

• Configuration that enables us to set the signing duration: https://github.com/openshift/ibm-roks-toolkit/blob/master/assets/kube-controller-manager/kube-controller-manager-deployment.yaml#L99

The default security group gets assigned to machines created from a nodepool instead of the expected
[parent-infra]-worker-sg.
This results in management cluster nodes not being able to communicate with the hosted cluster machines.
In addition to that, the current CAPI AWS provider doesn't honor a name filter for security groups. It expects that the ID is specified in the template, causing a panic if it's not:
https://github.com/openshift-hive/cluster-api-provider-aws/blob/e1fd3ca87e0b3445d17d5aab877d5b90046dfa5a/controllers/awsmachine_security_groups.go#L75-L78

We need to be able to configure the ability to encrypt Kubernetes secrets with the encryption provider defined here:
https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/

Specifically for IBM Cloud: we offer support for customers to configure this provider with a IBM Cloud Key Protect instance to encrypt secrets:

https://cloud.ibm.com/docs/openshift?topic=openshift-encryption#kms_cli

The relevant config currently used in IBM-ROKS-TOOLKIT is here: https://github.com/openshift/ibm-roks-toolkit/blob/master/assets/kube-apiserver/kube-apiserver-deployment.yaml#L215

Acceptance Criteria:

• Ability to configure KMS for a given cluster to encypt Kubernetes secrets

We need to extend hypershift to provide an option to utilize Konnectivity: https://github.com/kubernetes-sigs/apiserver-network-proxy for master -> cluster communications (exec, webhooks, logs, etc)

In addition to the Kube-Apiserver needing this support: the openshift-apiserver will need to be adjusted as well to ensure the functionality it provides (image imports, webhooks served by the openshift-apiserver) function in this environment as well

Acceptance Critieria:

• Option to enable Konnectivity for master -> cluster communications
• Ensure Konnectivity components properly deployed when enabled.
• Vet E2E tests work appropriately with Konnectivity. At a minimum: logs, exec, openshift-apiserver image imports, port-forward should all be tested. Image imports can be done by creating image streams.

There needs to be a way to configure the necessary master components to be a highly available configuration that has multiple replicas (IBM's use case is 3) and hard affinity rules to ensure these are spread across zones.

Acceptance Criteria:

• Configuration option to ensure enable affinity rules on master component manifests to ensure workload is spread across zones

We need the ability to specify resource request/limits on all master component manifests in order to allow kube to accurately schedule these components.

IBM Roks toolkit example configuration:
https://github.com/openshift/ibm-roks-toolkit/blob/ec875b0243ac58d095a3eec840ee4d09c24995c2/assets/kube-apiserver/kube-apiserver-deployment.yaml#L217

Acceptance Criteria:

• Specify resource requests and limits on master deployment manifests

Installing the operator as follows fails:

$ hypershift install 
I0414 13:05:31.372856 1535925 request.go:655] Throttling request took 1.027699017s, request: GET:https://api.myclustername.mydomain:6443/apis/apiextensions.k8s.io/v1?timeout=32s
applied CustomResourceDefinition /hostedclusters.hypershift.openshift.io
applied CustomResourceDefinition /nodepools.hypershift.openshift.io
applied CustomResourceDefinition /hostedcontrolplanes.hypershift.openshift.io
applied CustomResourceDefinition /externalinfraclusters.hypershift.openshift.io
applied CustomResourceDefinition /machineconfigservers.hypershift.openshift.io
panic: timed out waiting for the condition

goroutine 1 [running]:
github.com/openshift/hypershift/cmd/install.NewCommand.func1(0xc000310b00, 0x32a6b58, 0x0, 0x0)
	/home/user/go/pkg/mod/github.com/openshift/[email protected]/cmd/install/install.go:75 +0x38c
github.com/spf13/cobra.(*Command).execute(0xc000310b00, 0x32a6b58, 0x0, 0x0, 0xc000310b00, 0x32a6b58)
	/home/user/go/pkg/mod/github.com/spf13/[email protected]/command.go:854 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0xc000310840, 0xc00069bf60, 0x1, 0x1)
	/home/user/go/pkg/mod/github.com/spf13/[email protected]/command.go:958 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
	/home/user/go/pkg/mod/github.com/spf13/[email protected]/command.go:895
main.main()
	/home/user/go/pkg/mod/github.com/openshift/[email protected]/main.go:44 +0x176

Installing it using this method works fine and creates all the required resources:

$ hypershift install --render | oc apply -f -
customresourcedefinition.apiextensions.k8s.io/hostedclusters.hypershift.openshift.io created
customresourcedefinition.apiextensions.k8s.io/nodepools.hypershift.openshift.io created
customresourcedefinition.apiextensions.k8s.io/hostedcontrolplanes.hypershift.openshift.io created
customresourcedefinition.apiextensions.k8s.io/externalinfraclusters.hypershift.openshift.io created
customresourcedefinition.apiextensions.k8s.io/machineconfigservers.hypershift.openshift.io created
namespace/hypershift created
serviceaccount/operator created
clusterrole.rbac.authorization.k8s.io/hypershift-operator created
clusterrolebinding.rbac.authorization.k8s.io/hypershift-operator created
deployment.apps/operator created
customresourcedefinition.apiextensions.k8s.io/clusters.cluster.x-k8s.io created
customresourcedefinition.apiextensions.k8s.io/machinedeployments.cluster.x-k8s.io created
customresourcedefinition.apiextensions.k8s.io/machinehealthchecks.cluster.x-k8s.io created
customresourcedefinition.apiextensions.k8s.io/machines.cluster.x-k8s.io created
customresourcedefinition.apiextensions.k8s.io/machinesets.cluster.x-k8s.io created
customresourcedefinition.apiextensions.k8s.io/awsclusters.infrastructure.cluster.x-k8s.io created
customresourcedefinition.apiextensions.k8s.io/awsmachinepools.infrastructure.cluster.x-k8s.io created
customresourcedefinition.apiextensions.k8s.io/awsmachines.infrastructure.cluster.x-k8s.io created
customresourcedefinition.apiextensions.k8s.io/awsmachinetemplates.infrastructure.cluster.x-k8s.io created
customresourcedefinition.apiextensions.k8s.io/awsmanagedclusters.infrastructure.cluster.x-k8s.io created
customresourcedefinition.apiextensions.k8s.io/awsmanagedmachinepools.infrastructure.cluster.x-k8s.io created
customresourcedefinition.apiextensions.k8s.io/etcdclusters.etcd.database.coreos.com created
customresourcedefinition.apiextensions.k8s.io/etcdbackups.etcd.database.coreos.com created
customresourcedefinition.apiextensions.k8s.io/etcdrestores.etcd.database.coreos.com created

Any ideas what could be wrong?

autoscaler is currently using the cluster-admin kubeconfig for the guest cluster.

We should create a separate kubeconfig with a tight set of permissions for it to use.

@elmiko

We need Liveness and readiness probe specifications on master deployment manifests for accurate health checking and monitoring of the components.

Ibm roks toolkit configuration: https://github.com/openshift/ibm-roks-toolkit/blob/ec875b0243ac58d095a3eec840ee4d09c24995c2/assets/kube-apiserver/kube-apiserver-deployment.yaml#L228

Acceptance Criteria:

• Liveness/readiness probes for all master deployments.

Currently: a unique Machine Config Server is added for every NodePool in hypershift. That requires a unique nodePort or route for every node pool created in the cluster and a unique deployment in every environment. As designed: this currently will not scale to the workload size IBM Cloud needs to support in a given hypershift management cluster.

We need enhancements to the Machine Config Controller and Machine Config Server where 1 machine config server can be used per cluster and through it multiple different pools of config can be served (workers, specialized worker pool configs, etc). There should be a way for these configs to get updated over time as well.

Acceptance Criteria:

• Enhance Machine Config Controller to be able to run in management cluster
• Define process to add MachineConfigs to a given pool. Currently this is done by adding configmaps to the management cluster with a specific label but might change in this design.
• Enable the Machine Config Server to be able to serve configuration for multiple MachineConfigPools in one server
• Ensure Machine Config Controller can update what the Machine Config Server is serving as MachineConfigs are added/removed.

I do not believe we want to rely on logs/exec in critical operations for reliability purposes. Currently it is a requirement for the imageLookup pod:

Opening this for a tracker to see how we can either move to configmaps/ a different system.

Instead of the auto approver just approving certs: it looks if the proper role bindings are added
https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#approval

The controller-manager can automate the approval of only kubelet certificates (client+server) in a secure fashion. This will provide all the necessary behavior we need for a secure kubelet bootstrapping workflow without needing the auto approval.

This task involves adding the images for the hypershift system (hypershift operator, IBM Cloud machine controller, control plane operator, etc) to the release image so it can seamlessly be cloned through image mirroring and signed with the rest of the Openshift images.

Acceptance Criteria:

• Include hypershift images in Openshift release images

Specific section of config is here:

We need the ability to configure the certs utilized on a per domain name basis. In IBM: we have two domains: one that serves certs issued by a well known CA Authority (DigiCert) and then a domain that is valid for the internal cluster CAs/PKI infrastructure.

Acceptance criteria:

• Ability to configure named certs in a hypershift cluster and the associated domain the certs are utilized for

For existing IKS tugboats: we need the ability to specify security contexts so we can ensure images run as non-root.

IBM ROKS Configuration: https://github.com/openshift/ibm-roks-toolkit/blob/ec875b0243ac58d095a3eec840ee4d09c24995c2/assets/kube-apiserver/kube-apiserver-deployment.yaml#L317

Acceptance Criteria:

• Ability to specify security context

This tasks involves support for specifying a "restart annotation" that when changed will trigger a rolling restart of all control plane components. This is a useful feature operationally that enables master components to be restarted which have been known to eliminate certain problems that customers hit. This is also critical for CA rotation as components need to be able to be restarted in order to pickup updates to CA certs (they are not dynamically loaded).

Configuration in IBM Roks toolkit: https://github.com/openshift/ibm-roks-toolkit/blob/ec875b0243ac58d095a3eec840ee4d09c24995c2/assets/kube-apiserver/kube-apiserver-deployment.yaml#L24

Acceptance Criteria:

• Ability to specify restart annotation which will then trigger a rolling restart of all master components managed by hypershift when changed.

The following branches are being fast-forwarded from the current development branch (main) as placeholders for future releases. No merging is allowed into these release branches until they are unfrozen for production release.

• release-4.8
• release-4.7

Contact the Test Platform or Automated Release teams for more information.

Requirements:

• Authentication for MCO
• NodePort services in control plane
• Adopt cluster/byo etcd/pki

Authentication for Machine Config Server:

Current

Machine Config server has no authentication and anyone that has network access to the server can download sensitive cluster payload

Solution:

Will add a haproxy sidecar that verifies a token url parameter that corresponds to the node-bootstrapper-token in the cluster. Machines in their boot user-data will specify this token and be able to fetch the payload. Controllers will be in place for when the token is rotated to update the user-data payload and haproxy config to reference new token automatically.

Control Plane Publishing strategy:

Current

Hypershift relies on services of type LoadBalancer for vpn and kube api and routes for oauth and ignition

MVP Solution:

Ability to publish all services using node ports

Exit criteria:

All external services exposed as a node port which include:

kube-apiserver          NodePort  172.21.217.92  <none>    	2040:31320/TCP  	39h
machine-config-server   NodePort  172.21.95.205  <none>    	80:31249/TCP        39h
konnectivity            NodePort  172.21.55.150  <none>    	1194:31250/TCP  	39h
oauth-openshift         NodePort  172.21.162.205 <none>    	443:31251/TCP       39h

Long term (ignore for this year):

Utilize SNI https://en.wikipedia.org/wiki/Server_Name_Indication and unique domain names per cluster per exposed service to be able to route everything through only port 443 using ISTIO

Example

konnectivity-clusterid1.SUBDOMAIN -> istio/router -> backend service TLS passthough
apiserver-clusterid1.SUBDOMAIN -> istio/router -> backend service TLS passthough
oauth-clusterid1.SUBDOMAIN -> istio/router -> backend service TLS passthough

and deploy config that will tell the router component to send TLS to these endpoints.

Benefit:

Customers will only need to expose 443 to the set domain name
All ingress traffic comes through single point of entry that can enable advanced audit log use cases

"Adopt" existing clusters/Asset payload control ("bring your own ETCD")

Current:

In automation upstream etcd operator is deployed and a single replica etcd cluster is deployed as well

Requirement:

Need to be able to seamlessly "adopt" all the existing ROKS clusters we have in production to be under hypershift control

One key difference: etcd clusters are named differently in ROKS and automation deploys etcd-operator/the etcd cluster. All backup/restore/operations associated with etcd depend on this system.

Example:

named etcd-CLUSTERID. All services correspond to that name as well

kubectl get etcdcluster -n master-bvnqi2710tsfq5mqtbrg
NAME                        AGE
etcd-bvnqi2710tsfq5mqtbrg   68d

We will also create the downstream PKI secrets that integrate into our CA rotation system.

The Makefile verify target needs to verify that a given commit hasn't forgotten to execute make api to prevent regressions like #67.

After I deleted my managed cluster with kubectl -n master delete hostedcluster clusteridtylergo1: resources in the master namespace corresponding to the cluster still are present. Specifically

kubectl -n master get secrets
clusteridtylergo1-admin-kubeconfig   Opaque                                1      5d20h
clusteridtylergo1-provider-creds     Opaque                                1      5d20h
clusteridtylergo1-pull-secret        Opaque                                1      5d20h
clusteridtylergo1-signing-key        Opaque                                1      5d20h
clusteridtylergo1-ssh-key            Opaque                                1      5d20h

The hosted cluster no longer exists

Tylers-MacBook-Pro:hypershift tylerlisowski$ kubectl -n master get hostedclusters clusteridtylergo1
Error from server (NotFound): hostedclusters.hypershift.openshift.io "clusteridtylergo1" not found

$ oc get nodepool
NAME                       NODECOUNT
example-cluster-external   

$ oc get machineset -n example example-c-c5ed10a5-example-cluster-external
NAME                                          REPLICAS   AVAILABLE   READY
example-c-c5ed10a5-example-cluster-external   2

This code should be syncing the two
https://github.com/openshift/hypershift/blob/main/hypershift-operator/controllers/nodepool_controller.go#L161-L165

We need to ensure all components are built in a FIPS enabled way (with go-toolset) and that there is a path to enable FIPS in RHCOS workers (already a field in the machine config).

One downstream dependency is we need go-toolset to have a go 1.16 image since hypershift depends on go 1.16

Acceptance Criteria:

• Ensure all applicable go programs built in go-toolset container with FIPS enablement. This will include cluster-api, cluster-api controller managers, etc.

For existing IKS tugboats running containerd: there currently is not an option similar to: https://github.com/containers/image/blob/master/docs/containers-registries.conf.5.md

that allows the dynamic translation of images through the container runtime configuration. In IBM Roks toolkit: control plane images are translated before the images are applied from the quay.io base to the mirrored registry (example: registry.ng.bluemix.net)

This same logic will be necessary in order to support existing ROKS clusters on IKS tugboats. Or containerd will need to provide a way to do namespace registry mirroring. The discussion with the containerd folks are is here:
containerd/containerd#5048

Current IBM Roks toolkit logic that performs it:
https://github.com/openshift/ibm-roks-toolkit/blob/62dba55df9e7fb0b0f5a352a653d7268d7558c56/pkg/release/info.go#L36

Acceptance Criteria:

• Provide support for image mapping before applying image in control plane manifest to properly support ROKs clusters in IKS based tugboats.

The Makefile verify target needs to perform some Go module validation (e.g. go mod vendor and go tidy) to detect when a commit changes dependencies but forgot to run those commands.

There is a use case in which a customer may want to bring their own etcd instance managed in their own desired way outside of hypershift (database provider, etc). In order for hypershift to support this: the pieces around the deployment/reconciliation/ and managment of etcd need to be able to be enabled/disabled. That means that in certain deployments: the etcd controller should be able to be turned off and the CRDs associated with the etcdcluster, etcdbackup, and etcdrestores should not be deployed in the cluster.

In addition, there are a few master components that need to have configs adjusted to properly point to the etcd instance (kube-apiserver, openshift-apiserver, etc). We need to ensure that config is controllable and can be pointed to the url the etcd cluster is available on.

We must define and codify the AWS policy requirements so users know what permissions they're granting HyperShift to provide managed clusters.

Currently we don't explicitly define the AWS IAM policy or policies necessary for HyperShift to work. With a codified definition of the requirements, we can more easily get user feedback about whether the policies are acceptable.

This issue tracks effort to introduce a more clearly and narrowly scoped cloud permissions model for AWS.

Cloud provider credentials are currently used in the following contexts:

• Kube Controller Manager (KCM) deployment
• Kube API server deployment
• Kube CSI driver cloud credentials
• CAPI AWS provider

Only the Kube Controller Manager and CAPI provider really need cloud credentials, so the CSI and API server usages should simply be removed.

The KCM and CAPI components have separate responsibilities and different permission needs. Each of these components should use distinct AWS IAM policies that we define and support so that users have more control over the exposed surface area of accounts attached to those policies.

Plan

Based on prior discussions, a viable approach has two main parts:

1. Update the HostedCluster API to replace the single ProviderCreds field with a new struct that embeds two credentials fields for the two known contexts (KCM and CAPI provider).
2. Update the CLI to produce canonical AWS policy resources and make creating default accounts using those policies easy for tests and development.

TODO:

In the control plane operator...

• Remove cloud credentials requirement from API server (#127)
• Remove cloud credentials requirement from CSI driver support (#127)
• Update HostedControlPlane API to indicate purpose (KCM) (#128)

In the hypershift operator...

• Implement HostedControlPlane API to enable declaring multiple contextual cloud credentials (#128)
• Refactor so CAPI and HostedControlPlane use contextually appropriate credentials (#128)

In the CLI...

• Teach the infra command to generate the two AWS IAM policies the cloud credentials must use
• Teach the infra command to generate IAM users for each cloud credential context
• Teach the infra command to (one way or another) associate generated IAM users with the policies
• TODO: Figure out how the secret contents should be passed from infra to create commands
• Introduce options to the create cluster command to specify multiple credentials secrets

There needs to be a way to configure the following network parameters associated with a cluster:
Pod CIDR
Service CIDR
Apiserver configuration: Advertised Address
Apiserver configuration: Secure Port

Acceptance Criteria:

• Configuration options for all networking parameters.

I get following error when trying to deploy a hosted cluster to a region other than us-east-1.

$ hypershift create cluster \
>   --pull-secret ~/pull-secret.txt \
>   --ssh-key ~/.ssh/id_rsa.pub \
>   --aws-creds ~/.aws/credentials \
>   --region ca-central-1 \
>   --name mycluster \
>   --base-domain mydomain
I0414 14:17:25.131112 1543354 request.go:655] Throttling request took 1.022221875s, request: GET:https://api.clustername.basedomain:6443/apis/packages.operators.coreos.com/v1?timeout=32s
INFO[0003] Creating infrastructure                       baseDomain=mydomain id=clusterid subdomain=clusters-mycluster.mydomain
INFO[0003] Discovered base domain zone                   baseDomain=mydomain id=ZONEID
Error: failed to create infra: failed to create stack: ValidationError: Template error: Unable to get mapping for RegionMap::ca-central-1::DHCPDomainName
	status code: 400, request id: e4d5c43b-3302-4bc2-8f7b-d056016c0eff
Usage:
  hypershift create cluster [flags]

Flags:
      --aws-creds string            Path to an AWS credentials file (required)
      --base-domain string          The base domain for the cluster
  -h, --help                        help for cluster
      --infra-id string             Infrastructure ID to use for AWS resources.
      --infra-json string           Path to file containing infrastructure information for the cluster. If not specified, infrastructure will be created
      --instance-type string        Instance type for AWS instances. (default "m4.large")
      --name string                 A name for the cluster (default "example")
      --namespace string            A namespace to contain the generated resources (default "clusters")
      --node-pool-replicas int      If >0, create a default NodePool with this many replicas (default 2)
      --overwrite                   If an existing cluster exists, overwrite it
      --preserve-infra-on-failure   Preserve infrastructure if creation fails and is rolled back
      --pull-secret string          Path to a pull secret (required)
      --region string               Region to use for AWS infrastructure. (default "us-east-1")
      --release-image string        The OCP release image for the cluster (default "quay.io/openshift-release-dev/ocp-release:4.7.7-x86_64")
      --render                      Render output as YAML to stdout instead of applying
      --ssh-key string              Path to an SSH key file

failed to create infra: failed to create stack: ValidationError: Template error: Unable to get mapping for RegionMap::ca-central-1::DHCPDomainName
	status code: 400, request id: e4d5c43b-3302-4bc2-8f7b-d056016c0eff

A new hosted control plane's ingresses.config.openshift.io/cluster resource has an empty .spec.domain value. This breaks the ability to create a hosted cluster within a hosted cluster because the OIDC endpoint can't be discovered as a function of the cluster ingress domain.

This is a regression introduced by #196.

Fixing a missing linebreak in the hypershift create cluster ... command
adding a few outputs to show expected results for the commands
and adding a trouble shooting section

If you run the hypershift destroy cluster ... command it doesn't clean up the clusters\example-pull-secret - which means on a subsequent run to recreate your cluster (maybe there was an issue and want to start over) the same potentially bad pull-secret is in the cluster.

Should destroy also clean this on each run since we require the --pull-secret parameter on the create cluster command?

For CAPI/autoscaler we are currently using the kubeconfig that has an external address for the APIServer and going through the external load balancer for that access. We should only need to point to the local service.

We need the ability to control the network type used here:

The immediate use case is for IBM Cloud Openshift clusters Calico is utilized as the SDN provider.

Acceptance Critieria:

• Support the ability to configure different NetworkTypes for hypershift clusters

This item involves working with the redhat team to properly upstream the monitoring setup we use with Prometheus + AlertManager and Pagerduty to operate our management clusters. There is interest in the specific alerts we use for monitoring the health of master components/etcd/ general infra health (nodes not ready, etc).

IBM Specific alerts will be left out (things like monitoring of our master components, IBM Cloud specific API monitoring, etc)

Acceptance Criteria:

• Upstream/outline general monitoring architecture and specific configs IBM Cloud uses to monitor masters deployed in the management cluster

related to #83 the autoscaler kubeconfig should use the service network to talk to the guest kube-apiserver, not the external API endpoint

@elmiko

We need to be able to configure the identity providers used in the oauth server for a Hypershift cluster. Currently that is configured with this parameter:

However, that parameter cannot be set in Hypershift. We need to be able to customize what that is for a given use case. The need in IBM Cloud is to integrate it with IBM Cloud IAM through the OpenID provider type:
https://docs.openshift.com/container-platform/4.1/authentication/identity_providers/configuring-oidc-identity-provider.html

Acceptance Criteria:

• Enable configuration of the oauth identity provider in Hypershift.

Following the CI docs, the hypershift e2e tests need updated to specify timeouts and grace periods. According to historical runs, 1h15m seems like enough for a hard timeout. Cluster deletion takes a while, so a lengthy grace period (10m? 15m?) might be necessary — would be worth auditing a few runs to get a more accurate estimate.

Since the machine config server has the potential to be a public endpoint and serves up cluster specific certificates and kube bootstrapping credentials we need some method to authorize access to the endpoint.

The discussed solution was adding a haproxy side car to the machine config server that parses a url parameter that must match the node bootstrapping token inside the cluster. The initial ignition url that the workers boot with is then modified to contain the token url parameter.

When the token is revoked or regenerated: there is a controller that picks up the new token, updates the haproxy side car with the new data, and ultimately updates the user data secret to have the new token value as well to provide an automated path to revoke any old exposed tokens.

PR is here:
#191

We need to enhance the hypershift operator to be able to deploy the necessary configuration for IBM Cloud environments. This includes the adjustment of some cluster bootstrap CRDs to have the platform type set appropriately. It also includes setting the cloud-provider configuration in the kube components to be an external cloud provider (since IBM Cloud's Cloud controller manager is an external out of tree plugin).