Story

As a user I can install the Rancher CAPI extension and use it to provision my Rancher clusters.

This backend needs to be integrated with Fleet to make "all Rancher CAPI" GitOps friendly.

Description

The Rancher UI extension is backed by the backend operator code that implements the logic to use CAPI with Rancher.

During development, we'll be using custom repositories for the backend operator code. It can be configured from Rancher UI -> Continuous Delivery -> Git Repos -> Add Repo.

Preliminary investigation

Validate backend operator workflow:

• Rancher user communicates with Rancher Manager.
• Rancher Manager communicates with CAPI extension.
• The logic of communication between Rancher and CAPI is the core backend operator.

What do we need to integrate this with Fleet?

• #81
• #117

Story

Description

Initially we're using custom repositories for quick development. Once we have an stable version, we'll need to move to an official Rancher repository.

What would you like to be added (User Story)?

As a developer
I want an ADR to document our use of unstructured
So future engineers understand the decision

Detailed Description

We need an ADR to document why we use unstructured when reading/creating Rancher specific CRs

Anything else you would like to add?

No response

Label(s) to be applied

/kind feature

When we install the Rancher Turtles chart we shoudl also install the CAPI Operator.

We can use chart dependencies

Currently, rancher cluster interactions are implemented using unstructured, this is done to avoid additional dependencies. However, working with unstructured objects is not a readable and pleasant experience. We can create a wrapper to simplify it.

Story

As a rancher operator,
I want to ensure the CAPI operator is installed when i install rancher turtles
So that i get CAPI and a default set of providers installed.

Description

We are going to rely on the CAPI Operator to handle the install of CAPI and providers. We are also going to provider a "default" configuration.

Tasks

• Install the operator automatically (#91)
• Define if we want a core set of providers installed automatically

Story

As a Rancher Operator,
I want to be able to have some way to configure which clusters are automatically imported,
So that i have the option to not import clusters into Rancher Manager

Description
The current implementation of auto import looks for a label on the CAPI cluster or on the Namespace. We need to revisit these options to see if they are sufficient or do we need something like a cluster selector instead?

Related to #12

What would you like to be added (User Story)?

As a developer
I want e2e tests that run quickly on PR and still have full coverage daily
So that PRs can be merged quicker but we still have daily testing

Detailed Description

We need to have 2 e2e GHA workflows

• On PR it executes tests that use Docker as the infra provider
• Periodically 1 time per day run all tests

Anything else you would like to add?

No response

Label(s) to be applied

/kind feature

What would you like to be added (User Story)?

As a developer
I want an ADR to document our new use of structured proxy types instead of unstructured
So future engineers understand the decision

Detailed Description

We need an ADR to document why we decided to use structured proxy types when reading/creating Rancher-specific CRs instead of unstructured and supersede the old documentation with this new approach ADR

Anything else you would like to add?

No response

Label(s) to be applied

/kind feature

The initial code was just to prove a point. Tidy the code up and split into packages so that it is easier for us to work on.

Story

As a Rancher Operator,
I want to be able to install Rancehr Turtles in the same cluster as Rancher Manager.
So that i can provision clusters with CAPI and the existing Rancher provisioning mechanisms

Description

This means that v2prov should continue to work and we shouldn't be required to disable any of the features in Rancher to use the CAPI extension.

This requires some small changes to Rancher (issue already created).

Story

As a Rancher Operator,
I want to automatically import a Ready CAPI cluster into Rancher,
So that i can explore the cluster, install applications and perform other tasks like any other imported cluster.

Description

The initial poc auto imported the cluster but we need to refactor the code to be more production ready. A few things we need to consider/look at it:

• Should we add a reference to github.com/rancher/rancher/pkg/apis so we can use the API types instead of using unstructured.
• We cannot have the Rancher Cluster named the same as the CAPI cluster due to a conflict in the kubeconfig generation. We need to think how we name the new instance, which namespace we place it in.
• Currently we use the import endpoint to get the manifests. Could we use the actual import generation code directly?

What would you like to be added (User Story)?

Add CI to check license headers in all to be added new/existing files

Detailed Description

As per the review comment, we could add a CI check (GH workflow/custom script) to check for license headers in all existing and newly added files to the repo.

Anything else you would like to add?

As I investigated it a bit, looks like we have these options:

https://github.com/denis-tingaikin/go-header - At first glance does not look like has been adopted by other projects but very simple
https://github.com/viperproject/check-license-header - same as above
https://github.com/hashicorp/copywrite - looks fine, mostly used with the hashicorp project GH org

If there are other options that can be used than the above listed, please suggest

Label(s) to be applied

/kind feature

The poc assumes that the controller is running in the same cluster as Rancher manager and it uses the k8s client directly. Change the POC so that you can choose to use the k8s client or use the Rancher API when starting the operator.

To use the rancher API the user will need to supply the following:

• URL
• Bearer token (probably from a secret)

Create e2e test that uses the AWS infra provider (CAPA) to ensure that we can import and access the cluster.

Story

As a user I can discover and install the Rancher CAPI extension from the UI.

Description

Rancher extensions are enabled from the Rancher UI which installs a Helm Chart and allows the user to use the extension in Rancher clusters.

• The chart will be defined in rancher/ui-plugin-charts and we can use existing extensions, such as Kubewarden, as a reference.
• Package UI extension: generate JS package

Preliminary investigation

• How to build UI extension: scaffolding, docs, tools, etc.
• Structure: is it equivalent to other existing extensions?

What would you like to be added (User Story)?

As a developer,
I want an ADR to document our "running out of Rancher Manager cluster" approach
So future engineers understand the decision

Detailed Description

We need to document the implications of our decision to support running Rancher Turtles in a separate cluster to Rancher Manager

Anything else you would like to add?

The RFC states that we want to support 2 deployment topologies for Rancher Turtles:

1. Rancher Manager & CAPI Management combined (in single cluster)
2. Separate Rancher Manager and 1 or more CAPI management clusters

This RFC is about documenting the decisions that where made that influenced the implementation of 2

Label(s) to be applied

/kind feature

What would you like to be added (User Story)?

As a developer
I would like to have a e2e that uses CAPV
So that i have confidence that it works with Rancher Turtles

Detailed Description

Add a e2e test that uses:

• vSphere infra provider
• Kubeadm bootstrap/control plave provider
• Use the Rancher generated kubeconfig to connect to created cluster

Anything else you would like to add?

No response

Label(s) to be applied

/kind feature

What would you like to be added (User Story)?

As a rancher capi extension user
I expected that the import url is validated
So that i am not open to malicious workload injection

Detailed Description

Currently we just get the content of the URL here...there is no validation.

We need to protect the downstream cluster for deploying malicous code.

We can validate the URL and the payload (i.e. yaml)

Anything else you would like to add?

No response

Label(s) to be applied

/kind feature

What would you like to be added (User Story)?

As a developer
I want stable/consistent e2e tests
So that they give confidence and value to me.

Detailed Description

The e2e tests can be a bit unstable, lets look into this and make them more stable.

Anything else you would like to add?

No response

Label(s) to be applied

/kind feature

Story

As a Rancher Operator,
I want to delete a CAPI cluster from Rancher,
So that it is no longer listed in Rancher

Description

We need to allow deleting a CAPI imported cluster from Rancher. The underlying CAPI cluster should not be deleted.

What would you like to be added (User Story)?

As a developer
I would like to have a e2e that uses CAPZ
So that i have confidence that it works with Rancher Turtles

Detailed Description

Add a e2e test that uses:

• Azure infra provider
• Kubeadm bootstrap/control plave provider
• Use the Rancher generated kubeconfig to connect to created cluster

Anything else you would like to add?

No response

Label(s) to be applied

kind/feature

Story

As a Rancher Operator,
I want to be able to install Rancher turtles via the marketplace
So that i have a standard way to install it

Description
This means packaing the operator as a Helm chart. Initially we will publish it to our own helm repository but ultimately this will need to be published in the main Rancher charts repo.

Story

As a Rancher Operator,
I want the option to supply a display name and description,
So that i can customize the display in the clusters list

Description
Add support for annotating the capi Cluster so that when we import the cluster that if these annotations are present we populate the required fields on the Rancher cluster.

The investigation will result in new issues being created to do the actual work.

Story

The extension should be built automatically via GitHub workflows.

Description

Existing Rancher extensions implement a series of GitHub Actions that automatically build the extension package. Based off these workflows, automate building and packaging the CAPI extension.

What steps did you take and what happened?

When using a separate CAPI management Rancher cluster never gets created successfully. We set an owner reference on the Rancher cluster https://github.com/rancher-sandbox/rancher-turtles/blob/main/internal/controllers/import_controller.go#L233 but in case of out-of-cluster scenario CAPI cluster doesn't exist on the same cluster as Rancher cluster so garbage collector instantly removes Rancher cluster.

We might need a separate controller that handles deletion, this controller will watch Rancher cluster and react on it's deletion instead of having in CAPI cluster controller https://github.com/rancher-sandbox/rancher-turtles/blob/main/internal/controllers/import_controller.go#L194.

What did you expect to happen?

Successfully import it to rancher

How to reproduce it?

1. Create rancher manager cluster
2. Create a separate CAPI cluster
3. Pass rancher manager cluster kubeconfig as an argument to rancher turtles
4. Create a workload cluster and apply import label

Rancher Turtles version

No response

Anything else you would like to add?

No response

Label(s) to be applied

/kind bug

We need to update the readme so that its more useful to potential user of Rancher Turtles.

Story

As rancher turtles dev,
I want GHA workflows for CI tasks like image building, linting, running tests
So that i know the code builds and pases tests

Description

Standard CI stuff 😄

What would you like to be added (User Story)?

As a user of Rancher Turtles
I want the option of using different provisioning mechanisms in Rancher
So that i can pick and choose depending on my needs

Detailed Description

We'll need to add a unit test where:

• Rancher is srated with Rancher Turtles (i.e. embedded capi feature disabled and operators deployed)
• Create a cluster using v2prov
• Ensure cluster is created without an issue

Anything else you would like to add?

No response

Label(s) to be applied

/kind feature

Story

Description

Define where the extension's backend code resides. For now we can use rancher-turtles but, is it the long-term choice?

Other extensions are split into two repositories, e.g. Kubewarden:

• UI - kubewarden/ui
• Backend - kubewarden/kubewarden-controller

We need to publish the chart and image to the Rancher Prime registry.

Story

As a rancher turtles dev,
I want to have e2e test coverage,
So that i have confidence that there are no regressions.

Description
We need a way to test e2e scenarios of the operator.

Story

As a Rancher User,
I want to have documentation on using the CAPI extension,
So that i know how to install & use the extension.

What would you like to be added (User Story)?

As a developer
I want the Rancher Turtles chart published to a private registry that is easily accesible
So that they can be used for testing

Detailed Description

We will publish the chart and images to 2 places:

1. Rancher Prime Registry (#81)
2. Private GHCR (this issue)

Anything else you would like to add?

No response

Label(s) to be applied

/kind feature

We need to add telemetry to the extension.

Investigate the following:

• How to use Rancher Telemetry package
• How to integrate it into an external operator
• Adhere to the opt-out (i.e. look at the Rancher settings)

Feed into #38 in the future

What would you like to be added (User Story)?

As a QA, I would like to have a Design Doc for deriving testing scenarios

Detailed Description

Design Doc template: https://confluence.suse.com/display/CECO/Design+Doc+for+QA?src=contextnavpagetreemode

Anything else you would like to add?

No response

Label(s) to be applied

No response

What would you like to be added (User Story)?

As a developer
I want an ADR to document our cluster import strategy
So future engineers understand the decision

Detailed Description

We need to document why we import clusters the way we do.

Anything else you would like to add?

No response

Label(s) to be applied

/kind feature

What would you like to be added (User Story)?

As a developer
I would like to have a e2e that uses CAPRKE2
So that i have confidence that it works with Rancher Turtles

Detailed Description

Lets add e2e test that uses:

• Docker infra provider
• CAPRKE2 control plane & bootstrap provider
• Use the Rancher generated kubeconfig to connect to created cluster

Anything else you would like to add?

No response

Label(s) to be applied

/kind feature

What would you like to be added (User Story)?

As a user of Rancher Turtles
I need to have the additional ClusterRoles deployed
So that the existing provisioning mechanisms still work when i use Rancher Turtles

Detailed Description

We need to use the feature of CAPI Operator that can deploy additional manifests to deploy the following cluster roles:

• https://github.com/rancher/provisioning/blob/ddba2c23cd70016f002448fd801d96aa9375782e/charts/capi/templates/clusterrole-cattle.yaml
• https://github.com/rancher/provisioning/blob/ddba2c23cd70016f002448fd801d96aa9375782e/charts/capi/templates/clusterrole-capi-aggregated-manager-role.yaml

Anything else you would like to add?

No response

Label(s) to be applied

/kind feature

What would you like to be added (User Story)?

As a rancher turtles dev
I want to have e2e tests
So that i have confidence that there are no regressions

Detailed Description

We need this as a base for specifically testing various scenarios such as #4

Anything else you would like to add?

No response

Label(s) to be applied

/kind feature

What would you like to be added (User Story)?

As a user, I want to use labels for mark cluster/namespace for import.

Detailed Description

Currently, we're using annotations, however, annotations are not meant to be used by humans there're for programmatic interactions, labels better suit the use-case

Anything else you would like to add?

No response

Label(s) to be applied

/kind feature

When a CAPI cluster is ready automatically import it into Rancher.

Description

As reported by @alexander-demicev, support for installation of Rancher Turtles in a different cluster from Rancher Manager (#82) is failing due to an incorrect use of clients ("same-cluster-client" being used when "out-of-cluster-client" should be configured).

In addition to this, the watcher is looking for Rancher Cluster in the same k8s cluster. Will be opening a different issue to address the problem with the watcher.

Story
As a Rancher Operator,
I want the choice to deploy Rancher Turtles in the same or a different cluster to Rancher Manager,
So that i have choice in my deployment topology.

Description
Currently, its assumed thats the operator is running in the same cluster as Rancher Manager. Change the operator so you can optionally supply a reference to a kubeconfig to use (via a secret) instead of using the incluster credentials.

• #186
• Change the operator to optionally support using a kubeconfig for creating a client in addition to the in cluster (#19)
• Define & document RBAC permissions for the kubeconfig (i.e. least privilege)
• e2e test covering both deployment models

What would you like to be added (User Story)?

As a developer
I want an ADR to document our deletion strategy
So future engineers understand the decision

Detailed Description

We need an ADR to document the decision around how we handle _deletion.

Anything else you would like to add?

No response

Label(s) to be applied

/kind feature

We would like to prove out credentials mapping from CAPI to Rancher cloud credentials.

We can use AWS credentials to prove this. We'd need to watch for AWSClusterStaticIdentity and create am AWS cloud-credentials in the Rancher manager cluster (via API see #3 or direct see #4).

Currently there are several issues with implementation of deletion strategy for the rancher cluster.

1. imported=true annotation is being set conditionally based on the fact it should also be present on the rancher cluster. It shouldn’t, the behavior supposed to be automatic upon cluster deletion.
2. Events coming from updates of the rancher cluster are not propagated to the CAPI cluster due to incorrect name mapping between the two.
3. Controller-runtime mapper methods do not support predicate filtering logic on enqueued requests, it triggers reconciliation on the resource directly by enqueuing the NamespaceName of the object and passing it into the work queue. Which causes the following race condition:
   • Rancher cluster is getting deleted. This generates a mapped reconcile request to the CAPI cluster, which triggers a reconcile loop without passing predicate logic for the enqueued CAPI cluster. Rancher cluster has deletion timestamp, so the CAPI cluster is annotated and nothing else happens.
   • At some point Rancher cluster is finally removed, which, again, triggers reconcile on CAPI cluster and it follows to the reconcileNormal scenario. The Rancher cluster is re-created again and is visible in UI.

Update the chart so that when the specific 1.4.x version of core CAPI is being installed we enable the following features:

• CRS
• Topology

This will give a better experience for users when they get started.

Story

As a rancher operator,
I want to use Rancher Manager cloud credentials with CAPI,
So that I don't have to enter credentials twice in the UI.

Description
We would like to prove out credentials mapping from Rancher cloud credentials to CAPI credentials.

We'd need to watch for AWS cloud-credentials in the Rancher manager cluster and then create the relevant CAPI credential type for the provider.

For example, for AWS credentials we would create an instance of AWSClusterStaticIdentity.

Unanswered Questions
Do we do this ahead of time? Or at the point of creating the cluster?

What would you like to be added (User Story)?

As a rancher-turtles dev/pm
I want to have telemetry available
So that i can see what the usage of the extension is and what features are used.

Detailed Description

We need to add telemetry to the project. And it should use the same telemetry mechanism as Rancher so needs to use https://github.com/rancher/telemetry.

This is the implementation of the design from #389

We have to be careful not to include any PII information.

Anything else you would like to add?

No response

Label(s) to be applied

/kind feature