kcp-dev / controller-runtime Goto Github PK

This project forked from kubernetes-sigs/controller-runtime

Repo for the controller-runtime subproject of kubebuilder (sig-apimachinery)

License: Apache License 2.0

Shell 0.99% Go 98.73% Makefile 0.29%

controller-runtime's Introduction

Kubernetes controller-runtime Project

The Kubernetes controller-runtime Project is a set of go libraries for building Controllers. It is leveraged by Kubebuilder and Operator SDK. Both are a great place to start for new projects. See Kubebuilder's Quick Start to see how it can be used.

Documentation:

Versioning, Maintenance, and Compatibility

The full documentation can be found at VERSIONING.md, but TL;DR:

Users:

We follow Semantic Versioning (semver)
Use releases with your dependency management to ensure that you get compatible code
The main branch contains all the latest code, some of which may break compatibility (so "normal" go get is not recommended)

Contributors:

All code PR must be labeled with 🐛 (patch fixes), ✨ (backwards-compatible features), or ⚠️ (breaking changes)
Breaking changes will find their way into the next major release, other changes will go into an semi-immediate patch or minor release
For a quick PR template suggesting the right information, use one of these PR templates:

FAQ

See FAQ.md

Community, discussion, contribution, and support

Learn how to engage with the Kubernetes community on the community page.

controller-runtime is a subproject of the kubebuilder project in sig apimachinery.

You can reach the maintainers of this project at:

Slack channel: #controller-runtime
Google Group: [email protected]

Contributing

Contributions are greatly appreciated. The maintainers actively manage the issues list, and try to highlight issues suitable for newcomers. The project follows the typical GitHub pull request model. See CONTRIBUTING.md for more details. Before starting any work, please either comment on an existing issue, or file a new one.

Code of conduct

Participation in the Kubernetes community is governed by the Kubernetes Code of Conduct.

controller-runtime's People

Contributors

Stargazers

Watchers

Forkers

matousjobanek robinbobbitt pradeepnnv p0lyn0mial fgiloux faroshq madhavjivrajani fossabot embik kcp-stuff mjudeikis arangogutierrez aaronjwood hailkomputer

controller-runtime's Issues

Update to logicalcluster v2

Set up CI

Prepare for prow
- Add .ci-operator.yaml
- Add DOWNSTREAM_OWNERS
- Add repo/jobs in openshift/release

Provide instructions / examples for writing tests with kcp-dev/controller-runtime & envtest

I've been taking a poke at running the Application Service Operator's tests against KCP, while using the kcp-dev/controller-runtime library, but have run into some issues.

Since the mock cluster stood up by envtest isn't KCP, I've been using the USE_EXISTING_CLUSTER toggle to use my existing kubeconfig (where my kubeconfig points to a KCP workspace). However, the tests as-is fail, since they were not written to be multi-workspace aware:

Application controller
�[90m/Users/johncollier/kcp/application-service/controllers/application_controller_test.go:41�[0m
  Create Application with no repositories set
  �[90m/Users/johncollier/kcp/application-service/controllers/application_controller_test.go:51�[0m
    �[91m�[1mShould create successfully with generated repositories [It]�[0m
    �[90m/Users/johncollier/kcp/application-service/controllers/application_controller_test.go:52�[0m

    �[91mTimed out after 10.001s.
    Expected
        <bool>: false
    to be true�[0m

    /Users/johncollier/kcp/application-service/controllers/application_controller_test.go:81

It seems that I need to make some modifications to suite_test.go and each individual controller's tests, similar to the modifications required for main.go and each controller.go file when using kcp-dev/controller-runtime library, but I haven't been able to get things to pass.

It'd be helpful if there were some instructions or an example (using https://github.com/kcp-dev/controller-runtime-example) for tests.

Maybe also if envtest could mock KCP, so that we don't need to point to an existing KCP instance for the tests (though I acknowledge that this likely isn't feasible).

Controller-runtime and custom cache

Trying to setup a custom cache to filter APIBindings it seems that the resulting URL is not properly processed.

options.NewCache = cache.BuilderWithOptions(cache.Options{
		SelectorsByObject: cache.SelectorsByObject{
			&apisv1alpha1.APIBinding{}: {
				Field: fields.SelectorFromSet(fields.Set{
					"spec.reference.workspace.exportName": "settings-configuration.pipeline-service.io",
					"spec.reference.workspace.path":       "root:pipeline-service:management",
				}),
			},
		},
	})

gives

W0909 17:14:34.487460 1472564 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v1alpha1.APIBinding: forbidden: User "kcp-admin" cannot get path "/services/apiexport/root:pipeline-service:management/settings-configuration.pipeline-service.io/apis/apis.kcp.dev/v1alpha1/apibindings": Path not resolved to a valid virtual workspace
E0909 17:14:34.487484 1472564 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1alpha1.APIBinding: failed to list *v1alpha1.APIBinding: forbidden: User "kcp-admin" cannot get path "/services/apiexport/root:pipeline-service:management/settings-configuration.pipeline-service.io/apis/apis.kcp.dev/v1alpha1/apibindings": Path not resolved to a valid virtual workspace

and also for other resources that my controller owns

W0909 17:14:34.050696 1472564 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v1.NetworkPolicy: forbidden: User "kcp-admin" cannot get path "/services/apiexport/root:pipeline-service:management/settings-configuration.pipeline-service.io/apis/networking.k8s.io/v1/networkpolicies": Path not resolved to a valid virtual workspace
E0909 17:14:34.050739 1472564 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.NetworkPolicy: failed to list *v1.NetworkPolicy: forbidden: User "kcp-admin" cannot get path "/services/apiexport/root:pipeline-service:management/settings-configuration.pipeline-service.io/apis/networking.k8s.io/v1/networkpolicies": Path not resolved to a valid virtual workspace

Without custom cache it works and the url is

https://xxx.xxx.xxx.xxx:6443/services/apiexport/root:pipeline-service:management/settings-configuration.pipeline-service.io"

I haven't had time to investigate and I am not excluding that I did something wrong but this is the only modification.

controller runtime client list fails with error "Index with name cluster does not exist"

hi,
When I use latest https://github.com/kcp-dev/controller-runtime, the controller runtime client List is fails with error "Index with name cluster does not exist".

Steps to reproduce:
in the project https://github.com/kcp-dev/controller-runtime-example, point to latest kcp-dev/controller-runtime, make change to compile the code and test the controller-runtime-example. you can use this branch to test https://github.com/prembhaskal/kcp-dev-controller-runtime-example/tree/client-list-issue


% export KUBECONFIG=/tmp/kcp.kubeconfig
% kubectl ws create prem-org --type organization
% kubectl ws prem-org

 % cat /tmp/data-api-binding.yaml 
apiVersion: apis.kcp.io/v1alpha1
kind: APIBinding
metadata:
  name: data.my.domain
spec:
  reference:
    export:
      name: data.my.domain
      path: root
  permissionClaims:
    - resource: "secrets"
      all: true
      state: Accepted
    - resource: "configmaps"
      all: true
      state: Accepted
    - resource: "namespaces"
      all: true
      state: Accepted
% kubectl apply -f /tmp/data-api-binding.yaml

% # create configmap with needed labels and data, check the output of the controller and see if anything wierd happens there.
% cat /tmp/test-cm.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: syncer-config
  labels:
    name: syncer-config
data:
  agent-target-cluster: "root:045aa3c9-017c-4824-a410-561b1e2ddef3:soumik-demo:cg-1"
  agent-target-name: "downstream"
  agent-target-namespace: "default"
  upstream-context: base
  secretData: syncer-config

% kubectl apply -f /tmp/test-cm.yaml

error seen

2024-01-29T23:23:54+05:30	INFO	Update: updated configMap	{"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"syncer-config","namespace":"default"}, "namespace": "default", "name": "syncer-config", "reconcileID": "38ae15cb-9d1c-433d-a3d6-645970009745", "cluster": "29qso51vovx9kz1x"}
2024-01-29T23:23:54+05:30	INFO	Get: retrieved configMap	{"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"syncer-config","namespace":"default"}, "namespace": "default", "name": "syncer-config", "reconcileID": "2b6493be-d719-4b86-9870-2ee1aa72fcea", "cluster": "29qso51vovx9kz1x"}
2024-01-29T23:23:54+05:30	ERROR	unable to list configmaps	{"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"syncer-config","namespace":"default"}, "namespace": "default", "name": "syncer-config", "reconcileID": "2b6493be-d719-4b86-9870-2ee1aa72fcea", "cluster": "29qso51vovx9kz1x", "error": "Index with name cluster does not exist"}
github.com/kcp-dev/controller-runtime-example/controllers.(*ConfigMapReconciler).Reconcile
	/Users/bpremkumar/code/github.com/prembhaskal/kcp-dev-controller-runtime-example/controllers/configmap_controller.go:87
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/Users/bpremkumar/go/pkg/mod/github.com/kcp-dev/[email protected]/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/Users/bpremkumar/go/pkg/mod/github.com/kcp-dev/[email protected]/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/Users/bpremkumar/go/pkg/mod/github.com/kcp-dev/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/Users/bpremkumar/go/pkg/mod/github.com/kcp-dev/[email protected]/pkg/internal/controller/controller.go:227

I had created a ticket here kcp-dev/controller-runtime-example#38 but i think the issue and fix belongs more here.
Please let me know so that i can close the ticket on example repo and continue it here.

APIReader client is not cluster-aware

The direct RO client, that can be retrieved with manager.GetAPIReader(), is not configured with the logical cluster context aware round-tripper.

That client is used by controllers instead of the default client, to bypass the cache that may caused some races in critical sections.

Update to Kubernetes 1.28

kcp-dev/kcp#3017 just got merged, so we should work on updating this fork to 1.28.

List requests with field selector are not cluster scoped

When the manager is configured with a field indexer like:

mgr.GetFieldIndexer().IndexField(ctx, &corev1.Pod{}, "status.phase",
	func(obj ctrl.Object) []string {
		pod, _ := obj.(*corev1.Pod)
		return []string{string(pod.Status.Phase)}
	})

The result of a list request like:

mgr.GetClient().List(ctx, &corev1.PodList{},
	ctrl.InNamespace(namespace),
	ctrl.MatchingFields{"status.phase": string(corev1.PodPending)})

Returns the resources indexed globally, not for the cluster propagated in the context.

Respect custom cache function

Problem description

For standard controller runtime it's possible to define cache function that would control what object kinds are cached partially by a selector. This is extremely useful if operator deals with a kind that have many instances in the cluster (usually Secrets and ConfigMaps), so caching them all would require a lot of memory and often causes OOM.

In KCP controller runtime this mechanism does not work.

Expected behavior

Cluster aware manager respects cache function option.

How to reproduce

options := ctrl.Options{
    ...
    NewCache: cache.BuilderWithOptions(cache.Options{
        SelectorsByObject: cache.SelectorsByObject{
                // The list of objects and the corresponding selectors are arbitrary
                // Here is a possible use-case
                &taskrunapi.PipelineRun{}: {
			Label: appStudioComponentPipelineRunSelector,
		},
		&corev1.Secret{}: {
			Label: partOfAppStudioSelector,
		},
        }
    }
}

...
mgr, err = kcp.NewClusterAwareManager(cfg, options)
...
mgr.Start(ctx)

Full example could be found here. Just uncomment the NewCache option.

Other notes

There is a hack that possibly could make it work, but:

it requires changing internal URL which is unsafe and error prone
it brings problems to the parts that already respect the path (so some functionality will not work as a consequence)
it requires some reordering in code which is undesirable, especially if we need to support non-KCP clusters by the operator.

Unable to use controller-runtime for ClusterWorkspaceType initializer

Consider that I have a ClusterWorkspaceType and an initializer that is supposed to create an APIBinding. That initializer is watching a VirtualWorkspace of the given CWT, however, when it tries to create an APIBinding, then it fails with:

1.65719959317604e+09	ERROR	controller.clusterworkspace	Reconciler error	{"reconciler group": "tenancy.kcp.dev", "reconciler kind": "ClusterWorkspace", "name": "appp", "namespace": "", "error": "no matches for kind \"APIBinding\" in version \"apis.kcp.dev/v1alpha1\""}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/mjobanek/go-workspace/src/github.com/kcp-dev/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/home/mjobanek/go-workspace/src/github.com/kcp-dev/controller-runtime/pkg/internal/controller/controller.go:227

The reason is that the rest mapper tries to do the discovery call before calling the POST for the APIBinding. But it does it for the VirtualWorkspace endpoint with the wildcard /clusters/* at the end of the URL, not with the name of the workspace that is being initialized /clusters/root:plane:usersignup:foo/:

I0707 13:13:13.510869       1 round_trippers.go:463] GET https://192.168.1.133:6443/services/initializingworkspaces/root:plane:usersignup:Appstudio/clusters/%2A/api/v1
I0707 13:13:13.510875       1 round_trippers.go:469] Request Headers:
I0707 13:13:13.510882       1 round_trippers.go:473]     Authorization: Bearer <masked>
I0707 13:13:13.510888       1 round_trippers.go:473]     Accept: application/json, */*
I0707 13:13:13.521749       1 round_trippers.go:574] Response Status: 404 Not Found in 10 milliseconds
I0707 13:13:13.521845       1 request.go:1372] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }
I0707 13:13:13.524075       1 round_trippers.go:574] Response Status: 200 OK in 13 milliseconds
1.6571995935242307e+09	ERROR	controller.clusterworkspace	Reconciler error	{"reconciler group": "tenancy.kcp.dev", "reconciler kind": "ClusterWorkspace", "name": "appp", "namespace": "", "error": "no matches for kind \"APIBinding\" in version \"apis.kcp.dev/v1alpha1\""}

There are two problems in the rest mapper code:

The rest mapper doesn't use the cluster-aware client nor the cluster-aware round-tripper. But even if I modify the code so it uses the cluster-aware round-tripper, then I face the second problem:
The context that is being passed as part of the discovery call is context.TODO() https://github.com/kubernetes/client-go/blob/release-1.23/discovery/discovery_client.go#L172 and not the one that is provided from the controller. This means that the context doesn't have the cluster name value so the cluster-aware round-tripper cannot do its job.

Problem

K8s Event creation fails if use standard approach in KCP.

Steps to reproduce

Create KCP aware manager using NewClusterAwareManager
Create an EventRecorder using mgr.GetEventRecorderFor
Create an event by EventRecorder.Event
The following error appears:

forbidden: User "system:serviceaccount:build-service:build-service-controller-manager" cannot post path "/services/apiexport/root:users:zu:yc:kcp-admin:redhat-appstudio/build-service/api/v1/namespaces/test/events": Path not resolved to a valid virtual workspace' (will not retry!)

Expected behavior

Event creation works out of the box.

Investigation

@ncdc investigated this issue and found that the problem might be here. The code called from here.

Add test that verifies multi-cluster behavior

Get an ObjectKey with cluster + name
Get an ObjectKey with cluster + namespace + name
List all cluster-scoped things in 1 logical cluster vs. another (make sure they're distinct)
List all namespace-scoped things in 1 logical cluster vs. another (make sure they're distinct)
Create works
Update works
Patch works
Delete works

kcp-dev / controller-runtime Goto Github PK

controller-runtime's Introduction

Kubernetes controller-runtime Project

Versioning, Maintenance, and Compatibility

FAQ

Community, discussion, contribution, and support

Contributing

Code of conduct

controller-runtime's People

Contributors

Stargazers

Watchers

Forkers

controller-runtime's Issues

Problem description

Expected behavior

How to reproduce

Other notes

Problem

Steps to reproduce

Expected behavior

Investigation

Recommend Projects

Recommend Topics

Recommend Org