openshift / custom-domains-operator Goto Github PK

Operator to manage custom domains on OpenShift

License: Apache License 2.0

Dockerfile 0.46% Shell 42.43% Go 37.88% Makefile 10.36% Python 8.87%

custom-domains-operator's Introduction

Openshift Dedicated Custom Domain Operator

This operator sets up a new ingresscontroller with custom certificate as a day-2 operation. The public DNS record of this new ingresscontroller can then be used by external DNS to create a wildcard CNAME record to for a custom domain.

On cluster, a CustomDomain custom resource creates an IngressController, which creates a set of router pods.

graph LR
  A[customdomains] --> B[ingresscontrollers]
  subgraph openshift-ingress-controller
  B
  end
  B --> C[Router pods]
  subgraph openshift-ingress
  C
  end

Deprecation

On versions of Managed Openshift (OSD/ROSA) greater than version 4.14 (or version 4.13 if the =ext-managed.openshift.io/legacy-ingress-support= flag is switched on for the cluster) the Custom Domains Operator will no longer reconcile new CustomDomain objects. Existing CustomDomain objects will be converted to native Openshift IngressController resources, and their HAProxy workloads allowed to be scheduled onto customer worker nodes. Consult https://access.redhat.com/articles/7028653 for further information.

Prerequisites

Go 1.19+
Operator-SDK v1.25+

Building And Deploying

Also see Deploying the operator from a development branch for testing

Setup

Create Custom Resource Definition (CRD)

oc apply -f deploy/crds/managed.openshift.io_customdomains_crd.yaml

Run locally outside of cluster

operator-sdk run --local --namespace ''

Build and Deploy To Cluster

Choose public container registry e.g. 'quay.io/acme'. Build and push the image, then update the operator deployment manifest.

Example:

# deploy manifests
oc apply -f deploy/crds/managed.openshift.io_customdomains.yaml
oc apply -f deploy/
# build
make docker-build docker-push
# update image with image in build output
oc set image -n openshift-custom-domains-operator deployment/custom-domains-operator custom-domains-operator=quay.io/dustman9000/custom-domains-operator:v0.1.29-a48b301e

Testing

See TESTING

custom-domains-operator's People

Contributors

Stargazers

Watchers

custom-domains-operator's Issues

Create new ingress with static IP address

Currently when a new ingress controller is created, an IP address is taken from a pool of IP addresses. We would like to see an option where we can set a static IP address for the ingress to keep openings in the customer firewall consistent.

8/18/2023 update crashes on startup

The latest PR merge got auto-updated in our clusters and the operator pod crashes on every startup attempt.

2023-08-18T15:24:07Z INFO setup starting manager
2023-08-18T15:24:07Z INFO Starting server {"kind": "health probe", "addr": "[::]:8081"}
2023-08-18T15:24:07Z INFO starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
2023-08-18T15:24:07Z INFO Starting EventSource {"controller": "customdomain", "controllerGroup": "managed.openshift.io", "controllerKind": "CustomDomain", "source": "kind source: *v1alpha1.CustomDomain"}
2023-08-18T15:24:07Z INFO Starting EventSource {"controller": "customdomain", "controllerGroup": "managed.openshift.io", "controllerKind": "CustomDomain", "source": "kind source: *v1.IngressController"}
2023-08-18T15:24:07Z INFO Starting EventSource {"controller": "customdomain", "controllerGroup": "managed.openshift.io", "controllerKind": "CustomDomain", "source": "kind source: *v1.Secret"}
2023-08-18T15:24:07Z INFO Starting Controller {"controller": "customdomain", "controllerGroup": "managed.openshift.io", "controllerKind": "CustomDomain"}
2023-08-18T15:24:07Z INFO Starting workers {"controller": "customdomain", "controllerGroup": "managed.openshift.io", "controllerKind": "CustomDomain", "worker count": 1}
2023-08-18T15:24:07Z INFO controller_customdomain Reconciling CustomDomain {"Request.Namespace": "", "Request.Name": "app-domain"}
2023-08-18T15:24:07Z INFO controller_customdomain Secret change detected, updating certificate. {"Request.Namespace": "", "Request.Name": "app-domain"}
2023-08-18T15:24:08Z INFO Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference {"controller": "customdomain", "controllerGroup": "managed.openshift.io", "controllerKind": "CustomDomain", "CustomDomain": {"name":"app-domain"}, "namespace": "", "name": "app-domain", "reconcileID": "df5fcd89-23cf-44ca-a433-9256e4567ac3"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1524186]
goroutine 119 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:115 +0x1fa
panic({0x1697800, 0x28b8990})
/usr/lib/golang/src/runtime/panic.go:884 +0x212
github.com/openshift/custom-domains-operator/controller.(*CustomDomainReconciler).Reconcile(0xc0000b5800, {0x1b40598?, 0xc0047ca330?}, {{{0x0?, 0x10?}, {0xc0004bf826?, 0x413247?}}})
/src/controller/customdomain_controller.go:368 +0x1466
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1b40598?, {0x1b40598?, 0xc0047ca330?}, {{{0x0?, 0x161cf20?}, {0xc0004bf826?, 0x10?}}})
pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118 +0xc8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0002c2960, {0x1b404f0, 0xc0002c0640}, {0x1716b80?, 0xc00017e5c0?})
pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:314 +0x3a5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0002c2960, {0x1b404f0, 0xc0002c0640})
pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222 +0x333

custom domain operator should allow me to set a routeSelector for the ingressController

I want to set:

apiVersion: managed.openshift.io/v1alpha1
kind: CustomDomain
metadata:
  name: acme
spec:
  domain: my-domain.io
  certificate:
    name: acme-tls
    namespace: my-custom-route
    routeSelector:
      matchLabels:
        route: acme

and have it create:

apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: acme
  namespace: openshift-ingress-operator
spec:
  defaultCertificate:
    name: acme
  domain: acme.xxxx.p1.openshiftapps.com
  endpointPublishingStrategy:
    loadBalancer:
      scope: External
    type: LoadBalancerService
  routeSelector:
      matchLabels:
        route: acme

This will allow me to ensure I don't pick up default routes, but do pick up my own labelled routes.

custom domains operator should be able to configure ClientTLS settings on it's managed Ingress Operator

OCP 4.9 brings the ability to configure ClientTLS back from the dead (it was supported in 3.x), some OCP 4.x users want to take advantage of it, especially those on various managed openshift platforms that don't allow direct Ingress Operator management.

Ideally we would update the CRD to allow for clientTLS in the spec that would map directly to the Ingress Operator's spec for it

CustomDomain certificate does not get reconciled

It is pretty common to use cert-manager to generate and manage certificates as you already know. In Openshift workloads, the use of custom-domains-operator combined with cert-manager and the cert-utils-operator does help to solve the problem to update the TLS information on routes upon certificate renewal issued by cert-manager.

Having said this, this formula works well for us, which means that whenever cert-manager refreshes a certificate, the respective secret gets updated and therefore, the expected route also gets the new secret content.

The problem we are seeing is that when we create the customDomain, the initial secret matches with the certificate managed by cert-manager, however, when the cert-manager refreshes it, the secret stored/copied in the openshift-ingress namespace does not get refreshed. For example:

customDomain

apiVersion: managed.openshift.io/v1alpha1
kind: CustomDomain
metadata:
  name: realm-domain
spec:
  certificate:
    name: domain-name-cloud-tls
    namespace: default
  domain: domain-name.example.com
  scope: External

As we can see, the referenced secret is stored in the default namespace. Upon the customDomain creation, a new secret named realm-domain is created on the openshift-ingress namespace which matches the content with the referenced secret (which is expected).

$ kubectl view-secret -n default domain-name-cloud-tls  tls.crt | md5sum 
4b87156c030f6d5a49dfbdfc4ffb5841  -
 
$ kubectl view-secret -n openshift-ingress realm-domain tls.crt | md5sum 
4b87156c030f6d5a49dfbdfc4ffb5841  -

Problem

Now if we trigger a certificate refresh via cert-manager, we can see the custom-domain-operator does not reconcile the secret configured and therefore breaks forces us to manually update the certificate.

$ kubectl cert-manager renew domain-name-cloud-tls -n default
Manually triggered issuance of Certificate default/domain-name-cloud-tls

$ kubectl view-secret -n openshift-ingress realm-domain tls.crt | md5sum
4b87156c030f6d5a49dfbdfc4ffb5841  -

## now they are different
$ kubectl view-secret -n default domain-name-cloud-tls  tls.crt | md5sum 
ba50299d1e8e3bff8d778857ddf25728

Expectation

The custom-domains-operator should reconcile or use the secret specified in its configuration instead of creating a copy of it.

2023 Change Freeze | branch:master

master branch is frozen for merges until Friday Jan-06-2023 @ 14:00 UTC