appuio / component-openshift4-service-mesh Goto Github PK

We install the RedHat elasticsearch operator into a custom namespace. This should be possible in theory, but the operator doesn't generate a correct ServiceMonitor resource if it's installed in any namespace other than openshift-operators-redhat.

Steps to Reproduce the Problem

1. Install the package on a cluster which uses LokiStack for logging
2. Observe that the ES operator target can't be scraped by Prometheus

Actual Behavior

Operator/OLM creates a ServiceMonitor which uses serverName: elasticsearch-operator-metrics.openshift-operators-redhat.svc in the scrape config's tlsConfig. Since the certificate is correctly issued for elasticsearch-operator-metrics.<custom ns>.svc, Prometheus fails to scrape the metrics with a TLS error.

Expected Behavior

Metrics can be scraped

Possible fixes

• Install ES operator in openshift-operators-redhat. Print a warning when installPlanApproval: Manual is configured
• Patch ServiceMonitor to use the correct tlsConfig -- TBD whether patch-operator is suitable, since we need to patch spec.endpoints[0].tlsConfig`.
• Create a second ServiceMonitor with the correct tlsConfig -- unclear how to ensure the operator/OLM-managed ServiceMonitor is ignored.
• Disable monitoring for ES operator -- probably not a good option

The by default created servicemonitor object contains the wrong servername. It doesn't take the default openshift-operators-redhat rather

$ oc -n syn-openshift-service-mesh-es-operator get servicemonitor elasticsearch-operator-metrics-monitor -o yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
...
  labels:
    name: elasticsearch-operator
  name: elasticsearch-operator-metrics-monitor
  namespace: syn-openshift-service-mesh-es-operator
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
...
    kind: ClusterServiceVersion
    name: elasticsearch-operator.v5.8.0
...
spec:
  endpoints:
...
    tlsConfig:
...
      serverName: elasticsearch-operator-metrics.openshift-operators-redhat.svc

Actual Behavior

Changing the ServiceMonitor serverName to elasticsearch-operator-metrics.syn-openshift-service-mesh-es-operator.svc does solve the certificate missmatch.

Get "https://10.128.20.18:8443/metrics": x509: certificate is valid for elasticsearch-operator-metrics.syn-openshift-service-mesh-es-operator.svc, elasticsearch-operator-metrics.syn-openshift-service-mesh-es-operator.svc.cluster.local, not elasticsearch-operator-metrics.openshift-operators-redhat.svc

Expected Behavior

The component does either install the operator in the namespace openshift-operators-redhat or the servicemonitor object contains the right serverName

Context

The initial implementation (cf. #1) annotates the service mesh namespace with openshift.io/node-selector: '', which allows the istio CNI pods to be scheduled on all nodes. It might be nicer to have a more restrictive node selector (e.g. only app nodes), but when leaving out the annotation completely the DaemonSet misschedules some pods.

We could set annotation openshift.io/node-selector: node-role.kubernetes.io/app=, but depending on the use-case this might be too restrictive. Setting openshift.io/node-selector: node-role.kubernetes.io/worker= is another option, but would probably lead to pods stuck in Pending on clusters which have storage nodes.

Ideally, the operator would be scheduled on infra nodes and the istio-cni daemonset on app nodes. This might be tricky or even not possible.

Also consider making the node selector configurable through a component parameter.