epic: Offer Prometheus metrics about kubestellar HOT 8 OPEN

/priority high

from kubestellar.

/remove-priority high
/priority medium

from kubestellar.

This is not entirely unrelated to #2094

from kubestellar.

Ugh. Of course controller-runtime does it differently. See https://book.kubebuilder.io/reference/metrics.html for some clues.

from kubestellar.

Also, OpenShift has an opinion about how to get metrics scraped (i.e., via the Prometheus Operator). https://book.kubebuilder.io/reference/metrics.html#exporting-metrics-for-prometheus talks about the provisions for that in controller-runtime. This is only an FYI, scraping is beyond the scope of this epic.

from kubestellar.

Regarding the OCM Status Add-On: although the framework (https://github.com/open-cluster-management-io/addon-framework/tree/v0.8.1) imports controller-runtime, that is only used in testing. The regular code does not use controller-runtime. Currently ks/OSA's controller and agent offer no metrics.

from kubestellar.

I ran the bash E2E test, using commit 5b6b18e, and scraped the metrics. I have attached the results.

kubestellar-controller-manager.metrics.txt

ocm-status-addon-agent.metrics.txt

ocm-status-addon-controller.metrics.txt

transport-controller.metrics.txt

The processes based on controller-runtime have an impoverished set of rest_client metrics.

The OCM Status Add-On Agent also lacks any workqueue metrics.

from kubestellar.

I think it is not going to be easy to change our controller-runtime based controllers to produce the normal client-go metrics that they currently do not. Witness the complaint and lack of response to https://kubernetes.slack.com/archives/C02MRBMN00Z/p1717604240073959 . Witness the exclusive registration of just one client-go metric at https://github.com/kubernetes-sigs/controller-runtime/blob/v0.18.3/pkg/metrics/client_go_adapter.go#L51-L54 , implying that controller-runtime has other ways of collecting the same or equivalent metrics, the absence of which I suppose is because our controllers are not using the relevant part(s) of controller-runtime.

To be clear, I am not saying that we can not add our own metrics; https://book.kubebuilder.io/reference/metrics.html#publishing-additional-metrics shows how to do that.

Since we plan to shift away from using controller-runtime, I am disinclined to pursue the missing client-go metrics unless and until I hear that some of those are specifically desired.

from kubestellar.