ADR required for monitoring setup about blueprint HOT 8 CLOSED

@hemajv see if the adr doc for github alerts can be incorporated here

from blueprint.

@hemajv see if the adr doc for github alerts can be incorporated here

I think it would be easier to have a separate ADR for this.
created #16, for adding the ADR of our alerting setup

from blueprint.

Since we have decided (relevant issue) to keep a single Prometheus instance that monitors all the other component namespaces.

Now we need to decide where to keep our service-monitors/pod-monitors resources.

For the ODH components these resources will probably go in the upstream odh-manifests but we will still need to add the following overlay to the upstream monitors:

spec:
    namespaceSelector:
        matchNames:
          - opf-observatorium # The namespace where the component is deployed in

from blueprint.

If the service monitors and pod monitors are to go in the same namespace as prometheus, then I think the monitoring folder makes sense. I think we can put them in the base since they aren't environment specific, then inherit them in all the overlays -- wdyt?

from blueprint.

@HumairAK @anishasthana I was thinking of keeping just one servicemonitor resource which looks into multiple namespaces. Maybe we can make this change upstream and only overlay the list of namespaces in operatefirst overlays?

from blueprint.

I like the idea of having one servicemonitor for odh,

It would be nice if we could specify a list of namespaces as a parameter in the kfdef that would be populated in the servicemonitor so we don't need to do an override.

from blueprint.

I am not opposed to it, but we should ask the monitoring team what they think. I wonder if it's abusing the idea of service monitors.

from blueprint.

I was able to test a service monitor that was able to monitor different services in multiple namespaces.

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: multiservice-monitor
  labels:
    k8s-app: prometheus
  namespace: opf-stage
spec:
  endpoints:
    - port: metrics
    - port: 8080-tcp
  namespaceSelector:
    matchNames:
      - opf-observatorium
      - opf-jupyterhub
  selector: {}

This works well. I think we should try to get something like this pushed upstream and only set the namespaceSelector as an overlay on the operatefirst side.

from blueprint.