canonical / alertmanager-k8s-operator Goto Github PK

This charmed operator automates operation procedures of Alertmanager, the alerting component of Prometheus and Loki, among others.

Home Page: https://charmhub.io/alertmanager-k8s

License: Apache License 2.0

Python 100.00%

alerting cos hacktoberfest loki observability prometheus

alertmanager-k8s-operator's People

Contributors

Stargazers

Watchers

Forkers

dstathis isabella232 sed-i mmanciop dnplas domfleischmann ca-scribner canonical gmerold samirps varshigupta12 benhoyt neilcam amandahla nobuto-m tonyandrewmeyer

alertmanager-k8s-operator's Issues

wrong port (`None`) in Prometheus scrape job

Bug Description

When relationg to Traefik, Alermanager inform a wrong port: None

To Reproduce

Deploy cos-lite edge (Alertmanager Rev 81)
Check Prometheus targets

Environment

cos-lite edge

Relevant log output

Additional context

No response

Set external URL as a Grafana Data Source

Enhancement Proposal

Unlike Loki, if the relation between Alertmanager and Traefik is established, Alertmanager set fqdn address instead of the external one as a Grafana Data Source.

Add an action to test external integrations like Pagerduty

Enhancement Proposal

It would be helpful to trigger test alerts as an action.
Ideally it could have options for the alert body (i.e. labels) and maybe limit it to some a single receiver. This would allow us to test the alerting of a cloud without creating an actual alert (e.g. by stopping grafana).

pebble restarting alertmanager every ~5 minutes

Bug Description

After deploying cos-alerter, I was investigating why alertmanager was sending Watchdog alerts very erratically.
I looked at the alertmanager logs, and found that it was constantly restarted by pebble (see the logs below).

To Reproduce

I did not initially deploy this COS cluster, nor do I have a k8s cluster at hand to reproduce. If it helps, here is the alertmanager config:

$ juju g -a alertmanager
juju config alertmanager config_file='receivers:
- name: cos-alerter
  webhook_configs:
  - url: http://<COS_ALERTER_IP>:8080/alive?clientid=<CLUSTER_ID>&key=<REDACTED>

route:
  routes:
    - matchers:
      - alertname="Watchdog"
      - juju_charm="alertmanager-k8s"
  receiver: cos-alerter
'
juju config alertmanager cpu='' #Default#
juju config alertmanager juju-application-path='/' #Default#
juju config alertmanager juju-external-hostname='' #Default#
juju config alertmanager kubernetes-ingress-allow-http='False' #Default#
juju config alertmanager kubernetes-ingress-class='nginx' #Default#
juju config alertmanager kubernetes-ingress-ssl-passthrough='False' #Default#
juju config alertmanager kubernetes-ingress-ssl-redirect='False' #Default#
juju config alertmanager kubernetes-service-annotations='' #Default#
juju config alertmanager kubernetes-service-external-ips='' #Default#
juju config alertmanager kubernetes-service-externalname='' #Default#
juju config alertmanager kubernetes-service-loadbalancer-ip='' #Default#
juju config alertmanager kubernetes-service-loadbalancer-sourceranges='' #Default#
juju config alertmanager kubernetes-service-target-port='' #Default#
juju config alertmanager kubernetes-service-type='' #Default#
juju config alertmanager memory='' #Default#
juju config alertmanager templates_file='' #Default#
juju config alertmanager trust='True'
juju config alertmanager web_external_url='' #Default#

Environment

The COS bundle runs on a microk8s cluster. The charms are the latest/stable versions.

Relevant log output

2024-03-08T09:55:19.483Z [pebble] GET /v1/files?action=list&path=%2Fetc%2Falertmanager%2Falertmanager.cert.pem&itself=true 70.653µs 404
2024-03-08T09:55:19.503Z [pebble] GET /v1/files?action=list&path=%2Fetc%2Falertmanager%2Falertmanager.cert.pem&itself=true 146.295µs 404
2024-03-08T09:55:19.514Z [pebble] GET /v1/files?action=list&path=%2Fetc%2Falertmanager%2Falertmanager.cert.pem&itself=true 72.145µs 404
2024-03-08T09:55:20.387Z [pebble] GET /v1/files?action=list&path=%2Fetc%2Falertmanager%2Falertmanager.cert.pem&itself=true 101.049µs 404
2024-03-08T09:55:20.443Z [pebble] GET /v1/files?action=list&path=%2Fetc%2Falertmanager%2Falertmanager.cert.pem&itself=true 143.089µs 404
2024-03-08T09:55:20.545Z [pebble] GET /v1/files?action=list&path=%2Fetc%2Falertmanager%2Falertmanager.cert.pem&itself=true 131.639µs 404
2024-03-08T09:55:20.615Z [pebble] GET /v1/files?action=list&path=%2Fetc%2Falertmanager%2Falertmanager.cert.pem&itself=true 75.052µs 404
2024-03-08T09:55:20.678Z [pebble] POST /v1/files 9.761966ms 200
2024-03-08T09:55:20.686Z [pebble] POST /v1/files 6.63311ms 200
2024-03-08T09:55:20.686Z [pebble] POST /v1/files 64.852µs 200
2024-03-08T09:55:20.692Z [pebble] POST /v1/files 5.479489ms 200
2024-03-08T09:55:20.692Z [pebble] POST /v1/files 55.415µs 200
2024-03-08T09:55:20.698Z [pebble] POST /v1/files 5.435406ms 200
2024-03-08T09:55:20.698Z [pebble] POST /v1/files 46.487µs 200
2024-03-08T09:55:20.745Z [pebble] POST /v1/exec 45.532637ms 202
2024-03-08T09:55:20.788Z [pebble] GET /v1/tasks/9298/websocket/control 42.379446ms 200
2024-03-08T09:55:20.789Z [pebble] GET /v1/tasks/9298/websocket/stdio 71.586µs 200
2024-03-08T09:55:20.789Z [pebble] GET /v1/tasks/9298/websocket/stderr 55.484µs 200
2024-03-08T09:55:20.895Z [pebble] GET /v1/changes/6974/wait?timeout=4.000s 104.877935ms 200
2024-03-08T09:55:20.897Z [pebble] GET /v1/files?action=list&path=%2Fetc%2Falertmanager%2Falertmanager.cert.pem&itself=true 38.281µs 404
2024-03-08T09:55:20.898Z [pebble] POST /v1/layers 267.033µs 200
2024-03-08T09:55:20.943Z [pebble] POST /v1/services 44.59849ms 202
2024-03-08T09:55:21.073Z [pebble] GET /v1/changes/6975/wait?timeout=4.000s 128.660168ms 200
2024-03-08T09:55:26.941Z [pebble] GET /v1/plan?format=yaml 157.276µs 200
2024-03-08T09:55:26.987Z [pebble] POST /v1/services 44.899247ms 202
2024-03-08T09:55:27.032Z [alertmanager] ts=2024-03-08T09:55:27.031Z caller=main.go:594 level=info msg="Received SIGTERM, exiting gracefully..."
2024-03-08T09:55:27.039Z [pebble] Service "alertmanager" stopped
2024-03-08T09:55:27.128Z [pebble] Service "alertmanager" starting: alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/alertmanager --web.listen-address=:9093 --cluster.listen-address= --web.external-url=http://alertmanager-0.alertmanager-endpoints.<cluster-name>.svc.cluster.local:9093
2024-03-08T09:55:27.144Z [alertmanager] ts=2024-03-08T09:55:27.144Z caller=main.go:245 level=info msg="Starting Alertmanager" version="(version=0.26.0, branch=HEAD, revision=d7b4f0c7)"
2024-03-08T09:55:27.144Z [alertmanager] ts=2024-03-08T09:55:27.144Z caller=main.go:246 level=info build_context="(go=go1.18.10, platform=linux/amd64, user=root@rockcraft-alertmanager-on-amd64-for-amd64-797928, date=2024-02-13T14:16:29Z, tags=unknown)"
2024-03-08T09:55:27.173Z [alertmanager] ts=2024-03-08T09:55:27.173Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml
2024-03-08T09:55:27.174Z [alertmanager] ts=2024-03-08T09:55:27.174Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml
2024-03-08T09:55:27.177Z [alertmanager] ts=2024-03-08T09:55:27.177Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9093
2024-03-08T09:55:27.177Z [alertmanager] ts=2024-03-08T09:55:27.177Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9093
2024-03-08T09:55:28.170Z [pebble] GET /v1/changes/6976/wait?timeout=4.000s 1.182784446s 200
2024-03-08T09:59:23.949Z [pebble] GET /v1/files?action=list&path=%2Fetc%2Falertmanager%2Falertmanager.cert.pem&itself=true 45.876µs 404
2024-03-08T09:59:23.963Z [pebble] GET /v1/files?action=list&path=%2Fetc%2Falertmanager%2Falertmanager.cert.pem&itself=true 39.204µs 404
2024-03-08T09:59:23.970Z [pebble] GET /v1/files?action=list&path=%2Fetc%2Falertmanager%2Falertmanager.cert.pem&itself=true 34.515µs 404
2024-03-08T09:59:29.684Z [pebble] GET /v1/files?action=list&path=%2Fetc%2Falertmanager%2Falertmanager.cert.pem&itself=true 42.861µs 404
2024-03-08T09:59:29.718Z [pebble] GET /v1/files?action=list&path=%2Fetc%2Falertmanager%2Falertmanager.cert.pem&itself=true 48.901µs 404
2024-03-08T09:59:29.781Z [pebble] GET /v1/files?action=list&path=%2Fetc%2Falertmanager%2Falertmanager.cert.pem&itself=true 50.245µs 404
2024-03-08T09:59:29.838Z [pebble] GET /v1/files?action=list&path=%2Fetc%2Falertmanager%2Falertmanager.cert.pem&itself=true 44.284µs 404
2024-03-08T09:59:29.884Z [pebble] POST /v1/files 6.790889ms 200
2024-03-08T09:59:29.891Z [pebble] POST /v1/files 6.069903ms 200
2024-03-08T09:59:29.891Z [pebble] POST /v1/files 37.4µs 200
2024-03-08T09:59:29.897Z [pebble] POST /v1/files 5.120127ms 200
2024-03-08T09:59:29.897Z [pebble] POST /v1/files 45.776µs 200
2024-03-08T09:59:29.903Z [pebble] POST /v1/files 5.680761ms 200
2024-03-08T09:59:29.904Z [pebble] POST /v1/files 51.216µs 200
2024-03-08T09:59:29.998Z [pebble] POST /v1/exec 93.772776ms 202
2024-03-08T09:59:29.999Z [pebble] GET /v1/tasks/9302/websocket/control 139.462µs 200
2024-03-08T09:59:30.000Z [pebble] GET /v1/tasks/9302/websocket/stdio 65.343µs 200
2024-03-08T09:59:30.000Z [pebble] GET /v1/tasks/9302/websocket/stderr 50.065µs 200
2024-03-08T09:59:30.097Z [pebble] GET /v1/changes/6977/wait?timeout=4.000s 95.534533ms 200
2024-03-08T09:59:30.099Z [pebble] GET /v1/files?action=list&path=%2Fetc%2Falertmanager%2Falertmanager.cert.pem&itself=true 39.754µs 404
2024-03-08T09:59:30.100Z [pebble] POST /v1/layers 223.682µs 200
2024-03-08T09:59:30.189Z [pebble] POST /v1/services 87.501436ms 202
2024-03-08T09:59:30.280Z [pebble] GET /v1/changes/6978/wait?timeout=4.000s 90.123621ms 200
2024-03-08T09:59:31.147Z [pebble] GET /v1/plan?format=yaml 233.52µs 200
2024-03-08T09:59:31.239Z [pebble] POST /v1/services 91.600339ms 202
2024-03-08T09:59:31.240Z [alertmanager] ts=2024-03-08T09:59:31.240Z caller=main.go:594 level=info msg="Received SIGTERM, exiting gracefully..."
2024-03-08T09:59:31.248Z [pebble] Service "alertmanager" stopped
2024-03-08T09:59:31.352Z [pebble] Service "alertmanager" starting: alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/alertmanager --web.listen-address=:9093 --cluster.listen-address= --web.external-url=http://alertmanager-0.alertmanager-endpoints.<cluster-name>.svc.cluster.local:9093
2024-03-08T09:59:31.366Z [alertmanager] ts=2024-03-08T09:59:31.366Z caller=main.go:245 level=info msg="Starting Alertmanager" version="(version=0.26.0, branch=HEAD, revision=d7b4f0c7)"
2024-03-08T09:59:31.366Z [alertmanager] ts=2024-03-08T09:59:31.366Z caller=main.go:246 level=info build_context="(go=go1.18.10, platform=linux/amd64, user=root@rockcraft-alertmanager-on-amd64-for-amd64-797928, date=2024-02-13T14:16:29Z, tags=unknown)"
2024-03-08T09:59:31.391Z [alertmanager] ts=2024-03-08T09:59:31.391Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml
2024-03-08T09:59:31.391Z [alertmanager] ts=2024-03-08T09:59:31.391Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml
2024-03-08T09:59:31.394Z [alertmanager] ts=2024-03-08T09:59:31.394Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9093
2024-03-08T09:59:31.394Z [alertmanager] ts=2024-03-08T09:59:31.394Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9093
2024-03-08T09:59:32.392Z [pebble] GET /v1/changes/6979/wait?timeout=4.000s 1.151803833s 200
2024-03-08T10:04:54.987Z [pebble] GET /v1/files?action=list&path=%2Fetc%2Falertmanager%2Falertmanager.cert.pem&itself=true 53.309µs 404
2024-03-08T10:04:55.002Z [pebble] GET /v1/files?action=list&path=%2Fetc%2Falertmanager%2Falertmanager.cert.pem&itself=true 93.055µs 404
2024-03-08T10:04:55.010Z [pebble] GET /v1/files?action=list&path=%2Fetc%2Falertmanager%2Falertmanager.cert.pem&itself=true 57.58µs 404
2024-03-08T10:05:00.798Z [pebble] GET /v1/files?action=list&path=%2Fetc%2Falertmanager%2Falertmanager.cert.pem&itself=true 55.334µs 404
2024-03-08T10:05:00.841Z [pebble] GET /v1/files?action=list&path=%2Fetc%2Falertmanager%2Falertmanager.cert.pem&itself=true 75.532µs 404
2024-03-08T10:05:00.930Z [pebble] GET /v1/files?action=list&path=%2Fetc%2Falertmanager%2Falertmanager.cert.pem&itself=true 60.214µs 404
2024-03-08T10:05:01.004Z [pebble] GET /v1/files?action=list&path=%2Fetc%2Falertmanager%2Falertmanager.cert.pem&itself=true 45.505µs 404
2024-03-08T10:05:01.056Z [pebble] POST /v1/files 8.33658ms 200
2024-03-08T10:05:01.079Z [pebble] POST /v1/files 23.099067ms 200
2024-03-08T10:05:01.080Z [pebble] POST /v1/files 49.173µs 200
2024-03-08T10:05:01.091Z [pebble] POST /v1/files 10.452614ms 200
2024-03-08T10:05:01.091Z [pebble] POST /v1/files 38.662µs 200
2024-03-08T10:05:01.115Z [pebble] POST /v1/files 23.346963ms 200
2024-03-08T10:05:01.116Z [pebble] POST /v1/files 69.56µs 200
2024-03-08T10:05:01.162Z [pebble] POST /v1/exec 45.465995ms 202
2024-03-08T10:05:01.207Z [pebble] GET /v1/tasks/9306/websocket/control 43.880841ms 200
2024-03-08T10:05:01.207Z [pebble] GET /v1/tasks/9306/websocket/stdio 74.14µs 200
2024-03-08T10:05:01.208Z [pebble] GET /v1/tasks/9306/websocket/stderr 37.782µs 200
2024-03-08T10:05:01.318Z [pebble] GET /v1/changes/6980/wait?timeout=4.000s 108.934344ms 200
2024-03-08T10:05:01.320Z [pebble] GET /v1/files?action=list&path=%2Fetc%2Falertmanager%2Falertmanager.cert.pem&itself=true 41.798µs 404
2024-03-08T10:05:01.321Z [pebble] POST /v1/layers 242.957µs 200
2024-03-08T10:05:01.366Z [pebble] POST /v1/services 44.264083ms 202
2024-03-08T10:05:01.498Z [pebble] GET /v1/changes/6981/wait?timeout=4.000s 131.152483ms 200
2024-03-08T10:05:02.364Z [pebble] GET /v1/plan?format=yaml 183.685µs 200
2024-03-08T10:05:02.409Z [pebble] POST /v1/services 44.809519ms 202
2024-03-08T10:05:02.455Z [alertmanager] ts=2024-03-08T10:05:02.455Z caller=main.go:594 level=info msg="Received SIGTERM, exiting gracefully..."
2024-03-08T10:05:02.460Z [pebble] Service "alertmanager" stopped
2024-03-08T10:05:02.552Z [pebble] Service "alertmanager" starting: alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/alertmanager --web.listen-address=:9093 --cluster.listen-address= --web.external-url=http://alertmanager-0.alertmanager-endpoints.<cluster-name>.svc.cluster.local:9093
2024-03-08T10:05:02.571Z [alertmanager] ts=2024-03-08T10:05:02.571Z caller=main.go:245 level=info msg="Starting Alertmanager" version="(version=0.26.0, branch=HEAD, revision=d7b4f0c7)"
2024-03-08T10:05:02.571Z [alertmanager] ts=2024-03-08T10:05:02.571Z caller=main.go:246 level=info build_context="(go=go1.18.10, platform=linux/amd64, user=root@rockcraft-alertmanager-on-amd64-for-amd64-797928, date=2024-02-13T14:16:29Z, tags=unknown)"
2024-03-08T10:05:02.603Z [alertmanager] ts=2024-03-08T10:05:02.603Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml
2024-03-08T10:05:02.604Z [alertmanager] ts=2024-03-08T10:05:02.604Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml
2024-03-08T10:05:02.610Z [alertmanager] ts=2024-03-08T10:05:02.610Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9093
2024-03-08T10:05:02.610Z [alertmanager] ts=2024-03-08T10:05:02.610Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9093
2024-03-08T10:05:03.598Z [pebble] GET /v1/changes/6982/wait?timeout=4.000s 1.188174618s 200

Additional context

No response

Need to fetch-lib prometheus_scrape

#103 was merged before the latest prometheus_scrape was published (canonical/prometheus-k8s-operator#384), so the issue described in canonical/prometheus-k8s-operator#368 is probably still present for alertmanager.

Need to publish-lib prometheus_scrape and fetch-lib here.

Rename Repo

Per the naming guidelines recently published, this repo should be renamed alertmanager-k8s-operator. Will need a ticket into IS.

Use fqdn and correct scheme in `alertmanager_dispatch`

Currently we have two problems with the address alertmanager_dispatch is forwarding to prometheus:

alertmanager-k8s-operator/lib/charms/alertmanager_k8s/v0/alertmanager_dispatch.py

Line 291 in 99dc29b

 self._external_url = external_url or (lambda: f"http://{socket.getfqdn()}:{api_port}") 

It is always "http" for fqdn.
It might be ingress address, whereas we only want fqdn for now. When one of the following is impl'd, we can go back to potentially forwarding an ingress address:
- canonical/prometheus-k8s-operator#530
- canonical/operator#970

Future TLS work: Encrypt traffic of the gossip ring.

Tracking issue: canonical/cos-lite-bundle#75

Note that encrypting gossip traffic is an experimental feature at the moment (see also: prometheus/alertmanager#3403).

`amtool check-config` and go into Blocked instead of crashloop

Enhancement Proposal

Currently, an invalid alertmanager.yml, submitted via juju config, would result in a crashloop.
Instead, could use amtool (already present in the workload container) to validate a config before restarting the service.

cc: @Gleland

TLS part 2: Enable TLS for Prometheus alerting and self-monitoring

Tracking issue: canonical/cos-lite-bundle#75

#164 enabled TLS for Alertmanager but both scrape target and cluster_info are still HTTP.

Potentially blocked by canonical/tls-certificates-interface#46 (prom may not be able to scrape).

Related: canonical/charm-microk8s#88

TODO:

Update the self-scrape config to use https scheme
Update the cluster_info logic in the alerting lib so prometheus charm generates the alertmanagers section correctly.
Think what needs to be changed so that prometheus WITHOUT tls is able to scrape alertmanager WITH tls enabled (probably need to set insecure-skip-verify somewhere in prometheus).

Support proxies

          Looks like this should just be setting the http_proxy environment variable for Grafana based on a config value. We should likely do this for Alertmanager as well

Originally posted by @dstathis in canonical/grafana-k8s-operator#240 (comment)

The readme is not consistent with the actual usage

The readme is not consistent with the actual usage. When charmhub got rid of the Docs tab, the content from discourse was moved to the Descriptions tab, making it super confusing as it now contains TWO overviews with entirely different content.

We should

Make sure the overview taken from the readme is up to date
Incorporate everything valuable from the Docs - Overview discourse post and then delete that post.
Incorporate everything valuable from the Relations discourse post and then delete that post.

Alertmanger fails hook "config changed" due to assert private_key is not None

Bug Description

In test run: https://solutions.qa.canonical.com/testruns/f5482205-0dd8-4ab8-9da7-32485bdb5158, deployment fails because alertmanager/1 is in error state:

App           Version  Status   Scale  Charm                     Channel  Rev  Address         Exposed  Message
alertmanager  0.25.0   waiting      2  alertmanager-k8s          stable    96  10.152.183.166  no       waiting for units to settle down
avalanche              active       2  avalanche-k8s             edge      36  10.152.183.172  no       
ca                     active       1  self-signed-certificates  edge      52  10.152.183.249  no       
catalogue              active       1  catalogue-k8s             stable    31  10.152.183.198  no       
external-ca            active       1  self-signed-certificates  edge      52  10.152.183.47   no       
grafana       9.2.1    active       1  grafana-k8s               stable    93  10.152.183.196  no       
loki          2.7.4    active       1  loki-k8s                  stable   105  10.152.183.89   no       
prometheus    2.47.2   active       1  prometheus-k8s            stable   156  10.152.183.39   no       
traefik       2.10.4   active       1  traefik-k8s               stable   166  10.246.167.226  no       

Unit             Workload  Agent  Address      Ports  Message
alertmanager/0*  active    idle   10.1.250.76         
alertmanager/1   error     idle   10.1.198.73         hook failed: "config-changed"
avalanche/0*     active    idle   10.1.250.73         
avalanche/1      active    idle   10.1.198.70         
ca/0*            active    idle   10.1.250.74         
catalogue/0*     active    idle   10.1.27.6           
external-ca/0*   active    idle   10.1.27.5           
grafana/0*       active    idle   10.1.198.74         
loki/0*          active    idle   10.1.27.7           
prometheus/0*    active    idle   10.1.250.79         
traefik/0*       active    idle   10.1.250.78

To Reproduce

1.Deploy cos-lite bundle

Environment

charm is latest/stable, rev 96

Relevant log output

unit-alertmanager-1: 2023-12-14 23:04:05 DEBUG jujuc running hook tool "juju-log" for alertmanager/1-config-changed-8622918567248683281
unit-alertmanager-1: 2023-12-14 23:04:05 ERROR unit.alertmanager/1.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 561, in <module>
    main(AlertmanagerCharm)
  File "/var/lib/juju/agents/unit-alertmanager-1/charm/venv/ops/main.py", line 441, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-alertmanager-1/charm/venv/ops/main.py", line 149, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-alertmanager-1/charm/venv/ops/framework.py", line 344, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-alertmanager-1/charm/venv/ops/framework.py", line 833, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-alertmanager-1/charm/venv/ops/framework.py", line 922, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-alertmanager-1/charm/lib/charms/observability_libs/v0/cert_handler.py", line 220, in _on_config_changed
    self._generate_csr(renew=True)
  File "/var/lib/juju/agents/unit-alertmanager-1/charm/lib/charms/observability_libs/v0/cert_handler.py", line 240, in _generate_csr
    assert private_key is not None  # for type checker
AssertionError



### Additional context

More logs, including crashdumps, can be found under: https://oil-jenkins.canonical.com/artifacts/f5482205-0dd8-4ab8-9da7-32485bdb5158/index.html

Data missing from panels in Grafana

Bug Description

Hi, I deployed latest COS Alertmanager, integrated it with Grafana, but some panels are missing data in the dashboard:

To Reproduce

07-03-cos-bundle.txt

Environment

juju-status.txt

Relevant log output

Additional context

No response

Perform proper acceptance tests of Alertmanager, Karma, alert routes and channels

add integration tests for receiver templates

alerts are sent to user-provided receiver
alerts use user-provided templates

Failed: juju run alertmanager/0 show-config

Bug Description

I tried to run:

juju run alertmanager/0 show-config

But got:

2024-02-07T14:32:19.610Z [container-agent] 2024-02-07 14:32:19 DEBUG juju-log Emitting Juju event show_config_action.
2024-02-07T14:32:19.654Z [container-agent] 2024-02-07 14:32:19 ERROR juju-log Uncaught exception while in charm code:
2024-02-07T14:32:19.654Z [container-agent] Traceback (most recent call last):
2024-02-07T14:32:19.654Z [container-agent]   File "./src/charm.py", line 572, in <module>
2024-02-07T14:32:19.654Z [container-agent]     main(AlertmanagerCharm)
2024-02-07T14:32:19.654Z [container-agent]   File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/main.py", line 441, in main
2024-02-07T14:32:19.654Z [container-agent]     _emit_charm_event(charm, dispatcher.event_name)
2024-02-07T14:32:19.654Z [container-agent]   File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/main.py", line 149, in _emit_charm_event
2024-02-07T14:32:19.654Z [container-agent]     event_to_emit.emit(*args, **kwargs)
2024-02-07T14:32:19.654Z [container-agent]   File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/framework.py", line 344, in emit
2024-02-07T14:32:19.654Z [container-agent]     framework._emit(event)
2024-02-07T14:32:19.654Z [container-agent]   File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/framework.py", line 833, in _emit
2024-02-07T14:32:19.654Z [container-agent]     self._reemit(event_path)
2024-02-07T14:32:19.654Z [container-agent]   File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/framework.py", line 922, in _reemit
2024-02-07T14:32:19.654Z [container-agent]     custom_handler(event)
2024-02-07T14:32:19.654Z [container-agent]   File "./src/charm.py", line 286, in _on_show_config_action
2024-02-07T14:32:19.654Z [container-agent]     if not self.alertmanager_workload.is_ready():
2024-02-07T14:32:19.654Z [container-agent] TypeError: 'bool' object is not callable

To Reproduce

juju run alertmanager/0 show-config

Environment

Juju 3.1.6
Alertmanager latest/edge 98

Relevant log output

Additional context

No response

Add an integration test insuring that alertmanager isn't restarted too often

Enhancement Proposal

We should make sure we use the hot reload option as much as possible, to avoid unnecessary outages.
Related: #212.

Support TLS certificate validation for receivers

Enhancement Proposal

I am using the charm on edge rev 98 and am trying to use a slack receiver, but it fails with

:~$ kubectl logs alertmanager-0 -c alertmanager | less

2024-02-20T14:25:35.026Z [alertmanager] ts=2024-02-20T14:25:35.026Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml
2024-02-20T14:25:35.027Z [alertmanager] ts=2024-02-20T14:25:35.027Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml
2024-02-20T14:25:35.181Z [alertmanager] ts=2024-02-20T14:25:35.181Z caller=notify.go:745 level=warn component=dispatcher receiver=mattermost-notifications integration=slack[0] aggrGroup="{}/{juju_model=\"prod-github-runner\"}:{juju_application=\"grafana-agent-without-dashboard\", juju_model=\"prod-github-runner\", juju_model_uuid=\"4572b1cc-0a39-40b7-818d-c68ed553f11a\"}" msg="Notify attempt failed, will retry later" attempts=1 err="Post \"<redacted>\": x509: certificate signed by unknown authority"

The configuration I provided looks like this

global:
  http_config:
    tls_config:
      insecure_skip_verify: false
receivers:
- name: 'default'
- name: 'mattermost-notifications'
  slack_configs:
    - send_resolved: true
      api_url: 'https://fake-webhook-url'
route:
  group_by:
    - juju_application
    - juju_model_uuid
    - juju_model
  group_interval: 5m
  group_wait: 30s
  receiver: 'default'
  repeat_interval: 4h
  routes:
    - receiver: 'mattermost-notifications'
      matchers:
       - juju_model = prod-github-runner

The server in the api_url doesn't use a self-signed certificate, but one from Let's Encrypt. The reason certificate validation fails is probably because there are no ca certs in the container.

~$ kubectl exec -it alertmanager-0 -c alertmanager -- ls -l /etc/ssl/certs
total 0

I can fix this by setting insecure_skip_verify to true, but I think the charm should also support validation for common ca's or at least provide the ability to provide a ca file for https://prometheus.io/docs/alerting/latest/configuration/#tls_config.

Missing GPG key

Bug Description

During a juju deployment of the mon stack Grafana does not deploy with error

To Reproduce

From documentation on deploying mon stack

juju deploy charmed-kubernetes --overlay ./logging-egf-overlay.yaml --overlay ./monitoring-pgt-overlay.yaml --overlay ./kubernetes-worker.yaml
juju status

Environment

on lxd in a maas deployed stack

Relevant log output

2023-09-11 20:39:45 INFO juju.worker.uniter resolver.go:155 awaiting error resolution for "install" hook
2023-09-11 20:43:05 INFO juju.worker.uniter resolver.go:155 awaiting error resolution for "install" hook
2023-09-11 20:44:45 INFO juju.worker.uniter resolver.go:155 awaiting error resolution for "install" hook
2023-09-11 20:44:46 ERROR unit.grafana/0.juju-log server.go:316 Unable to find implementation for relation: requires of register-application
2023-09-11 20:44:55 WARNING unit.grafana/0.install logger.go:60 W: GPG error: https://packages.grafana.com/oss/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 963FA27710458545
2023-09-11 20:44:55 WARNING unit.grafana/0.install logger.go:60 E: The repository 'https://packages.grafana.com/oss/deb stable InRelease' is not signed.
2023-09-11 20:45:08 WARNING unit.grafana/0.install logger.go:60 W: GPG error: https://packages.grafana.com/oss/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 963FA27710458545
2023-09-11 20:45:08 WARNING unit.grafana/0.install logger.go:60 E: The repository 'https://packages.grafana.com/oss/deb stable InRelease' is not signed.
2023-09-11 20:45:21 WARNING unit.grafana/0.install logger.go:60 W: GPG error: https://packages.grafana.com/oss/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 963FA27710458545
2023-09-11 20:45:21 WARNING unit.grafana/0.install logger.go:60 E: The repository 'https://packages.grafana.com/oss/deb stable InRelease' is not signed.
2023-09-11 20:45:34 WARNING unit.grafana/0.install logger.go:60 W: GPG error: https://packages.grafana.com/oss/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 963FA27710458545
2023-09-11 20:45:34 WARNING unit.grafana/0.install logger.go:60 E: The repository 'https://packages.grafana.com/oss/deb stable InRelease' is not signed.
2023-09-11 20:45:48 WARNING unit.grafana/0.install logger.go:60 W: GPG error: https://packages.grafana.com/oss/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 963FA27710458545
2023-09-11 20:45:48 WARNING unit.grafana/0.install logger.go:60 E: The repository 'https://packages.grafana.com/oss/deb stable InRelease' is not signed.
2023-09-11 20:46:01 WARNING unit.grafana/0.install logger.go:60 W: GPG error: https://packages.grafana.com/oss/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 963FA27710458545
2023-09-11 20:46:01 WARNING unit.grafana/0.install logger.go:60 E: The repository 'https://packages.grafana.com/oss/deb stable InRelease' is not signed.
2023-09-11 20:46:14 WARNING unit.grafana/0.install logger.go:60 W: GPG error: https://packages.grafana.com/oss/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 963FA27710458545
2023-09-11 20:46:14 WARNING unit.grafana/0.install logger.go:60 E: The repository 'https://packages.grafana.com/oss/deb stable InRelease' is not signed.
2023-09-11 20:46:27 WARNING unit.grafana/0.install logger.go:60 W: GPG error: https://packages.grafana.com/oss/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 963FA27710458545
2023-09-11 20:46:27 WARNING unit.grafana/0.install logger.go:60 E: The repository 'https://packages.grafana.com/oss/deb stable InRelease' is not signed.
2023-09-11 20:46:40 WARNING unit.grafana/0.install logger.go:60 W: GPG error: https://packages.grafana.com/oss/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 963FA27710458545
2023-09-11 20:46:40 WARNING unit.grafana/0.install logger.go:60 E: The repository 'https://packages.grafana.com/oss/deb stable InRelease' is not signed.
2023-09-11 20:46:54 WARNING unit.grafana/0.install logger.go:60 W: GPG error: https://packages.grafana.com/oss/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 963FA27710458545
2023-09-11 20:46:54 WARNING unit.grafana/0.install logger.go:60 E: The repository 'https://packages.grafana.com/oss/deb stable InRelease' is not signed.
2023-09-11 20:47:07 WARNING unit.grafana/0.install logger.go:60 W: GPG error: https://packages.grafana.com/oss/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 963FA27710458545
2023-09-11 20:47:07 WARNING unit.grafana/0.install logger.go:60 E: The repository 'https://packages.grafana.com/oss/deb stable InRelease' is not signed.
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60 W: GPG error: https://packages.grafana.com/oss/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 963FA27710458545
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60 E: The repository 'https://packages.grafana.com/oss/deb stable InRelease' is not signed.
2023-09-11 20:47:20 ERROR unit.grafana/0.juju-log server.go:316 Hook error:
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['apt-get', 'update']' returned non-zero exit status 100.
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60 Traceback (most recent call last):
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60   File "/var/lib/juju/agents/unit-grafana-0/charm/hooks/install", line 22, in <module>
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60     main()
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60   File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60     bus.dispatch(restricted=restricted_mode)
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60   File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60     _invoke(other_handlers)
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60   File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60     handler.invoke()
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60   File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60     self._action(*args)
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60   File "/var/lib/juju/agents/unit-grafana-0/charm/reactive/grafana.py", line 260, in install_packages
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60     fetch.configure_sources(update=True)
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60   File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/charmhelpers/fetch/__init__.py", line 149, in configure_sources
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60     _fetch_update(fatal=True)
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60   File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/charmhelpers/fetch/ubuntu.py", line 400, in apt_update
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60     _run_apt_command(cmd, fatal)
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60   File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/charmhelpers/fetch/ubuntu.py", line 945, in _run_apt_command
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60     _run_with_retries(
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60   File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/charmhelpers/fetch/ubuntu.py", line 922, in _run_with_retries
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60     result = subprocess.check_call(cmd, env=env, **kwargs)
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60   File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60     raise CalledProcessError(retcode, cmd)
2023-09-11 20:47:20 WARNING unit.grafana/0.install logger.go:60 subprocess.CalledProcessError: Command '['apt-get', 'update']' returned non-zero exit status 100.
2023-09-11 20:47:20 ERROR juju.worker.uniter.operation runhook.go:153 hook "install" (via explicit, bespoke hook script) failed: exit status 1
2023-09-11 20:47:20 INFO juju.worker.uniter resolver.go:155 awaiting error resolution for "install" hook
2023-09-11 20:48:34 INFO juju.worker.uniter resolver.go:155 awaiting error resolution for "install" hook
2023-09-11 20:52:20 INFO juju.worker.uniter resolver.go:155 awaiting error resolution for "install" hook

which repeats

Additional context

Correct with
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
followed by a
sudo apt-get update

then juju took over and installed the needed parts and the rest of the env came up

provider should NOT become unready on relation departed

https://github.com/canonical/alertmanager-operator/blob/489c32cc35131be42d601746ae169d6408cc1adb/src/charm.py#L431-L435

Create alert rule for cos-alerter

Enhancement Proposal

We need an alert rule set up which will be always firing for cos-alerter.

group_by parameter should not be hardcoded

Enhancement Proposal

The possibility of setting the "group_by" parameter to any value is needed. This is useful, e.g. when integrating alertmanager with external monitoring solutions.

AlertmanagerNotificationsFailed will fire forever if fired once

With AlertmanagerNotificationsFailed's rule query, once the alert fires once (i.e., there is at least a single notification failure), it'll remain fired till the end of times.

Something based on rate of changed or something similar is probably what's needed here.

Only leader scrape jobs are sent to Prometheus

Enhancement Proposal

As @simskij said in this comment Alertmanager is not providing a scrape job for all units, but only for the leader.

This is because Alertmanager requires ingress-per-app:

requires:
  ingress:
    interface: ingress
    limit: 1
    description: |
      Alertmanager typically needs a "per app" ingress, which is available in the traefik charm[1].

      [1] https://charmhub.io/traefik-k8s

We need ingress-per-app so alerts can be seen from outside the k8s cluster, but in the other hand we need ingress-per-unit so we can send unit scrapes jobs to Prometheus.

Use the Dockerhub Ubuntu ROCK instead of the ghcr.io one

Enhancement Proposal

See title. Replace the upstream-source with our oci-factory ROCK using the major version tag.

Wrong URL provided to Prometheus when the relation with Traefik is established.

Bug Description

An internal URL is provided to Prometheus when the relation with Traefik is established:

To Reproduce

Deploy cos-lite: juju deploy cos-lite --channel=edge --trust
Once all units are activeand idle, check the targets in Prometheus.

Environment

juju 3.0.2
micork8s v1.25.6
cos-lite: edge

Relevant log output

No erros in juju debug-log

Additional context

Charm versions:

$ juju status --color --relations           

Model  Controller           Cloud/Region        Version  SLA          Timestamp
cos    charm-dev-batteries  microk8s/localhost  3.0.2    unsupported  16:38:04-03:00

App           Version  Status  Scale  Charm             Channel  Rev  Address         Exposed  Message
alertmanager  0.23.0   active      1  alertmanager-k8s  edge      53  10.152.183.232  no       
catalogue              active      1  catalogue-k8s     edge      13  10.152.183.209  no       
grafana       9.2.1    active      1  grafana-k8s       edge      69  10.152.183.52   no       
loki          2.4.1    active      1  loki-k8s          edge      65  10.152.183.59   no       
prometheus    2.42.0   active      1  prometheus-k8s    edge     110  10.152.183.71   no       
traefik       2.9.6    active      1  traefik-k8s       edge     116  192.168.122.10  no       

Unit             Workload  Agent  Address      Ports  Message
alertmanager/0*  active    idle   10.1.36.93          
catalogue/0*     active    idle   10.1.36.124         
grafana/0*       active    idle   10.1.36.109         
loki/0*          active    idle   10.1.36.112         
prometheus/0*    active    idle   10.1.36.96          
traefik/0*       active    idle   10.1.36.81          

Relation provider                   Requirer                     Interface              Type     Message
alertmanager:alerting               loki:alertmanager            alertmanager_dispatch  regular  
alertmanager:alerting               prometheus:alertmanager      alertmanager_dispatch  regular  
alertmanager:grafana-dashboard      grafana:grafana-dashboard    grafana_dashboard      regular  
alertmanager:grafana-source         grafana:grafana-source       grafana_datasource     regular  
alertmanager:replicas               alertmanager:replicas        alertmanager_replica   peer     
alertmanager:self-metrics-endpoint  prometheus:metrics-endpoint  prometheus_scrape      regular  
catalogue:catalogue                 alertmanager:catalogue       catalogue              regular  
catalogue:catalogue                 grafana:catalogue            catalogue              regular  
catalogue:catalogue                 prometheus:catalogue         catalogue              regular  
grafana:grafana                     grafana:grafana              grafana_peers          peer     
grafana:metrics-endpoint            prometheus:metrics-endpoint  prometheus_scrape      regular  
loki:grafana-dashboard              grafana:grafana-dashboard    grafana_dashboard      regular  
loki:grafana-source                 grafana:grafana-source       grafana_datasource     regular  
loki:metrics-endpoint               prometheus:metrics-endpoint  prometheus_scrape      regular  
prometheus:grafana-dashboard        grafana:grafana-dashboard    grafana_dashboard      regular  
prometheus:grafana-source           grafana:grafana-source       grafana_datasource     regular  
prometheus:prometheus-peers         prometheus:prometheus-peers  prometheus_peers       peer     
traefik:ingress                     alertmanager:ingress         ingress                regular  
traefik:ingress                     catalogue:ingress            ingress                regular  
traefik:ingress-per-unit            loki:ingress                 ingress_per_unit       regular  
traefik:ingress-per-unit            prometheus:ingress           ingress_per_unit       regular  
traefik:metrics-endpoint            prometheus:metrics-endpoint  prometheus_scrape      regular  
traefik:traefik-route               grafana:ingress              traefik_route          regular

Crash due to missing relation

Bug Description

In the scenario where Alertmanager is related to something, in this example Prometheus, and the remote charm is removed, the next update-status event causes alertmanager to go into an error state.

To Reproduce

juju deploy cos-lite --channel edge --trust
juju remove-application prometheus --force
wait

Environment

Relevant log output

$ juju status                        
Model  Controller  Cloud/Region        Version  SLA          Timestamp
cos    uk8s        microk8s/localhost  2.9.28   unsupported  14:41:46+02:00

App           Version  Status   Scale  Charm               Channel  Rev  Address         Exposed  Message
alertmanager           waiting      1  alertmanager-k8s    edge      14  10.152.183.203  no       installing agent
exporter               active       1  node-exporter-k8s              7  10.152.183.53   no       
grafana                active       1  grafana-k8s         edge      36  10.152.183.181  no       
kube-metrics           active       1  kube-metrics-k8s              10  10.152.183.87   no       
metrics                active       1  kube-state-metrics             0  10.152.183.202  no       
prometheus             active       1  prometheus-k8s      edge      30  10.152.183.16   no       
zinc                   active       1  zinc-k8s                       5  10.152.183.97   no       

Unit             Workload  Agent  Address     Ports  Message
alertmanager/0*  error     idle   10.1.14.40         hook failed: "update-status"
exporter/0*      active    idle   10.1.14.20         
grafana/0*       active    idle   10.1.14.50         
kube-metrics/0*  active    idle   10.1.14.54         
metrics/0*       active    idle   10.1.14.52         
prometheus/0*    active    idle   10.1.14.15         
zinc/0*          active    idle   10.1.14.47 


$ juju debug-log

unit-alertmanager-0: 14:40:15 INFO juju.worker.uniter awaiting error resolution for "update-status" hook
unit-alertmanager-0: 14:40:54 INFO juju.worker.uniter awaiting error resolution for "update-status" hook
unit-alertmanager-0: 14:40:54 INFO unit.alertmanager/0.juju-log alertmanager 0.21.0 is up and running (uptime: 2022-04-25T12:23:48.939Z); cluster mode: disabled, with 0 peers
unit-alertmanager-0: 14:40:54 ERROR unit.alertmanager/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/model.py", line 1545, in _run
    result = run(args, **kwargs)
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-alertmanager-0/network-get', 'alerting', '-r', '17', '--format=json')' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./src/charm.py", line 491, in <module>
    main(AlertmanagerCharm, use_juju_for_storage=True)
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/main.py", line 431, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/main.py", line 142, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/framework.py", line 283, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/framework.py", line 743, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/framework.py", line 790, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 458, in _on_update_status
    self._common_exit_hook()
  File "./src/charm.py", line 380, in _common_exit_hook
    self.alertmanager_provider.update_relation_data()
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/lib/charms/alertmanager_k8s/v0/alertmanager_dispatch.py", line 282, in update_relation_data
    relation.data[self.charm.unit].update(self._generate_relation_data(relation))
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/lib/charms/alertmanager_k8s/v0/alertmanager_dispatch.py", line 262, in _generate_relation_data
    self.charm.model.get_binding(relation).network.bind_address, self.api_port
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/model.py", line 556, in network
    self._network = Network(self._backend.network_get(self.name, self._relation_id))
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/model.py", line 1801, in network_get
    return self._run(*cmd, return_output=True, use_json=True)
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/model.py", line 1547, in _run
    raise ModelError(e.stderr)
ops.model.ModelError: b'ERROR relation 17 not found (not found)\n'

Additional context

No response

Add scheme to alertmanager client

Currently the client code has http hard-coded:

alertmanager-k8s-operator/src/alertmanager_client.py

Line 38 in 1ed7c14

 self.base_url = urllib.parse.urljoin(f"http://{address}:{port}/", web_route_prefix) 

Ref: https://github.com/canonical/prometheus-k8s-operator/blob/d33f51f39d990de3b8dcb9436bf69291a8e8b891/src/prometheus_client.py#L20

Add an interface allowing sending Alertmanager's configuration through the relation

Enhancement Proposal

Hello,

On behalf of the Telco team, I'd like to propose adding an interface similar to Prometheus's prometheus_remote_write, which would allow configuring the Alertmanager through the relation data.
Main reason for this change is the fact, that Magma uses an external component, called alertmanager-configurer, to update Alertmanager with the configuration created via Magma NMS (GUI). Additionally, we believe that such change would add to the overall flexibility of the alertmanager-k8s charm.

From the implementation perspective, this would work just like in the Prometheus - consumer of the remote-write relation would push the config to the relation data bag and the charm would simply save this config to the alertmanager.yml file and reload the service.

Thanks in advance for looking into this and let the discussion begin.

BR,
Bartek

use goal state before updating relation data

To reduce the typical event storm on startup/upgrade/rescale, one could compare current state to goal state before updating relation data (peer relation data in particular).

self.app.planned_units: canonical/operator#597

version freeze everything

Before promoting the alertmanager-k8s charm to beta it might be a good idea to version lock a few things:

pip packages
image versions
add to metadata: assumes: juju >= 2.9.23
juju version for ci itests:

    - name: Setup operator environment
      uses: charmed-kubernetes/actions-operator@main
      with:
        juju-channel: latest/candidate

consumer library should get ip addresses from app data rather than StoredState

the provider should be responsible for aggregating "public address" of all Alertmanager peers
should use application data rather than StoredState

`certificates-relation-changed` hook failed: creating secrets: Regular expression is invalid: nothing to repeat

Solutions QA has observed this issue in one of our runs when deploying COS-lite on microk8s 1.28/stable where the non-primary alertmanager unit stays in error state.

The unit's debug logs in the crashdump show the following error being repeated:

unit-alertmanager-1: 2024-02-17 00:16:54 DEBUG jujuc running hook tool "juju-log" for alertmanager/1-certificates-relation-changed-172755738486250708
unit-alertmanager-1: 2024-02-17 00:16:54 WARNING unit.alertmanager/1.juju-log certificates:30: cannot determine if reload succeeded
unit-alertmanager-1: 2024-02-17 00:16:54 DEBUG jujuc running hook tool "status-set" for alertmanager/1-certificates-relation-changed-172755738486250708
unit-alertmanager-1: 2024-02-17 00:16:54 DEBUG jujuc running hook tool "juju-log" for alertmanager/1-certificates-relation-changed-172755738486250708
unit-alertmanager-1: 2024-02-17 00:16:54 DEBUG unit.alertmanager/1.juju-log certificates:30: Emitting custom event <CertChanged via AlertmanagerCharm/CertHandler[am-server-cert]/on/cert_changed[270]>.
unit-alertmanager-1: 2024-02-17 00:16:54 DEBUG jujuc running hook tool "relation-set" for alertmanager/1-certificates-relation-changed-172755738486250708
unit-alertmanager-1: 2024-02-17 00:16:54 DEBUG jujuc running hook tool "relation-set" for alertmanager/1-certificates-relation-changed-172755738486250708
unit-alertmanager-1: 2024-02-17 00:16:54 DEBUG jujuc running hook tool "relation-set" for alertmanager/1-certificates-relation-changed-172755738486250708
unit-alertmanager-1: 2024-02-17 00:16:54 DEBUG jujuc running hook tool "relation-set" for alertmanager/1-certificates-relation-changed-172755738486250708
unit-alertmanager-1: 2024-02-17 00:16:54 DEBUG jujuc running hook tool "relation-set" for alertmanager/1-certificates-relation-changed-172755738486250708
unit-alertmanager-1: 2024-02-17 00:16:54 DEBUG jujuc running hook tool "relation-set" for alertmanager/1-certificates-relation-changed-172755738486250708
model-a0121c38-4204-408f-82f9-6b08d0a522c3: 2024-02-17 00:16:55 DEBUG juju.worker.muxhttpserver received tls client hello for server name modeloperator.cos.svc
model-a0121c38-4204-408f-82f9-6b08d0a522c3: 2024-02-17 00:16:55 DEBUG juju.worker.caasadmission received admission request for unit-alertmanager-1 of /v1, Kind=ServiceAccount in namespace cos
model-a0121c38-4204-408f-82f9-6b08d0a522c3: 2024-02-17 00:16:55 DEBUG juju.worker.caasadmission received admission request for unit-alertmanager-1 of /v1, Kind=ServiceAccount in namespace cos
model-a0121c38-4204-408f-82f9-6b08d0a522c3: 2024-02-17 00:16:55 DEBUG juju.worker.caasadmission received admission request for unit-alertmanager-1 of rbac.authorization.k8s.io/v1, Kind=Role in namespace cos
model-a0121c38-4204-408f-82f9-6b08d0a522c3: 2024-02-17 00:16:55 DEBUG juju.worker.caasadmission received admission request for unit-alertmanager-1 of rbac.authorization.k8s.io/v1, Kind=RoleBinding in namespace cos
unit-alertmanager-1: 2024-02-17 00:16:55 DEBUG juju.kubernetes.provider opening model "cos".
model-a0121c38-4204-408f-82f9-6b08d0a522c3: 2024-02-17 00:16:55 DEBUG juju.worker.muxhttpserver received tls client hello for server name modeloperator.cos.svc
model-a0121c38-4204-408f-82f9-6b08d0a522c3: 2024-02-17 00:16:55 DEBUG juju.worker.caasadmission received admission request for cn7vlr7mp25c7b4j91jg-1 of /v1, Kind=Secret in namespace cos
unit-alertmanager-1: 2024-02-17 00:16:55 ERROR juju.worker.uniter.context cannot apply changes: creating secrets: Regular expression is invalid: nothing to repeat
model-a0121c38-4204-408f-82f9-6b08d0a522c3: 2024-02-17 00:16:55 DEBUG juju.worker.caasadmission received admission request for unit-alertmanager-1 of /v1, Kind=ServiceAccount in namespace cos
model-a0121c38-4204-408f-82f9-6b08d0a522c3: 2024-02-17 00:16:55 DEBUG juju.worker.caasadmission received admission request for unit-alertmanager-1 of /v1, Kind=ServiceAccount in namespace cos
model-a0121c38-4204-408f-82f9-6b08d0a522c3: 2024-02-17 00:16:55 DEBUG juju.worker.caasadmission received admission request for unit-alertmanager-1 of rbac.authorization.k8s.io/v1, Kind=Role in namespace cos
model-a0121c38-4204-408f-82f9-6b08d0a522c3: 2024-02-17 00:16:55 DEBUG juju.worker.caasadmission received admission request for unit-alertmanager-1 of rbac.authorization.k8s.io/v1, Kind=RoleBinding in namespace cos
unit-alertmanager-1: 2024-02-17 00:16:55 DEBUG juju.kubernetes.provider opening model "cos".
unit-alertmanager-1: 2024-02-17 00:16:55 ERROR juju.worker.uniter.operation hook "certificates-relation-changed" (via hook dispatching script: dispatch) failed: creating secrets: Regular expression is invalid: nothing to repeat
unit-alertmanager-1: 2024-02-17 00:16:55 DEBUG juju.machinelock created rotating log file "/var/log/juju/machine-lock.log" with max size 10 MB and max backups 5
unit-alertmanager-1: 2024-02-17 00:16:55 DEBUG juju.machinelock machine lock "machine-lock" released for alertmanager/1 uniter (run relation-changed (30; app: ca) hook)
unit-alertmanager-1: 2024-02-17 00:16:55 DEBUG juju.worker.uniter.operation lock released for alertmanager/1
unit-alertmanager-1: 2024-02-17 00:16:55 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-alertmanager-1: 2024-02-17 00:16:55 DEBUG juju.worker.uniter [AGENT-STATUS] error: hook failed: "certificates-relation-changed"

Race in alertmanager configuration

Bug Description

There seems to be a race in the alertmanager_dispatch library that sometimes prevents Prometheus from sending alerts to alertmanager.

To Reproduce

juju deploy cos-lite --channel edge --trust
check the alertmanager ui
remove the prometheus alerting relation and add it back
check ui again

Environment

n/a

Relevant log output

n/a

Additional context

No response

Fix CodeQL scheduling

I wonder if we should run it also nightly on main.

Originally posted by @mmanciop in #39 (comment)

I think it might make sense to run this check on any PR, irrespective of the target branch (these are github workflows we do not current use, but still)

Originally posted by @mmanciop in #39 (comment)

deprecation of CodeQL Action v1

Enhancement Proposal

In December 2022 CodeQL Action v1 will be deprecated, so a enhancement will be use CodeQL Actions V2

More information here:

https://github.blog/changelog/2022-04-27-code-scanning-deprecation-of-codeql-action-v1/

Action `show-config` fails

Bug Description

After configuring the alertmanger config via juju config alertmanager config_file I try to look at the config using the show-config action but that fails, although I can check the config on the Alertmanger WebUI and it is there.

To Reproduce

juju deploy alertmanger --channel latest/edge
juju config alertmanager config_file="$(cat alertmanager.yaml)"
juju run-action alertmanager/0 show-config --wait

Environment

This is using alertmanager-k8s from latest/edge rev. 79

Model is using juju 3.1.5 but client is 2.9.43-ubuntu-amd64 given same juju client is used to manage models using juju 2.

Relevant log output

$ juju run-action alertmanager/0 show-config --wait                                                                                                                                             
unit-alertmanager-0:                                                                                                                                                                                                                          
  UnitId: alertmanager/0                                                                                                                                                                                                                      
  id: "22"                                                                                                                                                                                                                                    
  log:   
  - 2023-07-06 10:33:29 +0000 UTC Fetching /etc/alertmanager/alertmanager.yml                                                                                                                                                                 
  message: exit status 1                                                                                               
  results:             
    ReturnCode: 1                                          
  status: failed
  timing:
    completed: 2023-07-06 10:33:30 +0000 UTC
    enqueued: 2023-07-06 10:33:29 +0000 UTC
    started: 2023-07-06 10:33:29 +0000 UTC

Additional context

No response

Add self-monitoring of Alertmanager metrics

Enhancement Proposal

We should replicate in this charm something like canonical/prometheus-k8s-operator#312

Figure out how to configure cos-alerter

Enhancement Proposal

We need a way to configure the cos-alerter alert. Currently, this would be to add it to the config_file configuration option. This feels clunky. A config variable or integrator charm would be a much better user experience. Perhaps we need to discuss the way config is handled in Alertmanager in general.

configurable API port

Alertmanager has currently two port numbers hard-coded as a "static" class attributes: API port, 9093 and HA port, 9094.
It would be helpful for ingress to have the API port more visible/obvious.

#17 (comment)

Remove alertmangers stored state from Prometheus Charm

Overview

A recent commit to Alertmanager created a charm library for managing relations between Prometheus and Alertmanager. This library uses stored state to store the list of Alertmanagers that Prometheus must forward alerts to. However the older design used a stored state within the Prometheus Charm to store this list of Alertmanagers. Hence there are now two locations where this list is stored. Even though this is a change that must be done in the Prometheus respository, it is a change that must happen in conjunction with switching Alertmanager to the Alertmanager library, hence this issue is being reported here.

Use `on.available` to communicate changes in alertmanager cluster, instead of emitting a custom `ClusterChanged` event

I am still concerned that on a relation changed event this library can emit a ClusterChanged event while ConsumerBase emits a ProviderInvalid. If this does happen wouldn't this result in Prometheus adding an alertmanager into its config that is and "invalid" alertmanager ?

Originally posted by @balbirthomas in #7 (comment)

Use a rolling restart

Enhancement Proposal

The workload may be restarted by pebble on config-changed and peer-relation-joined.
This may lead to an entire cluster being down for some time.

Currently, the charm avoids restarting when possible. With the introduction of ingress (#76), tracking when a workload needs to restart would be more involved.

Might be useful to take advantage of "planned units" as well to reduce downtime.

Related: canonical/operator#530

Wrong Alert's source URL

Bug Description

Despite of the fact Alertmanager and Prometheus are behind an ingress, the source of the alert use k8s-based URL

To Reproduce

Deploy cos-lite bundle
In Alertmanager UI check for the Source of the alert.

Environment

juju 3.1.6
cos-lite bundle

Relevant log output

Additional context

No response

Alert manager is constantly restarting

Bug Description

Alertmanager is constantly restarting and it is not able to resolve alerts in PD because of that.

To Reproduce

juju deploy cos-lite

Environment

alertmanger rev. 96

Relevant log output

$ juju ssh --container alertmanager alertmanager/0 /charm/bin/pebble changes | tail -n 20
6803  Done    today at 10:10 UTC      today at 10:10 UTC      Restart service "alertmanager"
6804  Done    today at 10:14 UTC      today at 10:14 UTC      Execute command "/usr/bin/amtool"
6805  Done    today at 10:14 UTC      today at 10:14 UTC      Replan service "alertmanager"
6806  Done    today at 10:14 UTC      today at 10:14 UTC      Restart service "alertmanager"
6807  Done    today at 10:20 UTC      today at 10:20 UTC      Execute command "/usr/bin/amtool"
6808  Done    today at 10:20 UTC      today at 10:20 UTC      Replan service "alertmanager"
6809  Done    today at 10:20 UTC      today at 10:20 UTC      Restart service "alertmanager"
6810  Done    today at 10:25 UTC      today at 10:25 UTC      Execute command "/usr/bin/amtool"
6811  Done    today at 10:25 UTC      today at 10:25 UTC      Replan service "alertmanager"
6812  Done    today at 10:25 UTC      today at 10:25 UTC      Restart service "alertmanager"
6813  Done    today at 10:29 UTC      today at 10:29 UTC      Execute command "/usr/bin/amtool"
6814  Done    today at 10:29 UTC      today at 10:29 UTC      Replan service "alertmanager"
6815  Done    today at 10:29 UTC      today at 10:29 UTC      Restart service "alertmanager"
6816  Done    today at 10:35 UTC      today at 10:35 UTC      Execute command "/usr/bin/amtool"
6817  Done    today at 10:35 UTC      today at 10:35 UTC      Replan service "alertmanager"
6818  Done    today at 10:35 UTC      today at 10:35 UTC      Restart service "alertmanager"
6819  Done    today at 10:41 UTC      today at 10:41 UTC      Execute command "/usr/bin/amtool"
6820  Done    today at 10:41 UTC      today at 10:41 UTC      Replan service "alertmanager"
6821  Done    today at 10:41 UTC      today at 10:41 UTC      Restart service "alertmanager"


$ kubectl logs -n cos alertmanager-0 -c charm
2023-12-13T09:12:26.903Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.remotestate watcher.go:809 update status timer triggered for alertmanager/0
2023-12-13T09:12:26.904Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.storage resolver.go:173 next hook op for storage-data-0: {Kind:2 Life:alive Attached:true Location:/var/lib/juju/storage/data/0}
2023-12-13T09:12:26.904Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter resolver.go:194 no operations in progress; waiting for changes
2023-12-13T09:12:26.905Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.relation resolver.go:285 unit "traefik/0" already joined relation 10
2023-12-13T09:12:26.905Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.relation resolver.go:285 unit "loki/0" already joined relation 15
2023-12-13T09:12:26.905Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.relation resolver.go:285 unit "catalogue/0" already joined relation 26
2023-12-13T09:12:26.906Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.relation resolver.go:285 unit "prometheus/0" already joined relation 11
2023-12-13T09:12:26.906Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.relation resolver.go:285 unit "grafana/0" already joined relation 22
2023-12-13T09:12:26.906Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.relation resolver.go:285 unit "grafana/0" already joined relation 14
2023-12-13T09:12:26.907Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.relation resolver.go:285 unit "ca/0" already joined relation 29
2023-12-13T09:12:26.907Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.relation resolver.go:285 unit "prometheus/0" already joined relation 17
2023-12-13T09:12:26.907Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.operation executor.go:85 running operation run update-status hook for alertmanager/0
2023-12-13T09:12:26.908Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.operation executor.go:88 acquiring machine lock for alertmanager/0
2023-12-13T09:12:26.908Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.machinelock machinelock.go:166 acquire machine lock "machine-lock" for alertmanager/0 uniter (run update-status hook)
2023-12-13T09:12:26.909Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.machinelock machinelock.go:176 machine lock "machine-lock" acquired for alertmanager/0 uniter (run update-status hook)
2023-12-13T09:12:26.909Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.operation executor.go:135 preparing operation "run update-status hook" for alertmanager/0
2023-12-13T09:12:26.952Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.operation executor.go:135 executing operation "run update-status hook" for alertmanager/0
2023-12-13T09:12:26.953Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.runner runner.go:719 starting jujuc server  {unix @/var/lib/juju/agents/unit-alertmanager-0/agent.socket <nil>}
2023-12-13T09:12:27.834Z [container-agent] 2023-12-13 09:12:27 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:27.834Z [container-agent] 2023-12-13 09:12:27 DEBUG juju-log ops 2.4.1 up and running.
2023-12-13T09:12:27.863Z [container-agent] 2023-12-13 09:12:27 DEBUG jujuc server.go:222 running hook tool "relation-ids" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:27.871Z [container-agent] 2023-12-13 09:12:27 DEBUG jujuc server.go:222 running hook tool "relation-list" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:27.880Z [container-agent] 2023-12-13 09:12:27 DEBUG jujuc server.go:222 running hook tool "relation-get" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:27.901Z [container-agent] 2023-12-13 09:12:27 DEBUG jujuc server.go:222 running hook tool "opened-ports" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:27.918Z [container-agent] 2023-12-13 09:12:27 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:27.918Z [container-agent] 2023-12-13 09:12:27 DEBUG juju-log load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False
2023-12-13T09:12:27.962Z [container-agent] 2023-12-13 09:12:27 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:27.962Z [container-agent] 2023-12-13 09:12:27 DEBUG juju-log load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt'
2023-12-13T09:12:27.987Z [container-agent] 2023-12-13 09:12:27 DEBUG jujuc server.go:222 running hook tool "relation-ids" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:27.999Z [container-agent] 2023-12-13 09:12:27 DEBUG jujuc server.go:222 running hook tool "relation-list" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.016Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.017Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log Emitting Juju event update_status.
2023-12-13T09:12:28.029Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-ids" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.053Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-list" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.067Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "network-get" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.093Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.094Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.107Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.108Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.118Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.119Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.127Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "is-leader" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.142Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.142Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log Could not locate cos-tool at: "cos-tool-amd64"
2023-12-13T09:12:28.151Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.151Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log Skipping injection of juju topology as label matchers
2023-12-13T09:12:28.159Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.160Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log `cos-tool` unavailable. Leaving expression unchanged: vector(1)
2023-12-13T09:12:28.169Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.169Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log Reading <property object at 0x7f746155f770> rule from /var/lib/juju/agents/unit-alertmanager-0/charm/src/prometheus_alert_rules/heartbeat.rule
2023-12-13T09:12:28.181Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.181Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log `cos-tool` unavailable. Leaving expression unchanged: alertmanager_config_last_reload_successful{} != 1
2023-12-13T09:12:28.190Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.190Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log Reading <property object at 0x7f746155f770> rule from /var/lib/juju/agents/unit-alertmanager-0/charm/src/prometheus_alert_rules/alertmanager_configuration_reload_failure.rule
2023-12-13T09:12:28.203Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.203Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log `cos-tool` unavailable. Leaving expression unchanged: absent(up{})
2023-12-13T09:12:28.212Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.212Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log Reading <property object at 0x7f746155f770> rule from /var/lib/juju/agents/unit-alertmanager-0/charm/src/prometheus_alert_rules/alertmanager_missing.rule
2023-12-13T09:12:28.227Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.227Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log `cos-tool` unavailable. Leaving expression unchanged: rate(alertmanager_notifications_failed_total{integration=~".*"}[5m]) > 0
2023-12-13T09:12:28.238Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.238Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log Reading <property object at 0x7f746155f770> rule from /var/lib/juju/agents/unit-alertmanager-0/charm/src/prometheus_alert_rules/alertmanager_notifications_failed.rule
2023-12-13T09:12:28.251Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.252Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.267Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.269Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.286Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.287Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.731Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.732Z [container-agent] 2023-12-13 09:12:28 ERROR juju-log Failed to obtain status: Bad response
2023-12-13T09:12:28.743Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "config-get" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.770Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.770Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log connect_tcp.started host='10.152.183.1' port=443 local_address=None timeout=None socket_options=None
2023-12-13T09:12:28.784Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.784Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7f7461490520>
2023-12-13T09:12:28.793Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.793Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log start_tls.started ssl_context=<ssl.SSLContext object at 0x7f74614f9940> server_hostname='10.152.183.1' timeout=None
2023-12-13T09:12:28.807Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.807Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7f7461490430>
2023-12-13T09:12:28.819Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.819Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log send_request_headers.started request=<Request [b'GET']>
2023-12-13T09:12:28.833Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.833Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log send_request_headers.complete
2023-12-13T09:12:28.841Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.842Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log send_request_body.started request=<Request [b'GET']>
2023-12-13T09:12:28.850Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.850Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log send_request_body.complete
2023-12-13T09:12:28.858Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.858Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log receive_response_headers.started request=<Request [b'GET']>
2023-12-13T09:12:28.870Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.871Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'faa60816-945e-4fdd-bd6a-ce9ad33c11bb'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'8cbe845f-d38d-4ac4-be06-8b835ac55ddd'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'bf3d64fb-a3b2-41a9-a04e-361c2673730b'), (b'Date', b'Wed, 13 Dec 2023 09:12:28 GMT'), (b'Transfer-Encoding', b'chunked')])
2023-12-13T09:12:28.882Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.882Z [container-agent] 2023-12-13 09:12:28 INFO juju-log HTTP Request: GET https://10.152.183.1/apis/apps/v1/namespaces/cos/statefulsets/alertmanager "HTTP/1.1 200 OK"
2023-12-13T09:12:28.891Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.891Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log receive_response_body.started request=<Request [b'GET']>
2023-12-13T09:12:28.901Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.902Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log receive_response_body.complete
2023-12-13T09:12:28.910Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.910Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log response_closed.started
2023-12-13T09:12:28.924Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.924Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log response_closed.complete
2023-12-13T09:12:28.948Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.948Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log send_request_headers.started request=<Request [b'GET']>
2023-12-13T09:12:28.961Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.961Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log send_request_headers.complete
2023-12-13T09:12:28.972Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.972Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log send_request_body.started request=<Request [b'GET']>
2023-12-13T09:12:28.982Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.982Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log send_request_body.complete
2023-12-13T09:12:28.991Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.991Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log receive_response_headers.started request=<Request [b'GET']>
2023-12-13T09:12:29.003Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.003Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'be4bdc66-3864-420f-9864-62d034999bb4'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'8cbe845f-d38d-4ac4-be06-8b835ac55ddd'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'bf3d64fb-a3b2-41a9-a04e-361c2673730b'), (b'Date', b'Wed, 13 Dec 2023 09:12:28 GMT'), (b'Transfer-Encoding', b'chunked')])
2023-12-13T09:12:29.019Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.019Z [container-agent] 2023-12-13 09:12:29 INFO juju-log HTTP Request: GET https://10.152.183.1/api/v1/namespaces/cos/pods/alertmanager-0 "HTTP/1.1 200 OK"
2023-12-13T09:12:29.028Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.028Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log receive_response_body.started request=<Request [b'GET']>
2023-12-13T09:12:29.038Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.039Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log receive_response_body.complete
2023-12-13T09:12:29.048Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.048Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log response_closed.started
2023-12-13T09:12:29.057Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.057Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log response_closed.complete
2023-12-13T09:12:29.071Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.071Z [container-agent] 2023-12-13 09:12:29 INFO juju-log reqs=ResourceRequirements(claims=None, limits={}, requests={'cpu': '0.25', 'memory': '200Mi'}), templated=ResourceRequirements(claims=None, limits=None, requests={'cpu': '250m', 'memory': '200Mi'}), actual=ResourceRequirements(claims=None, limits=None, requests={'cpu': '250m', 'memory': '200Mi'})
2023-12-13T09:12:29.084Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.084Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log send_request_headers.started request=<Request [b'GET']>
2023-12-13T09:12:29.094Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.094Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log send_request_headers.complete
2023-12-13T09:12:29.102Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.102Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log send_request_body.started request=<Request [b'GET']>
2023-12-13T09:12:29.111Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.111Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log send_request_body.complete
2023-12-13T09:12:29.120Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.120Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log receive_response_headers.started request=<Request [b'GET']>
2023-12-13T09:12:29.129Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.129Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'1bf99a1c-4f9b-4a2e-b8d5-26728041cbae'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'8cbe845f-d38d-4ac4-be06-8b835ac55ddd'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'bf3d64fb-a3b2-41a9-a04e-361c2673730b'), (b'Date', b'Wed, 13 Dec 2023 09:12:29 GMT'), (b'Transfer-Encoding', b'chunked')])
2023-12-13T09:12:29.140Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.140Z [container-agent] 2023-12-13 09:12:29 INFO juju-log HTTP Request: GET https://10.152.183.1/apis/apps/v1/namespaces/cos/statefulsets/alertmanager "HTTP/1.1 200 OK"
2023-12-13T09:12:29.149Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.149Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log receive_response_body.started request=<Request [b'GET']>
2023-12-13T09:12:29.159Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.160Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log receive_response_body.complete
2023-12-13T09:12:29.172Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.172Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log response_closed.started
2023-12-13T09:12:29.181Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.181Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log response_closed.complete
2023-12-13T09:12:29.195Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.195Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log send_request_headers.started request=<Request [b'GET']>
2023-12-13T09:12:29.205Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.205Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log send_request_headers.complete
2023-12-13T09:12:29.214Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.214Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log send_request_body.started request=<Request [b'GET']>
2023-12-13T09:12:29.224Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.224Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log send_request_body.complete
2023-12-13T09:12:29.232Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.232Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log receive_response_headers.started request=<Request [b'GET']>
2023-12-13T09:12:29.242Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.242Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'ebbb68ab-0969-4a10-80e0-682610aa7a4b'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'8cbe845f-d38d-4ac4-be06-8b835ac55ddd'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'bf3d64fb-a3b2-41a9-a04e-361c2673730b'), (b'Date', b'Wed, 13 Dec 2023 09:12:29 GMT'), (b'Transfer-Encoding', b'chunked')])
2023-12-13T09:12:29.251Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.251Z [container-agent] 2023-12-13 09:12:29 INFO juju-log HTTP Request: GET https://10.152.183.1/api/v1/namespaces/cos/pods/alertmanager-0 "HTTP/1.1 200 OK"
2023-12-13T09:12:29.261Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.261Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log receive_response_body.started request=<Request [b'GET']>
2023-12-13T09:12:29.270Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.270Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log receive_response_body.complete
2023-12-13T09:12:29.282Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.282Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log response_closed.started
2023-12-13T09:12:29.291Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.291Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log response_closed.complete
2023-12-13T09:12:29.311Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-ids" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.322Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-list" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.333Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-list" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.342Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.344Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.356Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.357Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.367Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.370Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.380Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.381Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.394Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.394Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.408Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.409Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.422Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-get" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.435Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.437Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.457Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.457Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.466Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.468Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.476Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.477Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.486Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.487Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.499Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.500Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.508Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.509Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.518Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.519Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.527Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.528Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.537Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.538Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.549Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.550Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.560Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.561Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.574Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-get" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.588Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.589Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.599Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.599Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.623Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.624Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.639Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.639Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.648Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.649Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.664Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.665Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log Could not locate cos-tool at: "cos-tool-amd64"
2023-12-13T09:12:29.675Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.675Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log Skipping injection of juju topology as label matchers
2023-12-13T09:12:29.684Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.684Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log `cos-tool` unavailable. Leaving expression unchanged: vector(1)
2023-12-13T09:12:29.693Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.694Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log Reading <property object at 0x7f746155f770> rule from /var/lib/juju/agents/unit-alertmanager-0/charm/src/prometheus_alert_rules/heartbeat.rule
2023-12-13T09:12:29.707Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.707Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log `cos-tool` unavailable. Leaving expression unchanged: alertmanager_config_last_reload_successful{} != 1
2023-12-13T09:12:29.717Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.717Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log Reading <property object at 0x7f746155f770> rule from /var/lib/juju/agents/unit-alertmanager-0/charm/src/prometheus_alert_rules/alertmanager_configuration_reload_failure.rule
2023-12-13T09:12:29.730Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.730Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log `cos-tool` unavailable. Leaving expression unchanged: absent(up{})
2023-12-13T09:12:29.739Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.739Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log Reading <property object at 0x7f746155f770> rule from /var/lib/juju/agents/unit-alertmanager-0/charm/src/prometheus_alert_rules/alertmanager_missing.rule
2023-12-13T09:12:29.752Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.752Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log `cos-tool` unavailable. Leaving expression unchanged: rate(alertmanager_notifications_failed_total{integration=~".*"}[5m]) > 0
2023-12-13T09:12:29.760Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.761Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log Reading <property object at 0x7f746155f770> rule from /var/lib/juju/agents/unit-alertmanager-0/charm/src/prometheus_alert_rules/alertmanager_notifications_failed.rule
2023-12-13T09:12:29.774Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.775Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.784Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.784Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.795Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.796Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.818Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.818Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.836Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-ids" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.846Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-ids" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.905Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-get" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.916Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.916Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log applying config changes
2023-12-13T09:12:31.136Z [container-agent] 2023-12-13 09:12:31 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:31.136Z [container-agent] 2023-12-13 09:12:31 WARNING juju-log config reload via HTTP POST failed: Bad response
2023-12-13T09:12:31.149Z [container-agent] 2023-12-13 09:12:31 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:31.149Z [container-agent] 2023-12-13 09:12:31 INFO juju-log Restarting service alertmanager
2023-12-13T09:12:32.678Z [container-agent] 2023-12-13 09:12:32 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:32.678Z [container-agent] 2023-12-13 09:12:32 WARNING juju-log cannot determine if reload succeeded
2023-12-13T09:12:32.691Z [container-agent] 2023-12-13 09:12:32 DEBUG jujuc server.go:222 running hook tool "relation-ids" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:32.702Z [container-agent] 2023-12-13 09:12:32 DEBUG jujuc server.go:222 running hook tool "relation-list" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:32.724Z [container-agent] 2023-12-13 09:12:32 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:32.726Z [container-agent] 2023-12-13 09:12:32 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:32.748Z [container-agent] 2023-12-13 09:12:32 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:32.748Z [container-agent] 2023-12-13 09:12:32 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:32.759Z [container-agent] 2023-12-13 09:12:32 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:32.762Z [container-agent] 2023-12-13 09:12:32 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:32.779Z [container-agent] 2023-12-13 09:12:32 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:32.779Z [container-agent] 2023-12-13 09:12:32 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:32.786Z [container-agent] 2023-12-13 09:12:32 DEBUG jujuc server.go:222 running hook tool "status-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:33.143Z [container-agent] 2023-12-13 09:12:33 INFO juju.worker.uniter.operation runhook.go:186 ran "update-status" hook (via hook dispatching script: dispatch)
2023-12-13T09:12:33.162Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.worker.uniter.operation executor.go:135 committing operation "run update-status hook" for alertmanager/0
2023-12-13T09:12:33.171Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.machinelock machinelock.go:206 created rotating log file "/var/log/machine-lock.log" with max size 10 MB and max backups 5
2023-12-13T09:12:33.172Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.machinelock machinelock.go:190 machine lock "machine-lock" released for alertmanager/0 uniter (run update-status hook)
2023-12-13T09:12:33.172Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.worker.uniter.operation executor.go:124 lock released for alertmanager/0
2023-12-13T09:12:33.172Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.worker.uniter.storage resolver.go:173 next hook op for storage-data-0: {Kind:2 Life:alive Attached:true Location:/var/lib/juju/storage/data/0}
2023-12-13T09:12:33.172Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.worker.uniter resolver.go:194 no operations in progress; waiting for changes
2023-12-13T09:12:33.172Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.worker.uniter.relation resolver.go:285 unit "ca/0" already joined relation 29
2023-12-13T09:12:33.172Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.worker.uniter.relation resolver.go:285 unit "traefik/0" already joined relation 10
2023-12-13T09:12:33.172Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.worker.uniter.relation resolver.go:285 unit "catalogue/0" already joined relation 26
2023-12-13T09:12:33.172Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.worker.uniter.relation resolver.go:285 unit "grafana/0" already joined relation 14
2023-12-13T09:12:33.172Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.worker.uniter.relation resolver.go:285 unit "prometheus/0" already joined relation 17
2023-12-13T09:12:33.172Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.worker.uniter.relation resolver.go:285 unit "prometheus/0" already joined relation 11
2023-12-13T09:12:33.173Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.worker.uniter.relation resolver.go:285 unit "loki/0" already joined relation 15

Additional context

No response

Discrepancy between alerts firing in prometheus and those reported in alertmanager

Bug Description

In a fresh deployment of an environment, we have a COS with Charmed K8s and Charmed Kubeflow.

Currently we see 13 alerts firing in prometheus, however only 4 show up in alertmanager.

To Reproduce

N/A

Environment

This is with Charmed K8s 1.29/beta and Charmed Kubeflow 1.8/stable on top of Juju 3.1.6

App                       Version  Status  Scale  Charm                         Channel        Rev  Address         Exposed  Message
alertmanager              0.25.0   active      1  alertmanager-k8s              stable          96  REMOVED   no       
catalogue                          active      1  catalogue-k8s                 stable          31  REMOVED   no       
grafana                   9.2.1    active      1  grafana-k8s                   stable          93  REMOVED  no       
loki                      2.7.4    active      1  loki-k8s                      stable         105  REMOVED  no       
prometheus                2.47.2   active      1  prometheus-k8s                stable         156  REMOVED   no       
prometheus-scrape-config  n/a      active      1  prometheus-scrape-config-k8s  latest/stable   44  REMOVED  no       
traefik                   2.10.4   active      1  traefik-k8s                   stable         166  REMOVED  no

Relevant log output

N/A

Additional context

No response

Using python 3.10 instead on 3.5

Enhancement Proposal

On the workflow release to edge i see this name: Set up Python 3.5 why use python 3.5 and not python 3.10 need to update ?

Grouping of alerts can not be disabled

Bug Description

Grouping of alerts can not be disabled.

To Reproduce

Put this into the config_file:

route:
  group_by:
  - ...

This should disable grouping entirely, according to the documentation.

When rolled out, the unit goes into blocked state and amtool reports:

root@alertmanager-0:/# amtool check-config /etc/alertmanager/alertmanager.yml 
Checking '/etc/alertmanager/alertmanager.yml'  FAILED: cannot have wildcard group_by (`...`) and other other labels at the same time

The config itself has:

route:
  group_by:
  - juju_model_uuid
  - juju_application
  - '...'
  - juju_model

Environment

I tried this locally, as well as on customer clouds running latest/stable.

Relevant log output

Checking '/etc/alertmanager/alertmanager.yml'  FAILED: cannot have wildcard group_by (`...`) and other other labels at the same time

Additional context

No response

Clear stale data

Need to think about this from a "juju model" POV. Rel-data, config options represent the model. Is it semantically correct to have stale data in this case? Not sure.

Wouldn't a new join generate the relation cascade?

Originally posted by @sed-i in #95 (comment)

See the thread for more details about what was discussed.