Alertmanager is constantly restarting and it is not able to resolve alerts in PD because of that.
alertmanger rev. 96
$ juju ssh --container alertmanager alertmanager/0 /charm/bin/pebble changes | tail -n 20
6803 Done today at 10:10 UTC today at 10:10 UTC Restart service "alertmanager"
6804 Done today at 10:14 UTC today at 10:14 UTC Execute command "/usr/bin/amtool"
6805 Done today at 10:14 UTC today at 10:14 UTC Replan service "alertmanager"
6806 Done today at 10:14 UTC today at 10:14 UTC Restart service "alertmanager"
6807 Done today at 10:20 UTC today at 10:20 UTC Execute command "/usr/bin/amtool"
6808 Done today at 10:20 UTC today at 10:20 UTC Replan service "alertmanager"
6809 Done today at 10:20 UTC today at 10:20 UTC Restart service "alertmanager"
6810 Done today at 10:25 UTC today at 10:25 UTC Execute command "/usr/bin/amtool"
6811 Done today at 10:25 UTC today at 10:25 UTC Replan service "alertmanager"
6812 Done today at 10:25 UTC today at 10:25 UTC Restart service "alertmanager"
6813 Done today at 10:29 UTC today at 10:29 UTC Execute command "/usr/bin/amtool"
6814 Done today at 10:29 UTC today at 10:29 UTC Replan service "alertmanager"
6815 Done today at 10:29 UTC today at 10:29 UTC Restart service "alertmanager"
6816 Done today at 10:35 UTC today at 10:35 UTC Execute command "/usr/bin/amtool"
6817 Done today at 10:35 UTC today at 10:35 UTC Replan service "alertmanager"
6818 Done today at 10:35 UTC today at 10:35 UTC Restart service "alertmanager"
6819 Done today at 10:41 UTC today at 10:41 UTC Execute command "/usr/bin/amtool"
6820 Done today at 10:41 UTC today at 10:41 UTC Replan service "alertmanager"
6821 Done today at 10:41 UTC today at 10:41 UTC Restart service "alertmanager"
$ kubectl logs -n cos alertmanager-0 -c charm
2023-12-13T09:12:26.903Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.remotestate watcher.go:809 update status timer triggered for alertmanager/0
2023-12-13T09:12:26.904Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.storage resolver.go:173 next hook op for storage-data-0: {Kind:2 Life:alive Attached:true Location:/var/lib/juju/storage/data/0}
2023-12-13T09:12:26.904Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter resolver.go:194 no operations in progress; waiting for changes
2023-12-13T09:12:26.905Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.relation resolver.go:285 unit "traefik/0" already joined relation 10
2023-12-13T09:12:26.905Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.relation resolver.go:285 unit "loki/0" already joined relation 15
2023-12-13T09:12:26.905Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.relation resolver.go:285 unit "catalogue/0" already joined relation 26
2023-12-13T09:12:26.906Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.relation resolver.go:285 unit "prometheus/0" already joined relation 11
2023-12-13T09:12:26.906Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.relation resolver.go:285 unit "grafana/0" already joined relation 22
2023-12-13T09:12:26.906Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.relation resolver.go:285 unit "grafana/0" already joined relation 14
2023-12-13T09:12:26.907Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.relation resolver.go:285 unit "ca/0" already joined relation 29
2023-12-13T09:12:26.907Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.relation resolver.go:285 unit "prometheus/0" already joined relation 17
2023-12-13T09:12:26.907Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.operation executor.go:85 running operation run update-status hook for alertmanager/0
2023-12-13T09:12:26.908Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.operation executor.go:88 acquiring machine lock for alertmanager/0
2023-12-13T09:12:26.908Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.machinelock machinelock.go:166 acquire machine lock "machine-lock" for alertmanager/0 uniter (run update-status hook)
2023-12-13T09:12:26.909Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.machinelock machinelock.go:176 machine lock "machine-lock" acquired for alertmanager/0 uniter (run update-status hook)
2023-12-13T09:12:26.909Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.operation executor.go:135 preparing operation "run update-status hook" for alertmanager/0
2023-12-13T09:12:26.952Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.operation executor.go:135 executing operation "run update-status hook" for alertmanager/0
2023-12-13T09:12:26.953Z [container-agent] 2023-12-13 09:12:26 DEBUG juju.worker.uniter.runner runner.go:719 starting jujuc server {unix @/var/lib/juju/agents/unit-alertmanager-0/agent.socket <nil>}
2023-12-13T09:12:27.834Z [container-agent] 2023-12-13 09:12:27 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:27.834Z [container-agent] 2023-12-13 09:12:27 DEBUG juju-log ops 2.4.1 up and running.
2023-12-13T09:12:27.863Z [container-agent] 2023-12-13 09:12:27 DEBUG jujuc server.go:222 running hook tool "relation-ids" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:27.871Z [container-agent] 2023-12-13 09:12:27 DEBUG jujuc server.go:222 running hook tool "relation-list" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:27.880Z [container-agent] 2023-12-13 09:12:27 DEBUG jujuc server.go:222 running hook tool "relation-get" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:27.901Z [container-agent] 2023-12-13 09:12:27 DEBUG jujuc server.go:222 running hook tool "opened-ports" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:27.918Z [container-agent] 2023-12-13 09:12:27 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:27.918Z [container-agent] 2023-12-13 09:12:27 DEBUG juju-log load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False
2023-12-13T09:12:27.962Z [container-agent] 2023-12-13 09:12:27 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:27.962Z [container-agent] 2023-12-13 09:12:27 DEBUG juju-log load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt'
2023-12-13T09:12:27.987Z [container-agent] 2023-12-13 09:12:27 DEBUG jujuc server.go:222 running hook tool "relation-ids" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:27.999Z [container-agent] 2023-12-13 09:12:27 DEBUG jujuc server.go:222 running hook tool "relation-list" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.016Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.017Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log Emitting Juju event update_status.
2023-12-13T09:12:28.029Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-ids" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.053Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-list" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.067Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "network-get" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.093Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.094Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.107Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.108Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.118Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.119Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.127Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "is-leader" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.142Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.142Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log Could not locate cos-tool at: "cos-tool-amd64"
2023-12-13T09:12:28.151Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.151Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log Skipping injection of juju topology as label matchers
2023-12-13T09:12:28.159Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.160Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log `cos-tool` unavailable. Leaving expression unchanged: vector(1)
2023-12-13T09:12:28.169Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.169Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log Reading <property object at 0x7f746155f770> rule from /var/lib/juju/agents/unit-alertmanager-0/charm/src/prometheus_alert_rules/heartbeat.rule
2023-12-13T09:12:28.181Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.181Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log `cos-tool` unavailable. Leaving expression unchanged: alertmanager_config_last_reload_successful{} != 1
2023-12-13T09:12:28.190Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.190Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log Reading <property object at 0x7f746155f770> rule from /var/lib/juju/agents/unit-alertmanager-0/charm/src/prometheus_alert_rules/alertmanager_configuration_reload_failure.rule
2023-12-13T09:12:28.203Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.203Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log `cos-tool` unavailable. Leaving expression unchanged: absent(up{})
2023-12-13T09:12:28.212Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.212Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log Reading <property object at 0x7f746155f770> rule from /var/lib/juju/agents/unit-alertmanager-0/charm/src/prometheus_alert_rules/alertmanager_missing.rule
2023-12-13T09:12:28.227Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.227Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log `cos-tool` unavailable. Leaving expression unchanged: rate(alertmanager_notifications_failed_total{integration=~".*"}[5m]) > 0
2023-12-13T09:12:28.238Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.238Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log Reading <property object at 0x7f746155f770> rule from /var/lib/juju/agents/unit-alertmanager-0/charm/src/prometheus_alert_rules/alertmanager_notifications_failed.rule
2023-12-13T09:12:28.251Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.252Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.267Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.269Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.286Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.287Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.731Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.732Z [container-agent] 2023-12-13 09:12:28 ERROR juju-log Failed to obtain status: Bad response
2023-12-13T09:12:28.743Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "config-get" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.770Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.770Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log connect_tcp.started host='10.152.183.1' port=443 local_address=None timeout=None socket_options=None
2023-12-13T09:12:28.784Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.784Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7f7461490520>
2023-12-13T09:12:28.793Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.793Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log start_tls.started ssl_context=<ssl.SSLContext object at 0x7f74614f9940> server_hostname='10.152.183.1' timeout=None
2023-12-13T09:12:28.807Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.807Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7f7461490430>
2023-12-13T09:12:28.819Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.819Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log send_request_headers.started request=<Request [b'GET']>
2023-12-13T09:12:28.833Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.833Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log send_request_headers.complete
2023-12-13T09:12:28.841Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.842Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log send_request_body.started request=<Request [b'GET']>
2023-12-13T09:12:28.850Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.850Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log send_request_body.complete
2023-12-13T09:12:28.858Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.858Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log receive_response_headers.started request=<Request [b'GET']>
2023-12-13T09:12:28.870Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.871Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'faa60816-945e-4fdd-bd6a-ce9ad33c11bb'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'8cbe845f-d38d-4ac4-be06-8b835ac55ddd'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'bf3d64fb-a3b2-41a9-a04e-361c2673730b'), (b'Date', b'Wed, 13 Dec 2023 09:12:28 GMT'), (b'Transfer-Encoding', b'chunked')])
2023-12-13T09:12:28.882Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.882Z [container-agent] 2023-12-13 09:12:28 INFO juju-log HTTP Request: GET https://10.152.183.1/apis/apps/v1/namespaces/cos/statefulsets/alertmanager "HTTP/1.1 200 OK"
2023-12-13T09:12:28.891Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.891Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log receive_response_body.started request=<Request [b'GET']>
2023-12-13T09:12:28.901Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.902Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log receive_response_body.complete
2023-12-13T09:12:28.910Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.910Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log response_closed.started
2023-12-13T09:12:28.924Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.924Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log response_closed.complete
2023-12-13T09:12:28.948Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.948Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log send_request_headers.started request=<Request [b'GET']>
2023-12-13T09:12:28.961Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.961Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log send_request_headers.complete
2023-12-13T09:12:28.972Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.972Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log send_request_body.started request=<Request [b'GET']>
2023-12-13T09:12:28.982Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.982Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log send_request_body.complete
2023-12-13T09:12:28.991Z [container-agent] 2023-12-13 09:12:28 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:28.991Z [container-agent] 2023-12-13 09:12:28 DEBUG juju-log receive_response_headers.started request=<Request [b'GET']>
2023-12-13T09:12:29.003Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.003Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'be4bdc66-3864-420f-9864-62d034999bb4'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'8cbe845f-d38d-4ac4-be06-8b835ac55ddd'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'bf3d64fb-a3b2-41a9-a04e-361c2673730b'), (b'Date', b'Wed, 13 Dec 2023 09:12:28 GMT'), (b'Transfer-Encoding', b'chunked')])
2023-12-13T09:12:29.019Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.019Z [container-agent] 2023-12-13 09:12:29 INFO juju-log HTTP Request: GET https://10.152.183.1/api/v1/namespaces/cos/pods/alertmanager-0 "HTTP/1.1 200 OK"
2023-12-13T09:12:29.028Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.028Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log receive_response_body.started request=<Request [b'GET']>
2023-12-13T09:12:29.038Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.039Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log receive_response_body.complete
2023-12-13T09:12:29.048Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.048Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log response_closed.started
2023-12-13T09:12:29.057Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.057Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log response_closed.complete
2023-12-13T09:12:29.071Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.071Z [container-agent] 2023-12-13 09:12:29 INFO juju-log reqs=ResourceRequirements(claims=None, limits={}, requests={'cpu': '0.25', 'memory': '200Mi'}), templated=ResourceRequirements(claims=None, limits=None, requests={'cpu': '250m', 'memory': '200Mi'}), actual=ResourceRequirements(claims=None, limits=None, requests={'cpu': '250m', 'memory': '200Mi'})
2023-12-13T09:12:29.084Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.084Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log send_request_headers.started request=<Request [b'GET']>
2023-12-13T09:12:29.094Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.094Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log send_request_headers.complete
2023-12-13T09:12:29.102Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.102Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log send_request_body.started request=<Request [b'GET']>
2023-12-13T09:12:29.111Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.111Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log send_request_body.complete
2023-12-13T09:12:29.120Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.120Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log receive_response_headers.started request=<Request [b'GET']>
2023-12-13T09:12:29.129Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.129Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'1bf99a1c-4f9b-4a2e-b8d5-26728041cbae'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'8cbe845f-d38d-4ac4-be06-8b835ac55ddd'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'bf3d64fb-a3b2-41a9-a04e-361c2673730b'), (b'Date', b'Wed, 13 Dec 2023 09:12:29 GMT'), (b'Transfer-Encoding', b'chunked')])
2023-12-13T09:12:29.140Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.140Z [container-agent] 2023-12-13 09:12:29 INFO juju-log HTTP Request: GET https://10.152.183.1/apis/apps/v1/namespaces/cos/statefulsets/alertmanager "HTTP/1.1 200 OK"
2023-12-13T09:12:29.149Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.149Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log receive_response_body.started request=<Request [b'GET']>
2023-12-13T09:12:29.159Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.160Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log receive_response_body.complete
2023-12-13T09:12:29.172Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.172Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log response_closed.started
2023-12-13T09:12:29.181Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.181Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log response_closed.complete
2023-12-13T09:12:29.195Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.195Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log send_request_headers.started request=<Request [b'GET']>
2023-12-13T09:12:29.205Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.205Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log send_request_headers.complete
2023-12-13T09:12:29.214Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.214Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log send_request_body.started request=<Request [b'GET']>
2023-12-13T09:12:29.224Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.224Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log send_request_body.complete
2023-12-13T09:12:29.232Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.232Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log receive_response_headers.started request=<Request [b'GET']>
2023-12-13T09:12:29.242Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.242Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'ebbb68ab-0969-4a10-80e0-682610aa7a4b'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'8cbe845f-d38d-4ac4-be06-8b835ac55ddd'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'bf3d64fb-a3b2-41a9-a04e-361c2673730b'), (b'Date', b'Wed, 13 Dec 2023 09:12:29 GMT'), (b'Transfer-Encoding', b'chunked')])
2023-12-13T09:12:29.251Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.251Z [container-agent] 2023-12-13 09:12:29 INFO juju-log HTTP Request: GET https://10.152.183.1/api/v1/namespaces/cos/pods/alertmanager-0 "HTTP/1.1 200 OK"
2023-12-13T09:12:29.261Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.261Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log receive_response_body.started request=<Request [b'GET']>
2023-12-13T09:12:29.270Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.270Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log receive_response_body.complete
2023-12-13T09:12:29.282Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.282Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log response_closed.started
2023-12-13T09:12:29.291Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.291Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log response_closed.complete
2023-12-13T09:12:29.311Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-ids" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.322Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-list" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.333Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-list" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.342Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.344Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.356Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.357Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.367Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.370Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.380Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.381Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.394Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.394Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.408Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.409Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.422Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-get" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.435Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.437Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.457Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.457Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.466Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.468Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.476Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.477Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.486Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.487Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.499Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.500Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.508Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.509Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.518Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.519Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.527Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.528Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.537Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.538Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.549Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.550Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.560Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.561Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.574Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-get" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.588Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.589Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.599Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.599Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.623Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.624Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.639Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.639Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.648Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.649Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.664Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.665Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log Could not locate cos-tool at: "cos-tool-amd64"
2023-12-13T09:12:29.675Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.675Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log Skipping injection of juju topology as label matchers
2023-12-13T09:12:29.684Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.684Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log `cos-tool` unavailable. Leaving expression unchanged: vector(1)
2023-12-13T09:12:29.693Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.694Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log Reading <property object at 0x7f746155f770> rule from /var/lib/juju/agents/unit-alertmanager-0/charm/src/prometheus_alert_rules/heartbeat.rule
2023-12-13T09:12:29.707Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.707Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log `cos-tool` unavailable. Leaving expression unchanged: alertmanager_config_last_reload_successful{} != 1
2023-12-13T09:12:29.717Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.717Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log Reading <property object at 0x7f746155f770> rule from /var/lib/juju/agents/unit-alertmanager-0/charm/src/prometheus_alert_rules/alertmanager_configuration_reload_failure.rule
2023-12-13T09:12:29.730Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.730Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log `cos-tool` unavailable. Leaving expression unchanged: absent(up{})
2023-12-13T09:12:29.739Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.739Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log Reading <property object at 0x7f746155f770> rule from /var/lib/juju/agents/unit-alertmanager-0/charm/src/prometheus_alert_rules/alertmanager_missing.rule
2023-12-13T09:12:29.752Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.752Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log `cos-tool` unavailable. Leaving expression unchanged: rate(alertmanager_notifications_failed_total{integration=~".*"}[5m]) > 0
2023-12-13T09:12:29.760Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.761Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log Reading <property object at 0x7f746155f770> rule from /var/lib/juju/agents/unit-alertmanager-0/charm/src/prometheus_alert_rules/alertmanager_notifications_failed.rule
2023-12-13T09:12:29.774Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.775Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.784Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.784Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.795Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.796Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.818Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.818Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.836Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-ids" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.846Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-ids" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.905Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "relation-get" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.916Z [container-agent] 2023-12-13 09:12:29 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:29.916Z [container-agent] 2023-12-13 09:12:29 DEBUG juju-log applying config changes
2023-12-13T09:12:31.136Z [container-agent] 2023-12-13 09:12:31 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:31.136Z [container-agent] 2023-12-13 09:12:31 WARNING juju-log config reload via HTTP POST failed: Bad response
2023-12-13T09:12:31.149Z [container-agent] 2023-12-13 09:12:31 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:31.149Z [container-agent] 2023-12-13 09:12:31 INFO juju-log Restarting service alertmanager
2023-12-13T09:12:32.678Z [container-agent] 2023-12-13 09:12:32 DEBUG jujuc server.go:222 running hook tool "juju-log" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:32.678Z [container-agent] 2023-12-13 09:12:32 WARNING juju-log cannot determine if reload succeeded
2023-12-13T09:12:32.691Z [container-agent] 2023-12-13 09:12:32 DEBUG jujuc server.go:222 running hook tool "relation-ids" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:32.702Z [container-agent] 2023-12-13 09:12:32 DEBUG jujuc server.go:222 running hook tool "relation-list" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:32.724Z [container-agent] 2023-12-13 09:12:32 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:32.726Z [container-agent] 2023-12-13 09:12:32 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:32.748Z [container-agent] 2023-12-13 09:12:32 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:32.748Z [container-agent] 2023-12-13 09:12:32 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:32.759Z [container-agent] 2023-12-13 09:12:32 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:32.762Z [container-agent] 2023-12-13 09:12:32 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:32.779Z [container-agent] 2023-12-13 09:12:32 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:32.779Z [container-agent] 2023-12-13 09:12:32 DEBUG jujuc server.go:222 running hook tool "relation-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:32.786Z [container-agent] 2023-12-13 09:12:32 DEBUG jujuc server.go:222 running hook tool "status-set" for alertmanager/0-update-status-8263556238986931756
2023-12-13T09:12:33.143Z [container-agent] 2023-12-13 09:12:33 INFO juju.worker.uniter.operation runhook.go:186 ran "update-status" hook (via hook dispatching script: dispatch)
2023-12-13T09:12:33.162Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.worker.uniter.operation executor.go:135 committing operation "run update-status hook" for alertmanager/0
2023-12-13T09:12:33.171Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.machinelock machinelock.go:206 created rotating log file "/var/log/machine-lock.log" with max size 10 MB and max backups 5
2023-12-13T09:12:33.172Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.machinelock machinelock.go:190 machine lock "machine-lock" released for alertmanager/0 uniter (run update-status hook)
2023-12-13T09:12:33.172Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.worker.uniter.operation executor.go:124 lock released for alertmanager/0
2023-12-13T09:12:33.172Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.worker.uniter.storage resolver.go:173 next hook op for storage-data-0: {Kind:2 Life:alive Attached:true Location:/var/lib/juju/storage/data/0}
2023-12-13T09:12:33.172Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.worker.uniter resolver.go:194 no operations in progress; waiting for changes
2023-12-13T09:12:33.172Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.worker.uniter.relation resolver.go:285 unit "ca/0" already joined relation 29
2023-12-13T09:12:33.172Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.worker.uniter.relation resolver.go:285 unit "traefik/0" already joined relation 10
2023-12-13T09:12:33.172Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.worker.uniter.relation resolver.go:285 unit "catalogue/0" already joined relation 26
2023-12-13T09:12:33.172Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.worker.uniter.relation resolver.go:285 unit "grafana/0" already joined relation 14
2023-12-13T09:12:33.172Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.worker.uniter.relation resolver.go:285 unit "prometheus/0" already joined relation 17
2023-12-13T09:12:33.172Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.worker.uniter.relation resolver.go:285 unit "prometheus/0" already joined relation 11
2023-12-13T09:12:33.173Z [container-agent] 2023-12-13 09:12:33 DEBUG juju.worker.uniter.relation resolver.go:285 unit "loki/0" already joined relation 15