PagerDuty alerts are stuck if I mute the alert after it's firing but before it's resolved about alertmanager HOT 3 OPEN

@TheMeier yes, but it's true by default, according to https://prometheus.io/docs/alerting/latest/configuration/#pagerduty_config
Additionally, I have most of the alerts resolving itself fine unless they are muted.

from alertmanager.

In the provided config there is no send_resolved: true for the pagerduty receiver

from alertmanager.

This is an unfortunate side effect of how the Alertmanager works. This is a frequent complaint and fixing it requires a lot of effort.

The issue is that the Alertmanager will delete the alert from the group if:

1. The alert is resolved.
2. A notification for the group was just sent (even if the alert was not in that notification because it was muted).

If this happens while a mute time is still active the resolved alert is deleted. That means when the mute time ends it doesn't even know that this alert existed – for example – it could have been deleted 6 hours ago.

A possible fix for this is to read the nflog when checking if an alert can be deleted – however it does not work if the Alertmanager is restarted because alerts are kept in memory and not persisted to disk.

from alertmanager.