multiple peers send alert that should only be sent once after failed notify attempt about alertmanager HOT 4 CLOSED

Hi! 👋 Thanks for opening this issue. I assume these Alertmanagers are clustered using the --cluster flags. If not, please make sure that clustering is configured using https://github.com/prometheus/alertmanager#high-availability as a reference.

This sounds like expected behavior to me. What has happened here is when Alertmanagers are clustered together they assume positions like position 0, 1, etc. The Alertmanager in position 0 will try to send the notification first, while the Alertmanager in position 1 will wait a timeout called --peer-timeout (default 15s). If the Alertmanager in position 0 cannot send the notification within --peer-timeout seconds then the Alertmanager in position 1 will try to send it as well.

In cases such as this, Alertmanager can send duplicate notifications. Most important is that Alertmanager guarantees to send a notification at least once, but it can send the same notification more than once.

from alertmanager.

Hey @grobinson-grafana, thanks for the quick reply! AFAIK alert managers should be clustered automatically by using more than one replica in a Kubernetes cluster.

While I agree that most times "Most important is that Alertmanager guarantees to send a notification at least once" is true, in my use case it's not. Do you know of any way to alter the behavior so it will always only send one alert?

If not I'll accept "expected behavior" as an answer and this issue can be closed. Thanks in advance!

from alertmanager.

Hi! 👋 No it's not possible to change the behavior without disabling clustering / HA, and even that does not guarantee exactly-once behavior. This is fundamental to how Alertmanager works.

from alertmanager.

ight, thanks!!

from alertmanager.