Comments (4)
Hi! 👋 Thanks for opening this issue. I assume these Alertmanagers are clustered using the --cluster
flags. If not, please make sure that clustering is configured using https://github.com/prometheus/alertmanager#high-availability as a reference.
This sounds like expected behavior to me. What has happened here is when Alertmanagers are clustered together they assume positions like position 0, 1, etc. The Alertmanager in position 0 will try to send the notification first, while the Alertmanager in position 1 will wait a timeout called --peer-timeout
(default 15s
). If the Alertmanager in position 0 cannot send the notification within --peer-timeout
seconds then the Alertmanager in position 1 will try to send it as well.
In cases such as this, Alertmanager can send duplicate notifications. Most important is that Alertmanager guarantees to send a notification at least once, but it can send the same notification more than once.
from alertmanager.
Hey @grobinson-grafana, thanks for the quick reply! AFAIK alert managers should be clustered automatically by using more than one replica in a Kubernetes cluster.
While I agree that most times "Most important is that Alertmanager guarantees to send a notification at least once" is true, in my use case it's not. Do you know of any way to alter the behavior so it will always only send one alert?
If not I'll accept "expected behavior" as an answer and this issue can be closed. Thanks in advance!
from alertmanager.
Hi! 👋 No it's not possible to change the behavior without disabling clustering / HA, and even that does not guarantee exactly-once behavior. This is fundamental to how Alertmanager works.
from alertmanager.
ight, thanks!!
from alertmanager.
Related Issues (20)
- Go stdlib vuln - please upgrade go version HOT 4
- AlertManager not respecting repeat_interval timer setting on duplicate alert HOT 1
- `amtool` ZSH completion is broken
- Prometheus Alertmanager not sending email notifications HOT 3
- Alerts are not triggering to its target communications HOT 2
- [UI] Feature: Localtime / Timezone option
- Alertmanager send resolved notification when problem not solved HOT 3
- Cannot send alerts to multiple receivers simultaneously using matchers field. HOT 8
- Unable to change time zone in Grafana Alerting Notification Template (Always in UTC) HOT 1
- bad response status 404 Not Found HOT 1
- Task: Improvements to silences
- Work around PrometheusOperatorSyncFailed, integration=opsgenie, max_runes=130? HOT 1
- config.SecretURL type returns Json.Unmarshal() error which can expose the secret url. HOT 1
- amtool does not have a way to specify a time to test with. HOT 4
- Support time_interval testing with amtool config routes test HOT 2
- AlertManager doesn't send alert to Slack HOT 1
- Docker image prom/alertmanager:v0.27.0 fails to parse webhook_url_file from msteams_config HOT 2
- An option to disable the default sorting of Alerts for a group of Alerts HOT 6
- AlertManager doesn't send some of alerts to Pagerduty HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from alertmanager.