Hello Mr. Karan :)
I am using calert 2.0.2 in my kube-prometheus-stack deployment (31.0.0) to send the alerts from alertmanager to Google Chat receivers.
I use the prometheus-telegram-bot 1.0.0. as well.
Google Chat receivers works properly with 99% alerts/rules, but I have a problem with an specific alertmanager rule:
![image](https://user-images.githubusercontent.com/102799006/161504390-cd02515d-2d5c-46f2-8ab8-114cb89e2c08.png)
- alert: CollectorsMetricsInputInfo expr: sum by (collector, namespace, component, container) (round(increase(collector_inputcount[4h]))) == 0 for: 24h labels: severity: info {{- if .Values.kubeprometheusstack.defaultRules.additionalRuleLabels }} {{ toYaml .Values.kubeprometheusstack.defaultRules.additionalRuleLabels | indent 8 }} {{- end }} annotations: summary: Collector pod did not receive messages for 4 hours message: Collector {{
{{}}$labels.container{{
}}}} in namespace {{
{{}}$labels.namespace{{
}}}} from {{
{{}}$labels.component{{
}}}} has NO INPUT messages since 4 hours ago.
This rule give me alerts like this:
![collectorinfoalerts](https://user-images.githubusercontent.com/102799006/161504536-6d172ffa-9293-4a6f-9616-b671c60c5403.png)
As I said, these rule/alerts works properly with Telegram receivers, but not with Google Chat. In Google Chat, I get a huge flood of the same alert message repeating (maybe 100 in a row, in less than 5 minutes), continuously, some with "resolved" status, but most like "firing".
![gchatalertscollector](https://user-images.githubusercontent.com/102799006/161504722-f714874c-9277-4e73-b71c-6df0055dca6c.png)
The only way to avoid that flood is to silence this alerts in AlertManager.
Meanwhile, the behavior with prometheus-telegram-bot is normal.
To me, it looks related to the way to identify what alert is new and what is not, or their fingerprint, according to #25, but I am unable to see what is wrong with this kind of rule.
A few calert pod logs; I don't see anything strange on it; but you can see how is dispatching new messages to the chat almost each 30 seconds, and always the same alerts:
2022/04/04 06:29:59 [google-chat-bot-7fc5cfb54-f5ngv/krB1cViUTd-010859] "POST http://google-chat-bot:6000/dispatch HTTP/1.1" from 110.124.152.4:34410 - 200 40B in 7.066438793s
time="2022-04-04T06:29:59Z" level=info msg="dispatching new alert" receiver=default-alertmanager-config-alert-info
time="2022-04-04T06:29:59Z" level=info msg="dispatching alerts" count=10
time="2022-04-04T06:29:59Z" level=info msg="dispatching alerts to google chat" count=10
2022/04/04 06:30:05 [google-chat-bot-7fc5cfb54-f5ngv/krB1cViUTd-010860] "POST http://google-chat-bot:6000/dispatch HTTP/1.1" from 110.124.152.4:34482 - 200 40B in 6.447221446s
time="2022-04-04T06:30:05Z" level=info msg="dispatching new alert" receiver=default-alertmanager-config-alert-info
time="2022-04-04T06:30:05Z" level=info msg="dispatching alerts" count=10
time="2022-04-04T06:30:05Z" level=info msg="dispatching alerts to google chat" count=10
2022/04/04 06:30:11 [google-chat-bot-7fc5cfb54-f5ngv/krB1cViUTd-010861] "POST http://google-chat-bot:6000/dispatch HTTP/1.1" from 110.124.152.4:34578 - 200 40B in 6.297160372s
time="2022-04-04T06:30:11Z" level=info msg="dispatching new alert" receiver=default-alertmanager-config-alert-info
time="2022-04-04T06:30:11Z" level=info msg="dispatching alerts" count=10
time="2022-04-04T06:30:11Z" level=info msg="dispatching alerts to google chat" count=10
Would you be so kind to guide me on what might be going on?
I would like these alerts to reach both, Telegram and Google Chat, but the flood makes the Google chat channel unusable.
If you need any extra information please let me know.
A lot of thanks!!! :)