incaller / prometheus_bot Goto Github PK
View Code? Open in Web Editor NEWTelegram bot for prometheus alerting
License: MIT License
Telegram bot for prometheus alerting
License: MIT License
Is it possible to send messages to several groups if multiple receivers are configured in the Alertmanager? Listing several telegram_token separated by commas does not work. It also fails to run by pointing to several configuration files (-c config1.yaml -c config2.yaml). Thanks.
How I can insert emoji in Template?
Hey! I would like to help you with the configuration, I see that you use global variables in your code and I didn’t really like it. Let me implement the gostructor configuration library into your bot for configuration. It allows you to configure from different sources, including secrets from vault, in the future there will be support for kubernetes secrets
Library place here: https://github.com/goreflect/gostructor
I will be glad to suggestions and comments on my library
I check out this project with master branch. When I press command "make" then the result is :
can't load package: package .: code in directory /Users/mac/Documents/prometheusbot/prometheus_bot expects import "github.com/inCaller/prometheus_bot"
make: *** [all] Error 1
Hi, I saw that you do not have an automatic build of the project and publication on the github registry, maybe you need help with this? I can issue an MR if you agree
What will it give? People do not have to build the project locally, they can immediately pull this image to their local / production env
I'm migrating all alerts from Slack to Telegram right now. My testing shows wrong output from prometheus_bot.
How to reproduce. The easiest way to trigger an alert is to turn off an exporter (node exporter in my case).
Expected behavior. Firing and resolved alerts are not mixed together. I don't see the way to achieve it with custom templates.
My environment.
All running in docker:
slack alerts are configured using these custom templates.
So i did a git clone go get and go build and i got the executable. but when i run the executable it just says 2019/02/03 16:08:24 Not Found. what could i be doing wrong, i have setup the config.yaml as described in urreadm.e
template_path: "testdata/default.tmpl"
doesn't work.
Hello,
I have set up telegram bot with the manual steps but after starting up, container stays in restarting phase and because of this I am not able to receive any alterts on Telegram channel.
What can I see in logs is:
2021/06/21 13:38:45 Load template file:/etc/telegrambot/template.tmpl 2021/06/21 13:38:45 Authorised on account mine_alerts_bot
Can anyone help here, please?
Occurs when generation URL is empty
ubuntu@somewhere:~/prometheus_bot$ podman build -t incaller_prometheus_bot .
[truncated output]
# github.com/gin-gonic/gin
/go/src/github.com/gin-gonic/gin/context.go:77:11: undefined: http.SameSite
github.com/technoweenie/multipartstreamer
gopkg.in/telegram-bot-api.v4
Error: error building at STEP "RUN cd / && apk update && apk add --no-cache git ca-certificates make tzdata && git clone https://github.com/inCaller/prometheus_bot && cd prometheus_bot &
& go get -d -v && CGO_ENABLED=0 GOOS=linux go build -v -a -installsuffix cgo -o prometheus_bot": error while running runtime: exit status 2
why are the type(summary), Instance and description empty?
config template.tmpl:
Type: "{{ .CommonAnnotations.description }}"
Summary: "{{.CommonAnnotations.summary}}"
Alertname: "{{ .CommonLabels.alertname }}"
Instance: "{{ .CommonLabels.instance }}"
Severity: {{ .CommonLabels.severity}}
Status: {{ .Status }}
output:
Type: ""
Summary: ""
Alertname: "HostOutOfMemory"
Instance: ""
Severity: warning
Status: firing
This happens because every time series is uniquely identified by its metric name and a set of labels. And every unique combination of key-value label pairs represents a new alert for this time series.
Hi,
I am wondering if there is a way to let Prometheus check whether the prometheus_bot
service is running fine.
Basically, I need an entry into the prometheus up
metric, for example:
up{instance="prometheus_bot_instance_name", job="my_job"}
This is useful to allow the monitoring of the monitoring infrastructure.
The use case is the following:
This ensure that as soon as one of the monitoring infrastructures stops to work, the other one will fire a warning.
To have the required up
metric, I believe it suffices to provide a /metrics
endpoint.
Regards
In some cases, we have very long alert message (for example, we have alert for underreplicated kafka topics, grouped by instance, and entire instance with 40 topics was failed).
When prometheus_bot tries to send the similar message - it fails and flood the channel with Error sending message, checkout logs
messages.
Telegram API have message length limit - 4096 UTF8 characters (https://core.telegram.org/method/messages.sendMessage). Looks like your bot doesn't check message length.
Hi, thanks for making this telegram adapter! It looks like just the thing I need. Could you publish it on Docker Hub to make it easier to try out? Thanks!
Hello i made setup and my chat ID contains -- in it.
I put it in alertmanager just like said in manual with - together, but all the time im getting error:
Error sending message: Bad Request: chat not found
Im my alert manager i put it as:
'http://prometheus-bot:9087/alert/-1001719930860'
Hi!
Thank you for your bot application.
Could you please add the endpoint for readiness probe of your bot.
It would be great if it won't create access log records while do the probe.
Can you tag releases and regularly build and publish a Docker Image? Would make it easier to include this in a docker-compose based deployment of prometheus.
I'm trying prometheus_bot from master branch (271ff17) compiled with Go 1.6.2 from Ubuntu 16.04 64-bit and have observed panic during handling of first requests with newly created Telegram bot.
I can't reproduce this issue currently and I believe the issue is related to freshly created Telegram bot or some issue on Telegram side.
I want to leave this issue here for the reference, in case anyone else will observe it later.
Here is what I've done.
./prometheus_bot -c config.yml
make test
without specification of TELEGRAM_CHATID
:$ make test
bash t/curl.t
1..2
not ok 1 - simpe.json
# 404
not ok 2 - noGenURL.json
# 404
This lead to 404 responses from prometheus_bot
.
TELEGRAM_CHATID
:$ make test TELEGRAM_CHATID=-...
bash t/curl.t
1..2
not ok 1 - simpe.json
not ok 2 - noGenURL.json
This lead to panic. Here is prometheus_bot log for the whole session:
$ ./prometheus_bot -c config.yml
Authorized on account SomeBot
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
- using env: export GIN_MODE=release
- using code: gin.SetMode(gin.ReleaseMode)
[GIN-debug] GET /ping/:chatid --> main.GET_Handling (3 handlers)
[GIN-debug] POST /alert/:chatid --> main.POST_Handling (3 handlers)
[GIN-debug] Listening and serving HTTP on :9087
[GIN] 2017/03/06 - 22:17:30 | 404 | 645ns | 127.0.0.1 | POST /alert/
[GIN] 2017/03/06 - 22:17:30 | 404 | 628ns | 127.0.0.1 | POST /alert/
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0xc8 pc=0x4012aa]
goroutine 50 [running]:
panic(0x8f5180, 0xc820010170)
/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
main.telegramBot(0xc8200771a0)
/home/bob/stuff/kubernetes/prometheus_bot/main.go:88 +0x2aa
created by main.main
/home/bob/stuff/kubernetes/prometheus_bot/main.go:138 +0x8f7
I will bump go in my forked repository and test the functionality of source code, but let's have this here to keep track of upgrading golang.
Hello!
I'm using bot in docker, and there is problem with big alerting message, in telegram chat we getting "Error sending message, checkout logs".
In docker logs there are errors "Error sending message: Bad Request: can't parse entities: Unexpected end tag at byte offset 0", with different digits at the end.
JSON data is here: https://pastebin.com/31uZTPZa
Hello.
I have copied your Dockerfile and executed:
docker build -t prometheus_bot:latest .
Then I have received message about error:
...
github.com/ugorji/go/codec
github.com/gin-gonic/gin/binding
github.com/gin-gonic/gin/render
github.com/gin-gonic/gin
# github.com/gin-gonic/gin
/go/src/github.com/gin-gonic/gin/context.go:778:91: undefined: http.SameSite
/go/src/github.com/gin-gonic/gin/context.go:788:11: unknown field 'SameSite' in struct literal of type http.Cookie
The command '/bin/sh -c cd / && apk update && apk add --no-cache git ca-certificates make tzdata && git clone https://github.com/inCaller/prometheus_bot && cd prometheus_bot && go get -d -v && CGO_ENABLED=0 GOOS=linux go build -v -a -installsuffix cgo -o prometheus_bot' returned a non-zero code: 2
Hi !
I have encountered some trouble is when alert via telegram is only like below:
Status: FIRING 🔥
Active Alert List:
Alert:
Current value:Severity: critical
Active from:
How can I get the server down?
Thanks u so much !!
There's no license defined for this project (or I just didn't find one), so it cannot be used, modified or distributed legally.
Could you please define it?
Steps to reproduce
summary
field of CommonAnnotations):Alert {{.Status}}
{{.CommonAnnotations.summary}}
###skipped###
"startsAt": "2017-03-15T09:23:37.861+03:00"
}
],
"commonAnnotations": {
},
"commonLabels": {
"alertname": "low_disk_space",
###skipped###
Expected behavior: bot sends empty string instead of {{.CommonAnnotations.summary}}
Real behavior:
Error sending message, checkout logs
and 503 error in logs:+--------------- F I N A L M E S S A G E ---------------+
Alert firing
<no value>
+-----------------------------------------------------------+
Error sending message: Bad Request
[GIN] 2017/03/15 - 17:06:04 | 503 | 1.585562539s | 127.0.0.1 | POST /alert/-XXXXXX
Error sending message, checkout logs
messages. Looks like Alertmanager send a POST, got 503, resend POST, got 503 again etc...prometheus_bot Version: Latest commit 3767980 on Jul 26
2018/10/05 16:48:41 getUpdates resp: {"ok":false,"error_code":409,"description":"Conflict: can't use getUpdates method while webhook is active"}
2018/10/05 16:48:41 Conflict: can't use getUpdates method while webhook is active
2018/10/05 16:48:41 Failed to get updates, retrying in 3 seconds...
Just wondering why the Dockerfile compiles the code disabling go garbage collector? There is no explanation of this decision...
Besides, executing a build that prints the escape analysis returns a lot of heap allocations:
Command:
go build -gcflags="-m" main.go 2>&1 | grep "escapes to heap"
Result:
./main.go:205:21: str_fl escapes to heap
./main.go:205:21: str_Size escapes to heap
./main.go:251:21: str_fl escapes to heap
./main.go:251:21: str_Size escapes to heap
./main.go:134:16: "Could not convert value to int" escapes to heap
./main.go:270:15: "template_time_zone is not set, if you use template and `str_FormatDa... escapes to heap
./main.go:275:15: "template_time_outdata param is not set, if you use template and `str... escapes to heap
./main.go:287:13: (interface {})(string("time: missing Location in call to Time.In")) escapes to heap
./main.go:330:104: update.Message.Chat.ID escapes to heap
./main.go:334:11: msg escapes to heap
./main.go:340:16: update escapes to heap
./main.go:364:14: tmplPath escapes to heap
./main.go:372:18: []string{} escapes to heap
./main.go:377:13: sub + string(r) escapes to heap
./main.go:462:48: string(~R0) escapes to heap
./main.go:463:46: gin.H{...} escapes to heap
./main.go:464:4: fmt.Sprint(... argument...) escapes to heap
./main.go:463:9: render.Render(render.JSON{...}) escapes to heap
./main.go:469:13: chatid escapes to heap
./main.go:470:25: chatid escapes to heap
./main.go:475:26: msg escapes to heap
./main.go:477:11: render.Render(render.String{...}) escapes to heap
./main.go:479:38: gin.H{...} escapes to heap
./main.go:480:4: fmt.Sprint(... argument...) escapes to heap
./main.go:481:4: sendmsg escapes to heap
./main.go:479:9: render.Render(render.JSON{...}) escapes to heap
./main.go:487:14: make([]string, 0, len(alerts.GroupLabels)) escapes to heap
./main.go:491:14: sort.Interface(sort.StringSlice(sort.x)) escapes to heap
./main.go:492:21: make([]string, 0, len(alerts.GroupLabels)) escapes to heap
./main.go:494:49: k escapes to heap
./main.go:497:13: make([]string, 0, len(alerts.CommonLabels)) escapes to heap
./main.go:501:14: sort.Interface(sort.StringSlice(sort.x)) escapes to heap
./main.go:502:22: make([]string, 0, len(alerts.CommonLabels)) escapes to heap
./main.go:505:52: k escapes to heap
./main.go:509:13: make([]string, 0, len(alerts.CommonAnnotations)) escapes to heap
./main.go:513:14: sort.Interface(sort.StringSlice(sort.x)) escapes to heap
./main.go:514:27: make([]string, 0, len(alerts.CommonAnnotations)) escapes to heap
./main.go:516:61: k escapes to heap
./main.go:519:22: make([]string, len(alerts.Alerts)) escapes to heap
./main.go:529:58: a.GeneratorURL escapes to heap
./main.go:529:85: alertDetails[i] escapes to heap
./main.go:534:9: alerts.ExternalURL escapes to heap
./main.go:535:9: alerts.Receiver escapes to heap
./main.go:536:18: strings.ToUpper(alerts.Status) escapes to heap
./main.go:537:6: len(alerts.Alerts) escapes to heap
./main.go:538:15: strings.Join(groupLabels, ", ") escapes to heap
./main.go:539:15: strings.Join(commonLabels, ", ") escapes to heap
./main.go:540:15: strings.Join(commonAnnotations, "") escapes to heap
./main.go:541:15: strings.Join(alertDetails, ", ") escapes to heap
./main.go:558:20: alerts escapes to heap
./main.go:565:25: string(bytes.b.buf[bytes.b.off:]) escapes to heap
./main.go:571:24: &strings.Reader{...} escapes to heap
./main.go:582:16: "HTML is valid, sending it..." escapes to heap
./main.go:587:16: "HTML is not valid, strip all tags to prevent error" escapes to heap
./main.go:607:13: chatid escapes to heap
./main.go:610:48: string(~R0) escapes to heap
./main.go:611:46: gin.H{...} escapes to heap
./main.go:612:4: fmt.Sprint(... argument...) escapes to heap
./main.go:611:9: render.Render(render.JSON{...}) escapes to heap
./main.go:619:24: alerts escapes to heap
./main.go:625:14: "+------------------ A L E R T J S O N -------------------+" escapes to heap
./main.go:626:13: s escapes to heap
./main.go:627:14: "+-----------------------------------------------------------+\n\n" escapes to heap
./main.go:643:15: "+--------------- F I N A L M E S S A G E ---------------+" escapes to heap
./main.go:644:14: subString escapes to heap
./main.go:645:15: "+-----------------------------------------------------------+" escapes to heap
./main.go:652:27: msg escapes to heap
./main.go:654:12: render.Render(render.String{...}) escapes to heap
./main.go:657:47: gin.H{...} escapes to heap
./main.go:658:5: fmt.Sprint(... argument...) escapes to heap
./main.go:659:5: sendmsg escapes to heap
./main.go:660:5: fmt.Sprint(... argument...) escapes to heap
./main.go:660:26: msgtext escapes to heap
./main.go:657:10: render.Render(render.JSON{...}) escapes to heap
./main.go:666:12: msg escapes to heap
./main.go:410:47: string(content) escapes to heap
./main.go:417:36: &http.Client{} escapes to heap
./main.go:432:9: -1 escapes to heap
./main.go:440:14: (interface {})(string("gin mode unknown: ") + gin.value) escapes to heap
./main.go:440:14: string("gin mode unknown: ") + gin.value escapes to heap
./main.go:443:49: bot.Self.UserName escapes to heap
./main.go:453:12: []gin.HandlerFunc{...} escapes to heap
./main.go:454:13: []gin.HandlerFunc{...} escapes to heap
./main.go:309:31: "html/template".FuncMap{...} escapes to heap
Hi,
I have loaded the current code onto my Raspberry Pi4 and then compiled it.
However, after a make test
I get the following error when using the templates from this project.
Excerpts from the bot.log
2022/03/16 20:07:04 HTML is valid, sending it...
2022/03/16 20:07:04 +--------------- F I N A L M E S S A G E ---------------+
2022/03/16 20:07:04 <a href='https://alert-manager.example.com/#/alerts?receiver=admins'>[FIRING:1]</a>
grouped by: alertname=<code>something_happend</code>, instance=<code>server01.int:9100</code>
labels: env=<code>prod</code>, job=<code>node</code>, service=<code>prometheus_bot</code>, severity=<code>warning</code>, supervisor=<code>runit</code>
summary: <code>runit service prometheus_bot restarted, server01.int:9100</code>
<a href='https://example.com/graph#...'>server01.int[node]</a>
2022/03/16 20:07:04 +-----------------------------------------------------------+
[GIN] 2022/03/16 - 20:07:04 | 200 | 65.982888ms | 127.0.0.1 | POST "/alert/-4*******3"
****** Run prometheus_bot with template testdata/default.tmpl ******
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x45db1c]
goroutine 1 [running]:
main.main()
/usr/src/prometheus_bot/main.go:418 +0x31c
This error only occurs in conjunction with the templates. Sending without using a template works.
Excerpts from the bot.log
2022/03/16 20:07:04 HTML is valid, sending it...
2022/03/16 20:07:04 +--------------- F I N A L M E S S A G E ---------------+
2022/03/16 20:07:04 <a href='http://alert.greco.cf/alert-manager/#/alerts?receiver=telegram_bot'>[FIRING:11]</a>
grouped by: scada_uuid=<code>483b197c-7fe8-11e6-b772-acb57db47f23</code>
labels:
<a href='http://localhost.localdomain:9090/graph?g0.expr=linux_loadavg%7Bmode%3D%2215min%22%7D+%3E+%283+%2A+10+%2A+4%29&g0.tab=0'>localhost[statsd]</a>, <a href='http://localhost.localdomain:9090/graph?g0.expr=linux_memory%7Bmode%3D%22memavailable%22%7D+%3E+%281024+%2A+100%29&g0.tab=0'>localhost[statsd]</a>, <a href='http://localhost.localdomain:9090/graph?g0.expr=100+-+%28avg%28irate%28linux_stats_cpu%7Bmode%3D%22idle%22%7D%5B2m%5D%29%29+BY+%28scada_uuid%29%29+%3E+60&g0.tab=0'></a>, <a href='http://localhost.localdomain:9090/graph?g0.expr=linux_loadavg%7Bmode%3D%221min%22%7D+%3E+%288+%2A+10+%2A+4%29&g0.tab=0'>localhost[statsd]</a>, <a href='http://localhost.localdomain:9090/graph?g0.expr=linux_loadavg%7Bmode%3D%221min%22%7D+%3E+%2810+%2A+10+%2A+4%29&g0.tab=0'>localhost[statsd]</a>, <a href='http://localhost.localdomain:9090/graph?g0.expr=linux_loadavg%7Bmode%3D%225min%22%7D+%3E+%285+%2A+10+%2A+4%29&g0.tab=0'>localhost[statsd]</a>, <a href='http://localhost.localdomain:9090/graph?g0.expr=linux_loadavg%7Bmode%3D%225min%22%7D+%3E+%288+%2A+10+%2A+4%29&g0.tab=0'>localhost[statsd]</a>, <a href='http://localhost.localdomain:9090/graph?g0.expr=linux_loadavg%7Bmode%3D%2215min%22%7D+%3E+%282+%2A+10+%2A+4%29&g0.tab=0'>localhost[statsd]</a>, <a href='http://localhost.localdomain:9090/graph?g0.expr=linux_loadavg%7Bmode%3D%2215min%22%7D+%3E+%282+%2A+10+%2A+4%29&g0.tab=0'>localhost[statsd]</a>, <a href='http://localhost.localdomain:9090/graph?g0.expr=linux_loadavg%7Bmode%3D%2215min%22%7D+%3E+%282+%2A+10+%2A+4%29&g0.tab=0'>localhost[statsd]</a>, <a href='http://localhost.localdomain:9090/graph?g0.expr=linux_loadavg%7Bmode%3D%2215min%22%7D+%3E+%282+%2A+10+%2A+4%29&g0.tab=0'>localhost[statsd]</a>
2022/03/16 20:07:04 +-----------------------------------------------------------+
[GIN] 2022/03/16 - 20:07:04 | 200 | 53.328811ms | 127.0.0.1 | POST "/alert/-4*******3"
2022/03/16 20:07:04 Bot alert post: -4*******3
2022/03/16 20:07:04 +------------------ A L E R T J S O N -------------------+
2022/03/16 20:07:04 {"alerts":[{"annotations":{"summary":"Oops, something happend!"},"endsAt":"0001-01-01T00:00:00Z","generatorURL":"https://example.com/graph#...","labels":{"alertname":"something_happend","env":"prod","instance":"server01.int:9100","job":"node","service":"prometheus_bot","severity":"warning","supervisor":"runit"},"startsAt":"2016-04-27T20:46:37.903Z","status":"firing"}],"commonAnnotations":{"summary":"runit service prometheus_bot restarted, server01.int:9100"},"commonLabels":{"alertname":"something_happend","env":"prod","instance":"server01.int:9100","job":"node","service":"prometheus_bot","severity":"warning","supervisor":"runit"},"externalURL":"https://alert-manager.example.com","groupKey":0,"groupLabels":{"alertname":"something_happend","instance":"server01.int:9100"},"receiver":"admins","status":"firing","version":0}
2022/03/16 20:07:04 +-----------------------------------------------------------+
Output after make test
make test
go build -o prometheus_bot
prove -v
t/curl.t ..
1..25
ok 1 - noGenURL.json template none
ok 2 - emptyValue.json template none
ok 3 - production_example.json template none
ok 4 - simpe.json template none
ok 5 - big_output.json template none
not ok 6 - noGenURL.json template default.tmpl
not ok 7 - emptyValue.json template default.tmpl
not ok 8 - production_example.json template default.tmpl
not ok 9 - simpe.json template default.tmpl
not ok 10 - big_output.json template default.tmpl
not ok 11 - noGenURL.json template malformed_html.tmpl
not ok 12 - emptyValue.json template malformed_html.tmpl
not ok 13 - production_example.json template malformed_html.tmpl
not ok 14 - simpe.json template malformed_html.tmpl
not ok 15 - big_output.json template malformed_html.tmpl
not ok 16 - noGenURL.json template detailed_vars.tmpl
not ok 17 - emptyValue.json template detailed_vars.tmpl
not ok 18 - production_example.json template detailed_vars.tmpl
not ok 19 - simpe.json template detailed_vars.tmpl
not ok 20 - big_output.json template detailed_vars.tmpl
not ok 21 - noGenURL.json template production_example.tmpl
not ok 22 - emptyValue.json template production_example.tmpl
not ok 23 - production_example.json template production_example.tmpl
not ok 24 - simpe.json template production_example.tmpl
not ok 25 - big_output.json template production_example.tmpl
Failed 20/25 subtests
Test Summary Report
-------------------
t/curl.t (Wstat: 0 Tests: 25 Failed: 20)
Failed tests: 6-25
Files=1, Tests=25, 16 wallclock secs ( 0.06 usr 0.02 sys + 0.73 cusr 0.79 csys = 1.60 CPU)
Result: FAIL
make: *** [Makefile:6: test] Fehler 1
I have already used this code in July 2021.
The commit at that time was:
commit 2dc161747f6a6ee8afbc8afd1b7e02f0bc195daf (HEAD -> master, origin/master, origin/HEAD)
Merge: ac4533f 1d366fe
Author: Roman Belyakovsky <[email protected]>
Date: Sun May 30 23:20:07 2021 +0300
Merge pull request #64 from dmaes/master
Clarify documentation about `-` in chat_id and url
This code from May 2021 still works today.
I can compile it in the same system environment and use it with the same templates.
There are no errors.
Where is the difference to today's version?
My current environment
$ uname -a
Linux node5.local 5.10.92-v8+ #1514 SMP PREEMPT Mon Jan 17 17:39:38 GMT 2022 aarch64 GNU/Linux
$ cat /etc/*rel*
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
$ go version
go version go1.17.2 linux/arm64
Can you help me fix the problem?
Greetings...
Stefan
Hello! Thanks for prometheus_bot.
Please create binary release.
Telegram introduced forums: a chat that has multiple topics inside (technically these are one big chat). Bots also have support for this feature. I'd like to send a notification in the specific topic, but can't find anything to set it, so I guess it's not implemented.
Adding this feature is as simple as adding one more parameter named message_thread_id
to the Telegram Bot API request.
log:
2019/10/31 11:51:38 Bot alert post: 638489237
2019/10/31 11:51:38 +------------------ A L E R T J S O N -------------------+
2019/10/31 11:51:38 {"alerts":[{"annotations":{"description":"Disk is almost running out of available space (\u003c 10% left)\n VALUE = 3.102\n LABELS: map[datacenter:fr device:/dev/mapper/san1ve-alertmanager fstype:xfs instance:st-csb2.fr:9100 job:node_exporter mountpoint:/mnt/san_volumes/alertmanager]","summary":"Out of disk space (instance st-csb2.fr:9100)"},"endsAt":"0001-01-01T00:00:00Z","generatorURL":"http://prometheus.local:9090/graph?g0.expr=node_filesystem_free_bytes+%2F+node_filesystem_size_bytes+%2A+100+%3C+10\u0026g0.tab=1","labels":{"alertname":"OutOfDiskSpace","datacenter":"fr","device":"/dev/mapper/san1ve-alertmanager","fstype":"xfs","instance":"st-csb2.fr:9100","job":"node_exporter","mountpoint":"/mnt/san_volumes/alertmanager","severity":"warning"},"startsAt":"2019-10-31T11:49:35.277221523Z"}],"commonAnnotations":{"description":"Disk is almost running out of available space (\u003c 10% left)\n VALUE = 3.102\n LABELS: map[datacenter:fr device:/dev/mapper/san1ve-alertmanager fstype:xfs instance:st-csb2.fr:9100 job:node_exporter mountpoint:/mnt/san_volumes/alertmanager]","summary":"Out of disk space (instance st-csb2.fr:9100)"},"commonLabels":{"alertname":"OutOfDiskSpace","datacenter":"fr","device":"/dev/mapper/san1ve-alertmanager","fstype":"xfs","instance":"st-csb2.fr:9100","job":"node_exporter","mountpoint":"/mnt/san_volumes/alertmanager","severity":"warning"},"externalURL":"http://alertmanager.fr.local:9093","groupKey":0,"groupLabels":{"alertname":"OutOfDiskSpace","instance":"st-csb2.fr:9100"},"receiver":"adm","status":"firing","version":0}
2019/10/31 11:51:38 +-----------------------------------------------------------+
2019/10/31 11:51:38 +--------------- F I N A L M E S S A G E ---------------+
2019/10/31 11:51:38 <a href='http://alertmanager.fr.local:9093/#/alerts?receiver=adm'>[FIRING:1]</a>
grouped by: alertname=<code>OutOfDiskSpace</code>, instance=<code>st-csb2.fr:9100</code>
labels: datacenter=<code>fr</code>, device=<code>/dev/mapper/san1ve-alertmanager</code>, fstype=<code>xfs</code>, job=<code>node_exporter</code>, mountpoint=<code>/mnt/san_volumes/alertmanager</code>, severity=<code>warning</code>
description: <code>Disk is almost running out of available space (< 10% left)
VALUE = 3.102
LABELS: map[datacenter:fr device:/dev/mapper/san1ve-alertmanager fstype:xfs instance:st-csb2.fr:9100 job:node_exporter mountpoint:/mnt/san_volumes/alertmanager]</code>
summary: <code>Out of disk space (instance st-csb2.fr:9100)</code>
<a href='http://prometheus.local:9090/graph?g0.expr=node_filesystem_free_bytes+%2F+node_filesystem_size_bytes+%2A+100+%3C+10&g0.tab=1'>st-csb2.fr[node_exporter]</a>
2019/10/31 11:51:38 +-----------------------------------------------------------+
2019/10/31 11:51:38 Error sending message: Bad Request: can't parse entities: Expected end tag at byte offset 472
[GIN] 2019/10/31 - 11:51:38 | 503 | 66.474775ms | 192.168.100.128 | POST /alert/638489237
config.yaml :
telegram_token: "5327342:GVJsadjgvwequJHSAGh"
time_zone: "Europe/Sofia"
split_token: "|"
alertmanager config:
- alert: OutOfDiskSpace
expr: node_filesystem_free_bytes / node_filesystem_size_bytes * 100 < 10
for: 5m
labels:
severity: warning
annotations:
summary: "Out of disk space (instance {{ $labels.instance }})"
description: "Disk is almost running out of available space (< 10% left)\n VALUE = {{ $value | humanize }}\n LABELS: {{ $labels }}"
# description: "Disk is almost full (< 10% left)\n {{ $labels.instance_short }}\n {{ $labels.mountpoint }}\n VALUE = {{ printf \"node_filesystem_avail_bytes{mountpoint='%s'}\" .Labels.mountpoint | query | first | value | humanize1024 }}"
The problem with description - (< 10% left)
, special symbols can not be parsed.
P.S. I am not a programmer and i dunno how to fix it, so i can't make a PR, sry.
Hi.
I have an error after docker container start:
Post "https://api.telegram.org/bot1255100632:xxxxxxxxxxxxxxxx/getMe": net/http: TLS handshake timeout
same request with browser returns correct json.
Can you tell me what should i fix?
It will be nice to have container like that.
Or at least have binary executable in release
I started a private chat with the bot, and got the following message: Chat id is '12345678'
.
Per README instructions, I then configured alertmanager with webhook url http://some.host.name/alert/-12345678
, resulting in Chat not found
error messages from the bot.
Broke my head over this, until I invited the bot into a group chat, and I got a chat_id with -
included (Chat id is '-5678901'
), which did work with url http://some.host.name/alert/-5678901
. Then trying to use the 'private' chat_id without -
in the url (http://some.host.name/alert/12345678
) also worked.
So I suggest updating README to something like:
### Configuring alert manager
Alert manager configuration file:
- name: 'admins'
webhook_configs:
- send_resolved: True
url: http://127.0.0.1:9087/alert/chat_id
Replace chat_id with the number you got from your bot, ***with everything inside the quotes*** (Some chat_id's start with `-`, so if there is a `-` in your chat_id, make sure to include it in the url). To use multiple chats just add more receivers.
As we have lots of pull requests adding more message styles it would be good to ensure original formats are not affected.
cat config.yaml
telegram_token: "xxx:xxx"
# ONLY IF YOU USING DATA FORMATTING FUNCTION, NOTE for developer: important or test fail
time_outdata: "02/01/2006 15:04:05"
#template_path: "template.tmpl" # ONLY IF YOU USING TEMPLATE
time_zone: "Asia/Shanghai" # ONLY IF YOU USING TEMPLATE
split_msg_byte: 4000
send_only: true # use bot only to send messages.
TELEGRAM_CHATID="-100xxx" make test
go build -o prometheus_bot
prove -v
t/curl.t ..
1..25
head: illegal line count -- -1
ok 1 - emptyValue.json template none
head: illegal line count -- -1
ok 2 - simpe.json template none
head: illegal line count -- -1
ok 3 - big_output.json template none
head: illegal line count -- -1
ok 4 - production_example.json template none
head: illegal line count -- -1
ok 5 - noGenURL.json template none
head: illegal line count -- -1
ok 6 - emptyValue.json template production_example.tmpl
head: illegal line count -- -1
ok 7 - simpe.json template production_example.tmpl
head: illegal line count -- -1
ok 8 - big_output.json template production_example.tmpl
head: illegal line count -- -1
ok 9 - production_example.json template production_example.tmpl
head: illegal line count -- -1
ok 10 - noGenURL.json template production_example.tmpl
head: illegal line count -- -1
ok 11 - emptyValue.json template default.tmpl
head: illegal line count -- -1
ok 12 - simpe.json template default.tmpl
head: illegal line count -- -1
ok 13 - big_output.json template default.tmpl
head: illegal line count -- -1
ok 14 - production_example.json template default.tmpl
head: illegal line count -- -1
ok 15 - noGenURL.json template default.tmpl
head: illegal line count -- -1
ok 16 - emptyValue.json template malformed_html.tmpl
head: illegal line count -- -1
ok 17 - simpe.json template malformed_html.tmpl
head: illegal line count -- -1
not ok 18 - big_output.json template malformed_html.tmpl
head: illegal line count -- -1
not ok 19 - production_example.json template malformed_html.tmpl
head: illegal line count -- -1
not ok 20 - noGenURL.json template malformed_html.tmpl
head: illegal line count -- -1
not ok 21 - emptyValue.json template detailed_vars.tmpl
head: illegal line count -- -1
not ok 22 - simpe.json template detailed_vars.tmpl
head: illegal line count -- -1
not ok 23 - big_output.json template detailed_vars.tmpl
head: illegal line count -- -1
not ok 24 - production_example.json template detailed_vars.tmpl
head: illegal line count -- -1
not ok 25 - noGenURL.json template detailed_vars.tmpl
Failed 8/25 subtests
Test Summary Report
-------------------
t/curl.t (Wstat: 0 Tests: 25 Failed: 8)
Failed tests: 18-25
Files=1, Tests=25, 40 wallclock secs ( 0.01 usr 0.01 sys + 1.29 cusr 0.66 csys = 1.97 CPU)
Result: FAIL
make: *** [test] Error 1
hello Thanks for prometheus_bot.
can I use the alertmanager templates for the telegram templates?
thank you~
Is it possible to make bot stop reading input from the channel? Is it possible to make one way communication? Because when I type something in chat, bot tell my chat ID.
thank you for your work. I am using this bot now in my prometheus installation.
I had to change the deserialization of alerts for me. Though it worked it always hat an 400 http error set - and at least with my windows environment the subsequent "c.AbortWithStatus(http.StatusOK)" would not work - so alertmanager would get an 400 back und retry the message endlessly.
So I changed:
var alerts Alerts
// c.BindJSON(&alerts)
binding.JSON.Bind(c.Request, &alerts)
Hey @AndreaGreco, I can see tests failing with go 1.8.3, any ideas how to fix this?
Problem reading parsing template file: template: production_example.tmpl:20: function "str_Format_MeasureUnit" not defined
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.