Comments (6)
Could you share your alert.rules?
"/metrics" is the standard endpoint for Prometheus to scrape metrics from
an instrumented job - could you explain what part of that seems strange to
you?
What do you mean, you are using cadvisor for "prometheus analytics", when
you say the scraped targets are all sinatra apps? Could it be that you
meant to scrape cAdvisor, but you are accidentally scraping the containers
directly (which would be correct if you instrumented them correctly with
Prometheus metrics)?
By the way, you don't need the "labels: job: '...'" sections in your
config, since the job label will default to what you set in the "job_name"
already, which has the same value in your config.
On Tue, Oct 27, 2015 at 10:22 PM, casertap [email protected] wrote:
here is my prometheus.yml
scrape_configs:
- job_name: 'streamapp'
scrape_interval: 5s
scrape_timeout: 10s
target_groups:
- targets: ['streamapp-container:3002']
labels:
job: 'streamapp'
- job_name: 'dashboardapp'
scrape_interval: 5s
scrape_timeout: 10s
target_groups:
- targets: ['dashboardapp-container:3000']
labels:
job: 'dashboardapp'
- job_name: 'analyzeapp'
scrape_interval: 5s
scrape_timeout: 10s
target_groups:
- targets: ['analyzeapp-container:3001']
labels:
job: 'analyzeapp'All the containers are sinatra apps. I have setup '/metrics' routes on
each service to have alertmanager behave correctly.
If I do not provide a '/metrics' route my container is detected has dead.
Everything looks ok but when I stop one container (let say dashboardapp).
I get all 3 alerts in red and received 3 msg via slack. The 3 messages are
saying that dashboardapp is down.Am I doing things right?
I found it strange to have to provide a '/metrics' route and the fact that
I received 3 times the same message is strange.—
Reply to this email directly or view it on GitHub
#139.
from alertmanager.
I give you my alert.rules
### alert.rules ###
# Alert for any instance that is unreachable for >5 minutes.
ALERT streamapp
IF up == 0
FOR 10s
SUMMARY "Instance {{$labels.instance}} down"
DESCRIPTION "{{$labels.instance}} of job {{$labels.job}} has been down for more than 10s"
ALERT dashboardapp
IF up == 0
FOR 10s
SUMMARY "Instance {{$labels.instance}} down"
DESCRIPTION "{{$labels.instance}} of job {{$labels.job}} has been down for more than 10s"
ALERT analyzeapp
IF up == 0
FOR 10s
SUMMARY "Instance {{$labels.instance}} down"
DESCRIPTION "{{$labels.instance}} of job {{$labels.job}} has been down for more than 10s"
And here is my docker-compose (so you can better understand my architecture
datadb:
image: busybox
container_name: datadb
volumes:
- /data/db
mongo:
image: mongo
container_name: mongo
volumes_from:
- datadb
ports:
- 27017:27017
streamapp:
build: streamapp/
container_name: streamapp-container
links:
- mongo
ports:
- 3002:3002
expose:
- 3002
env_file:
- './.env'
dashboardapp:
build: dashboardapp/
container_name: dashboardapp-container
links:
- mongo
ports:
- 3000:3000
expose:
- 3000
env_file:
- './.env'
analyzeapp:
build: analyzeapp/
container_name: analyzeapp-container
ports:
- 3001:3001
expose:
- 3001
links:
- mongo
env_file:
- './.env'
cadvisor:
image: google/cadvisor
container_name: cadvisor
ports:
- "8080:8080"
volumes:
- "/:/rootfs:ro"
- "/var/run:/var/run:rw"
- "/sys:/sys:ro"
- "/var/lib/docker/:/var/lib/docker:ro"
promalertmgr:
build: promalertmgr/
container_name: promalertmgr
ports:
- "9093:9093"
expose:
- "9093"
command: "-config.file=/alertmanager.conf"
env_file:
- './.env'
prometheus:
build: prometheus/
container_name: prometheus
links:
- cadvisor
- promalertmgr
- streamapp
- dashboardapp
- analyzeapp
- mongo
ports:
- "9090:9090"
command: "-config.file=/etc/prometheus/prometheus.yml -alertmanager.url=http://192.168.99.100:9093"
env_file:
- './.env'
Prometheus works fine when I connect it to cadvisor:
scrape_configs:
- job_name: 'cadvisor'
scrape_interval: 5s
scrape_timeout: 10s
target_groups:
- targets: ['cadvisor:8080']
But what I want is monitor my streamapp, dashboardapp, and analyzeapp container and receive an alert when one of them is down.
If I connect prometheus to cadvisor like I did and add the 3 rules above. Those rules stay green even if I stop the dashboard container.
If I add "/metrics" end point to my apps and connect prometheus to each app while I did in my first post. If I stop one container I get alerted. But the metric do not work because I did not connect cadvisor.
How can I have the metric with cadvisor and detect if one of my container is down?
Thanks for your answer.
I hope to hear from you soon.
from alertmanager.
Ok so what I needed was something like this
ALERT streamapp
IF absent(container_last_seen{name="streamapp-container"})
SUMMARY "Instance {{$labels.instance}} down"
DESCRIPTION "{{$labels.instance}} of job {{$labels.job}} has been down for more than 10s"
I only need to connect prometheus to cadvisor and than use a cadvisor metric to detect if the container is down.
It takes a lot of time to detect the app is down but still, it is working.
Thanks.
from alertmanager.
Now that I have updated the docker image it does not work anymore.
I need to do it like this:
ALERT wpprog
IF absent(((time() - container_last_seen{name="wpprog"}) < 5)) FOR 15s
SUMMARY "Instance is down"
DESCRIPTION "You should do something"
This is working fine.
from alertmanager.
This seems to be resolved.
from alertmanager.
hi castertap
can u post the dockerfile for above prometheus version
from alertmanager.
Related Issues (20)
- multiple peers send alert that should only be sent once after failed notify attempt HOT 4
- Alertmanager send resolved notification when problem not solved HOT 3
- Cannot send alerts to multiple receivers simultaneously using matchers field. HOT 8
- Unable to change time zone in Grafana Alerting Notification Template (Always in UTC) HOT 1
- bad response status 404 Not Found HOT 1
- Task: Improvements to silences
- Work around PrometheusOperatorSyncFailed, integration=opsgenie, max_runes=130? HOT 1
- config.SecretURL type returns Json.Unmarshal() error which can expose the secret url. HOT 1
- amtool does not have a way to specify a time to test with. HOT 4
- Support time_interval testing with amtool config routes test HOT 2
- AlertManager doesn't send alert to Slack HOT 1
- Docker image prom/alertmanager:v0.27.0 fails to parse webhook_url_file from msteams_config HOT 2
- An option to disable the default sorting of Alerts for a group of Alerts HOT 6
- AlertManager doesn't send some of alerts to Pagerduty HOT 3
- Alertmanager's template obtains multiple metrics HOT 1
- Silence with unused label matching All Alerts HOT 2
- matcher is not native regexp why? HOT 2
- Feature Request: Matrix Receiver
- Microsoft deprecating Teams webhooks, new "Workflows" uses incompatible schema HOT 34
- Discord webhook : Cannot send an empty message HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from alertmanager.