prymitive / karma Goto Github PK

We authenticate using nginx auth_request.
When are using proxy mode in the configuration file.

When submitting a new silence, cookies are not sent with the POST, resulting in a 401.

Seems to be related to #24 ?

It would be great if we could specify the path to the configuration file in a single command flag (instead of using two), e.g.:

$ karma --config.file /etc/karma/karma.yaml

We've also noticed that karma is appending a .yaml suffix to the file name we provide, which is a bit counter-intuitive, especially with yaml as some prefer .yml (like in our case 😄) while others use .yaml

Again, thanks!!!

See video on https://www.dropbox.com/s/lfyr7jv8n6dwzur/Screen%20Recording%202019-05-22%20at%2010.34.26.mov?dl=0

Running with chart v1.1.13 which is karma 0.34: https://github.com/helm/charts/blob/master/stable/karma/values.yaml#L9

We use Karma on top of an AlertManager high availability cluster (with 2 or more AM). We use filter on labels. Sometimes no alerts rendering on Karma.
In this example, the number on top indicate that no alert match 'pf="sdr"' but in AM we can show the alerts.

We don't see error in Karma logs

time="2019-07-09T07:06:46Z" level=info msg="Pulling latest alerts and silences from Alertmanager"
time="2019-07-09T07:06:46Z" level=info msg="[sdr] Collecting alerts and silences"
time="2019-07-09T07:06:46Z" level=info msg="[xymon] Collecting alerts and silences"
time="2019-07-09T07:06:46Z" level=info msg="GET http://alertmanager.xymon:9093/metrics timeout=10s"
time="2019-07-09T07:06:46Z" level=info msg="GET http://alertmanager.prometheus:9093/metrics timeout=10s"
time="2019-07-09T07:06:46Z" level=info msg="Upstream version: 0.18.0"
time="2019-07-09T07:06:46Z" level=info msg="GET http://alertmanager.prometheus:9093/api/v1/status timeout=10s"
time="2019-07-09T07:06:46Z" level=info msg="[sdr] Remote Alertmanager version: 0.18.0"
time="2019-07-09T07:06:46Z" level=info msg="[sdr] Got 3 silences(s) in 764.867µs"
time="2019-07-09T07:06:46Z" level=info msg="[sdr] Detecting JIRA links in silences (3)"
time="2019-07-09T07:06:46Z" level=info msg="Upstream version: 0.17.0"
time="2019-07-09T07:06:46Z" level=info msg="GET http://alertmanager.xymon:9093/api/v1/status timeout=10s"
time="2019-07-09T07:06:46Z" level=info msg="[xymon] Remote Alertmanager version: 0.17.0"
time="2019-07-09T07:06:46Z" level=info msg="[xymon] Got 0 silences(s) in 4.333911ms"
time="2019-07-09T07:06:46Z" level=info msg="[xymon] Detecting JIRA links in silences (0)"
time="2019-07-09T07:06:46Z" level=info msg="[sdr] Got 136 alert group(s) in 67.698158ms"
time="2019-07-09T07:06:46Z" level=info msg="[sdr] Deduplicating alert groups (136)"
time="2019-07-09T07:06:46Z" level=info msg="[sdr] Processing unique alert groups (56)"
time="2019-07-09T07:06:46Z" level=info msg="[sdr] Merging autocomplete data (480)"
time="2019-07-09T07:06:46Z" level=info msg="[xymon] Got 997 alert group(s) in 585.932343ms"
time="2019-07-09T07:06:46Z" level=info msg="[xymon] Deduplicating alert groups (997)"
time="2019-07-09T07:06:46Z" level=info msg="[xymon] Processing unique alert groups (838)"
time="2019-07-09T07:06:46Z" level=info msg="[xymon] Merging autocomplete data (2908)"
time="2019-07-09T07:06:46Z" level=info msg="Pull completed"

time="2019-07-09T06:59:34Z" level=info msg="[10.244.6.0 MIS] <200> GET /alerts.json?q=pf%3Dsdr&q=%40state%3Dactive took 15.087184ms"
time="2019-07-09T07:00:34Z" level=info msg="[10.244.6.0 MIS] <200> GET /alerts.json?q=pf%3Dsdr&q=%40state%3Dactive took 18.785712ms"
time="2019-07-09T07:01:35Z" level=info msg="[10.244.6.0 MIS] <200> GET /alerts.json?q=pf%3Dsdr&q=%40state%3Dactive took 15.025122ms"
time="2019-07-09T07:02:35Z" level=info msg="[10.244.6.0 MIS] <200> GET /alerts.json?q=pf%3Dsdr&q=%40state%3Dactive took 11.880314ms"
time="2019-07-09T07:03:36Z" level=info msg="[10.244.6.0 MIS] <200> GET /alerts.json?q=pf%3Dsdr&q=%40state%3Dactive took 16.137345ms"
time="2019-07-09T07:04:37Z" level=info msg="[10.244.6.0 MIS] <200> GET /alerts.json?q=pf%3Dsdr&q=%40state%3Dactive took 19.860792ms"
time="2019-07-09T07:05:37Z" level=info msg="[10.244.6.0 MIS] <200> GET /alerts.json?q=pf%3Dsdr&q=%40state%3Dactive took 18.43056ms"
time="2019-07-09T07:06:38Z" level=info msg="[10.244.6.0 MIS] <200> GET /alerts.json?q=pf%3Dsdr&q=%40state%3Dactive took 21.778087ms"
time="2019-07-09T07:07:38Z" level=info msg="[10.244.6.0 MIS] <200> GET /alerts.json?q=pf%3Dsdr&q=%40state%3Dactive took 22.852808ms"
time="2019-07-09T07:08:39Z" level=info msg="[10.244.6.0 MIS] <200> GET /alerts.json?q=pf%3Dsdr&q=%40state%3Dactive took 14.763488ms"

Did you encounter this problem ?

First of all, thank you for developing this dashboard! It is very handy for us at trivago.

Would it be possible to have a "Nightmode" or "Dark mode" setting in the dashboard?

Some of us work in a low luminosity environment and the karma dashboard is bright. It strains the eye after a while.

If the little alarm windows could have a dark gray background for example, and the window could have an even darker shade of gray in this option, that would be great! Grafana does this very well.

Do you think this would be possible? It would certainly make some of your users very happy!

I'm not really fond of the disappearing search bar, could that be made static ?

It does not seem to work.

Hi!

In new version of alertmanager API version will be changed to v2. URI /api/v1/ will no longer work. And dashboard not work - request to http://alertmanager:9093/api/v1/alerts/groups failed with 404 Not Found

docker image for test prometheus/alertmanager:master

Is there any way to configure LDAP authentication with this tool?

Our alertmanager instance comes with self-signed certs by vendor (IBM ICP).
It would be nice if you could provide a flag to skip validation of certificate.
I could not find any env var to set this which go would act on - so it seems it has to be coded as an option/flag to pass.

Get https://monitoring-prometheus-alertmanager.kube-system:9093/alertmanager/api/v1/silences: x509: certificate signed by unknown authority

For those nostalgic of the unsee colors like me I made a custom css :

    body {
      font-size: 1.03rem !important;
      background-color: #2c3e50 !important;
    }

    .bg-primary-transparent {
      background-color: #2c3e50 !important;
    }

    .flex-shrink-1 {
      text-align: center !important;
    }

    .components-label-name {
      color: #fff !important;
    }

    .components-label-bright .components-label-name {
      color: #000 !important;
    }

    .text-muted {
      color: #000 !important;
      font-weight: bold !important;
    }

    .card {
      border: 1px solid #e74c3c !important;
    }

    .card > .card-header.bg-light  {
      background-color: #e74c3c !important;
    }

    .card .card-header.bg-light .badge-danger {
      color: #e74c3c !important;
      background-color: #fff !important;
    }

    .card .card-header.bg-light .text-muted {
      color: #fff !important;
    }

    .card-footer {
      line-height: 1;
    }

    .components-label .components-label-name,
    .components-label .components-label-value,
    .components-label-with-hover time {
      font-size: 1.1em !important;
    }

alertmanager remove /api/v1/alerts/groups api since version 0.16.1 , and karma get "/api/v1/alerts/groups failed with 404 Not Found" , how to fix it?

https://sentry.io/organizations/karma-demo/issues/1106915006/?project=1279017

Happens when removing last value from the matcher value select

Not sure if I am not missing something or it is currently not possible.
I was following: #87
The creator of the issue had a point: where to view silences.

To clarify, I mean silences which do NOT match any current alert.

Use case:

1. You create a silence in preparation of maintenance (match non-existing alerts, for example whole env=production)
2. During the maintenance, the silence matches alerts you wanted to silence (no issues here)
3. The maintenance lasted less then expected so you want to delete the silence, but currently no alerts are triggered -> can't do it via karma, have to go to alertmanager directly.

Please let me know if I am missing something obvious.
And still want to thank you for your great work, really like the project.

After upgrading from Alertmanager 0.15.3 to .0.17.0, Karma 0.39 stopped working.

curling the endpoint works without any issues.

./karma-linux-amd64 --version
"v0.39"

alertmanager --version
alertmanager, version 0.17.0 (branch: non-git, revision: c7551cd)
  build date:       20190624-21:36:45
  go version:       go1.11.10

INFO[0000] [am] Configured Alertmanager source at https://alertmanager:9093 (proxied: true) 
INFO[0000] Initial Alertmanager query                   
INFO[0000] Pulling latest alerts and silences from Alertmanager 
INFO[0000] [am] Collecting alerts and silences    
INFO[0000] GET https://alertmanager:9093/metrics timeout=0s 
INFO[0000] Upstream version: 0.17.0                     
INFO[0000] GET https://alertmanager:9093/api/v1/status timeout=0s 
INFO[0000] [am] Remote Alertmanager version: 0.17.0 
ERRO[0000] [am] Get https://alertmanager:9093/api/v2/silences: context deadline exceeded```

Hey!

First of all this tool is totally dope! thanks 🙌

Small request, it would be great if you could change the Dockerfile to use ENTRYPOINT instead of CMD, this would make things much simpler in terms of passing cmd line arguments to the container, as these are set into CMD and are appended by default to ENTRYPOINT.

Thanks!

Is there any current method to mimic the inhibition rule behaviour of Alertmanager in Karma?

For example, two alert rules that fire at different thresholds, one with severity: warning, the other with severity: critical. Alertmanager is able to inhibit the notification for the warning alert if the critical alert is firing. However, Karma displays both alerts, since this is obviously what is returned by the Alertmanager API.

We authenticate to karma using apache

But now somehow with the rewrite to react, the /alerts.json is not authenticated like it was with unsee. In the console I see 401 errors on that endpoint.

Let's test basic auth credentials:

curl -v -u admin:secretPassword https://alertmanager.example.com
[...]
< HTTP/2 200

Works fine.

karma.yaml:

    alertmanager:
      interval: 1m
      servers:
      - name: example
        uri: https://admin:[email protected]

karma generates error message:

time="2019-05-08T15:01:23Z" level=info msg="GET https://admin:[email protected]/api/v1/status timeout=10s"
time="2019-05-08T15:01:23Z" level=info msg="[example] Remote Alertmanager version: 0.17.0"
time="2019-05-08T15:01:23Z" level=error msg="[example] unknown error (status 401): {resp:0xc0006c4750} "

karma:v0.35

We should be able to specify a wildcard in the custom colors dict. This allows us to set a custom color for the all the label values. Useful to change the default static color.

labels:
  color:
    custom:
      severity:
        info: "#87c4e0"
        warning: "#ffae42"
        critical: "#ff220c"
        "*": "#ffffff"

Problem : delay when loading the page

Request: When running in environments where there is no internet access, remove the dependancy to fonts.google.api. Otherwise there is always an annoying delay/timeout when loading the karma web page.

Hi, thanks for the job, we like Unsee, and we appreciate Karma for reading gain.

We have two requests if that's possible, the first is the possibility of a "Collapse All" function to minimize the current view.
The seconds requests is the possibility to colorize the card according to the severity label (or other label). We need to differentiate quickly a warning or a critial alert.

We made some test with custom.js but I thnink we can't easily achieved this feature like this.

What do you think about this?

Regards

🚨 You need to enable Continuous Integration on all branches of this repository. 🚨

To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because it uses your CI build statuses to figure out when to notify you about breaking changes.

Since we didn’t receive a CI status on the greenkeeper/initial branch, it’s possible that you don’t have CI set up yet. We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.

If you have already set up a CI for this repository, you might need to check how it’s configured. Make sure it is set to run on all new branches. If you don’t want it to run on absolutely every branch, you can whitelist branches starting with greenkeeper/.

Once you have installed and configured CI on this repository correctly, you’ll need to re-trigger Greenkeeper’s initial pull request. To do this, please delete the greenkeeper/initial branch in this repository, and then remove and re-add this repository to the Greenkeeper App’s white list on Github. You'll find this list on your repo or organization’s settings page, under Installed GitHub Apps.

Hi,

We use annotations for storing display informations which are used in alertmanagers.

We'd like to strip them completely in karma (as we can do with labels), not only hide them.

Thanks!

When clicking on an annotation the value is hidden which makes it impossible to select and copy the value.

Could the hiding feature be binded to a double click or just activated when clicking on the key of the annotation ?

I imagine that it would be helpful to add new line with the same size like search bar and show label and count in it.

Currently, except suppresed, i have 102 alerts.
i want to get quick overview of alert count across team label. for example
"team: red: 23, blue: 15, white: 12"
and so on.

I find myself clicking labels only to then edit the "key=value" entry to "key!=value".

A shortcut like Ctrl-Click would save a lot of time in filtering a busy dashboard!

We all have a starting point for our monitoring systems: In our case it is a Grafana page that has a bunch of links to dashboard/documentation, some metrics, a live log of events, etc.
When we arrive in Karma, there is no way to get back to our starting page. Having an icon and a link for going back to our monitoring start page would be very helpful. Also document where the image is located and that it can be replaced by a user provided image (so we can put our system related icon)
It is a minor enhancement, but I think it would greatly improve usability.
Keep up the good work!

alertmanager version

alertmanager, version 0.15.2 (branch: HEAD, revision: d19fae3bae451940b8470abb680cfdd59bfa7cfa) build user: root@3101e5b68a55 build date: 20180814-10:53:39 go version: go1.10.3
alertmanager has command line options --web.external-url=https://prom.ctcfin.com/am --web.route-prefix=am

karma version "v0.21"

karma config

alertmanager:
  interval: 15s
  servers:
    - name: alertmanager-headq
      uri: http://1.2.3.4:9093/am/
      timeout: 10s
      proxy: true
debug: true
labels:
  color:
    static:
      - job
    unique:
      - instance_ip
      - app_name
      - env

  keep: []
  strip: []
listen:
  address: "0.0.0.0"
  port: 6677
  prefix: /alert
log:
  config: false
  level: debug
receivers:
  keep: []
  strip: []

When I try to add a silence request in karma, the following url returns a 404 response:

https://karma.example.com/alert/proxy/alertmanager/alertmanager-idc/api/v1/silences

And when I set the proxy off, a CORS issue comes up.

Hi!

I'm trying to check karma config on ci before applying it to production.
I want to check it with live version of karma to ensure syntax is ok.
I see two approaches for that

1. launch timeout 5 karma --config.file karma/karma.yml and wait for something like

FATA[0000] yaml: unmarshal errors:
  line 18: cannot unmarshal !!str `flase` into bool 

but image build on top of gcr.io/distroless/base that does not contain timeout and even sh

2. add --configtest key to karma and exit if config is wrong

of cause i can rebuild image by myself but that is probably not the best idea

Moving from Unsee to Karma does not work on our signage system.

On one screen I get

Internal error
TypeError: Object.entries is not a function
This page will auto refresh in xxx s

The number of seconds (xxx) is changing quickly - for every second that passes it is reduced by 4 seconds. There is a graphical exclamation mark in front of Internal.

On another screen I tried Karma and it just returns a blank turquoise screen without any elements.

The browser in our signage system is a Firefox "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0". Unfortunately I can't access any developer tools to give more information.

https://sentry.io/organizations/karma-demo/issues/1058218126/?project=1279017

I set the config file for some customized settings，when I configed the label sections custom filed ,my severity label match value like P1,P2,P3, but the web UI doesn't seems to support that ,no change color for that label.

Howdy Lukasz,

Great work on this, I just found out about your work on Karma after CloudFlare. I'm evaluating it now to replace unsee in swarmstack (https://www.youtube.com/watch?v=3FpTcVnvfRg - https://github.com/swarmstack/swarmstack) - which rebuilt Stefan Prodan's swarmprom starter kit for Docker Prometheus/Grafana/Alertmanager/Unsee and adds Docker swarm node installation and ongoing management with ease of installation as a focus.

Dockerfile below works, but requires addition of the CONFIG_FILE env variable. My reading of docs/CONFIGURATION.md was that Karma would look for a file named karma.yaml in the same directory. Didn't see an existing ticket on this issue, but might be pebkac as I just found out about Karma an hour ago :)

#cat docker-compose.yml
version: "3.4"
configs:
  karma-conf:
    file: ./karma/karma.yaml

services:
  karma:
    image: lmierzwa/karma:latest
    configs:
      - source: karma-conf
        target: /karma.yaml
    deploy:
      mode: replicated
      replicas: 1
      labels:
        prometheus.enable: "true"
        prometheus.port: "8080"
        prometheus.path: "/metrics"
    environment:
      - CONFIG_FILE=/karma.yaml

Btw, strip a label named @receiver doesn't work.

My configuration below.

Thanks in advance.

Karma should relax the constraint, either via a configuration parameter or general, that an email is required for the author field. Alertmanager makes no restriction that the author field should be an email, Karma shouldn't either.

When multiple Alertmanagers are configured in Karma, they are treated as separate standalone instances, even if they are part of an HA cluster. This has the disadvantage that when creating silences, it is by default sent to all configured Alertmanagers.

By design, such HA Alertmanagers will propagate the silence via gossip, and it ends up being duplicated.

When Karma then scrapes e.g. two Alertmanagers that are part of a cluster, it sees two silences from each Alertmanager, and ends up displaying four silences. I expect that this will multiply, so 3 Alertmanagers would show 9 silences, 4 Alertmanagers would show 16, and so on.

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.

Tooltip text when hovering over an alertname is not rendering correctly on Chrome 73/Mac. It's rendering as faint black text, as seen in the image below.

Karma v0.32

Creating silences in Karma is super nice! we use it all the time, especially that we can define the same silence on multiple alert managers (in our case in different datacenters).

It would be great if we could also view and edit silences, as we need to jump to the AM UI for that now.

WDYT? @prymitive

It would be nice if we could set the background color back to which Unsee is using :)

Dependabot can't resolve your JavaScript dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Error while updating /ui/yarn.lock:
Couldn't find any versions for "scheduler" that matches "^16.6.2"

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

You can mention @dependabot in the comments below to contact the Dependabot team.

Use case is similar to the one outlined in the following document:

TooMany500StatusCodes
Served 10.7% 5xx results in the last 3 minutes!
Also 3 inhibited firing (collapsed by default)
	JanitorProcessNotKeepingUp
	UserDatabaseShardDown
	FreshnessIndexBehind

https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/edit#

See "Causes are still useful" section.

Karma v0.36
Alertmanager v0.17
Both behind nginx proxy with their domains.
After submitting silence i get error:
Access to fetch at 'https://alertmanager..../api/v2/silences' from origin 'https://karma....' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: The value of the 'Access-Control-Allow-Origin' header in the response must not be the wildcard '*' when the request's credentials mode is 'include'.

If Karma is configured with multiple alertmanager URLs, it seems that it insists on displaying a @alertmanager label for each in the alerts on the dashboard.

I've tried to strip the label, without success.

labels:
  strip:
  - "@alertmanager"

If only a single alertmanager URL is specified, the label is not shown.

Hello,
is there any way to disable silence functionality?
We want to provide an ability to see a list of alerts to our Dev teams, but we don't want to allow them to create or delete any silences.
That is why I'm looking for a way to turn it off

When using the proxy: true option, it would be useful to have the option to use an HTTP/SOCKS proxy. The use case is for an Alertmanager endpoint running in another network (or the Internet) only reachable through a corporate proxy.

Hello! Really like your project.
Its hard to see and visually sort the alerts when there are more than a few.
A good'ol sortable table would work really well.
Any plans to implement a "table view"-mode?

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Awaiting Schedule

These updates are awaiting their schedule. Click on a checkbox to get an update now.

• chore(deps): update dependency @percy/cli to v1.12.0

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• chore(deps): update dependency @testing-library/react to v13
• chore(deps): update dependency @types/react to v18
• chore(deps): update dependency @types/react-dom to v18
• fix(deps): update dependency react-idle-timer to v5
• fix(deps): update react monorepo to v18 (major) (react, react-dom)

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

This is a great tool and it really helps to see things clearer. I read about it on the Prometheus-User-Mailinglist. Thank you very much for presenting it there and moving it over here.

Following the spirit of clarifying I think the actual alertname (the value) should be more prominent than the label’s key. So in alertname: HardToRead the string HardToRead should be more prominent than the alertname.
It is good that the keys are so prominent for other labels, but for alertname less would do because one might use this to find an alert by skimming over and every alert has this label, seeing the precise letters of its key in detail is not as important compared to other labels’ keys.

I am not sure how to do this the best way, but here are a few proposals/ideas:

• left-align the label (to simplify vertical skimming)
• change the font redendering for the alertname key
  • less bold
  • less contrast
  • not render the key alertname at all

What do you think?