labbsr0x / bb-promster Goto Github PK

Remove restrictions that only allow alertmanager to manage alerts on level 1 of the federation.

BB Promster should be able to deal with canary deployment comparison rules.

To be able to do that, we'll leverage ETCD to register which is the pilot version and which is the prod version. BB Promster's alerting "sidecar" will then reach for such configs and load new template based alerting rules.

The sidecar should be installed in BB Promster's docker image by leveraging Docker's multi-stage build capabilities.

Issue #15 creates a mechanism to update alert rules and also shows an example of a comparative alert using the error rate metric. Create more examples of comparative alerts using latency and traffic metrics would be interesting.

Bb-promster is being used in production for a while. Before bb-promster, scrape targets were set statically on building run-time. bb-promster introduces an ETCD database to manage all IPs that Prometheus needs to monitor, speeding up the targets update process.
However, it has some issues to update some attributes that belong to a target. Currently, bb-promster only watch changes on IPs, but on real monitoring, it's also necessary to keep tracking of scrape path, relabel_configs, and SSL certificate. It could be interesting to allow bb-promster to watch changes on other attributes, so the need to modify the deploy will be minimum.
Considering that, we propose bb-promster not watch just IPs, but a complete app object as bb-bot does.

App structure will be responsible for keeping data related to a context being monitored, attributes like name, scrapePath, and so on. App also has an internal struct called Env to manage contexts with blue/green deployment.
Settings is an auxiliary structure to keep data related to all bb-promster, as the remote write URL and etcd service where apps are saved.
It is an initial proposal that can also include the bb-promsters status or pod names in order to centralize information for operation teams.

Thanks to @KarineValenca for helping me with the text.

The current configuration of bb-promster uses the same scheme of Prometheus at Alertmanager config:

alerting:
  alertmanagers:
  - scheme: {{.scheme}}
    static_configs:
    - targets: ['$ALERT_MANAGER_URLS']

It may be necessary to change this configuration to use different schemes.

In order to interact to telegram bot, the alertmanager rules must have an annotation app. The annotation will be used to search all subscriptions. The follow example shows how to define it:

 - alert: dependency_unresponsive
    expr: dependency_up < 1
    for: 2m
    annotations:
      app: {{$labels.prsn}}
      description: A dependência {{$labels.name}} do serviço {{$labels.prsn}} não está respondendo.
      summary: Alerta para caso uma dependência passe mais de 2 minutos sem resposta

We should provide a cortex example

Prometheus remote write config can add two types of Authorization headers: basic auth e bearer token.
As described on the documentation:

# The URL of the endpoint to send samples to.
url: <string>

# Timeout for requests to the remote write endpoint.
[ remote_timeout: <duration> | default = 30s ]

# List of remote write relabel configurations.
write_relabel_configs:
  [ - <relabel_config> ... ]

# Sets the `Authorization` header on every remote write request with the
# configured username and password.
# password and password_file are mutually exclusive.
basic_auth:
  [ username: <string> ]
  [ password: <string> ]
  [ password_file: <string> ]

# Sets the `Authorization` header on every remote write request with
# the configured bearer token. It is mutually exclusive with `bearer_token_file`.
[ bearer_token: <string> ]

# Sets the `Authorization` header on every remote write request with the bearer token
# read from the configured file. It is mutually exclusive with `bearer_token`.
[ bearer_token_file: /path/to/bearer/token/file ]

The run.sh script, on bb-promster, just add the the remote_url as the code shows:

# the user can also configure a remote url to send prometheus metrics to 
if [[ "$REMOTE_WRITE_URL" != "" ]]; then
    cat >> "/prometheus.yml.tmpl" <<- EOM
remote_write:
  - url: $REMOTE_WRITE_URL
EOM
fi

In order to support others prometheus' remote config, I propose adding more two env variables: REMOTE_BEARER_TOKEN and REMOTE_BASIC_AUTH. So the run.sh script can add the config if one of them is set.

It's necessary to add more labels to comparatives alerts to give more information about the alert to the user. Useful labels:
prsn -> informs the App in which the alarm is firing. Also, this label is mandatory in order to bb-bot send the alert to Telegram.
addr -> inform the URL in which the alarm is firing.

After a container/pod restart, when there's /versions registered in ETCD, the bb-promster container breaks.
The problem happens due to the main.go file, that takes care of updating alert rules files. It needs further investigation to find out why the problem is happening.

Add the alert rules from big-brother monitor and create an option to user activate these alerts or not.

Em todos níveis deveriamos ter a liberdade de configurar regras de alertas e endereço do alertmanager.

The new functionality should be documented on BB Promster's readme.

Add cortex tag on docker-compose in cortex folder.