monitor-promdiscovery

Overview

The monitor-promdiscovery tool generate file-based service discovery files for Prometheus, for the monitor-exporter and icinga2-exporter. The tool is typically run from cron or equivalent tool to check OP5 Monitor or Icinga2 for hosts service that should be scraped for performance metrics by the exporters.
For a host to be detected, it must be a part of hostgroup(s) or servicegroup(s) that is defined in the configuration file. For a servicegroup(s) configuration all hosts that has at least on service in the configured servicegroup(s) will be added.

Servicegroups configuration is not supported for icinga2

Flow

monitor-promdiscovery is executed at regular intervals, e.g. every minute, by cron.
monitor-promdiscovery performs a filter query against the configured Monitor instance for all hosts that are part of a specific hostgroup. All hosts in the hostgroup(s) will be scraped for all their services' performance data monitor-exporter or icinga2-exporter.
monitor-promdiscovery checks against the configured file-based discovery if any host from the above call is new or not existing anymore.
Only if there is not a match the file-based discovery file will be updated
If the file is updated, Prometheus will reload the configuration file.

Running

Only use configuration file

python -m monitor_promdiscovery -f ./config.yml

Override system property in the configuration file

python -m monitor_promdiscovery -f ./config.yml -s <system>

Where can be either icinga2 or monitor

Installing

Check out / clone the git repo.
Install dependencies

pip install -r requirements.txt
Build a distribution

python setup.py sdist
Install locally

pip install dist/monitor-promdiscovery-X.Y.Z.tar.gz

Configuration

The monitor-promdiscovery configuration file should be rather self-explanatory.

NB! Only icinga2 or op5monitor has to be configured. If both are used, use the -s switch instead of configure system in the property file. Using the below config executing without specifiying -s , the icinga2 entry will be used.

# The system can be either op5monitor or icinga2 
system: icinga2

op5monitor:  
  url: https://monitor.local  
  user: monitor  
  passwd: monitor  
  # Connection timeout - default 5
  # timeout: 5
  # Verify ssl - default False
  # verify: False

  # The hostgroup(s) used to select hosts as targets  
  hostgroup: 
    - prometheus
  # The servicegroup(s) used to select hosts as targets
  # Will include all hosts that has at least on service in the configured servicegroup(s)
  servicegroup:
    - servicepoint
  prometheus:  
    # path where to create the file-based discovery file - must be set  
    sd_file: <sd_directory>/monitor_sd.yml  
    # Additional labels to tag metrics with - optional
    labels:  
      source: monitor  
      env: prod

icinga2:
  url: https://127.0.0.1:5665
  user: root
  passwd: cf593406ffcfd2ef
  # Connection timeout - default 5
  # timeout: 5
  # Verify ssl - default False
  # verify: False

  # The hostgroup used to select hosts as targets
  hostgroup: 
    - prometheus
    - linux_servers
  prometheus:
    # path where to create the file-based discovery file - must be set
    sd_file: /home/andersh/programs/prometheus/sd/icinga2_sd.yml
    # Additional labels to tag metrics with - optional
    labels:
      source: icinga2
      env: prod

logger:
  # Path and name for the log file. If not set sent to stdout
  logfile: monitor-promdiscovery.log
  # Format if day will have current day as post fix
  # format: day
  # Log level
  #level: INFO

The hostgroup property can be both a single value or a list of different hostgroups as shown above.

Prometheus configuration

The Prometheus configuration file would typical have the following job configuration in the scrape_configs section for the monitor-exporter:

  - job_name: 'op5'
    scrape_interval: 2m
    metrics_path: /metrics
    file_sd_configs:
    - files:
      - 'sd/monitor_sd.yml'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        # The address for the monitor-exporter
        replacement: localhost:5000

And the format for icinga2 is almost the same:

  - job_name: 'icinga2'
    scrape_interval: 2m
    metrics_path: /metrics
    file_sd_configs:
    - files:
      - 'sd/icinga2_sd.yml'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        # The address for the monitor-exporter
        replacement: localhost:5000

The file_sd_configs must have the same filename as created by the tool, as defined in the configuration file.

System requirement

Python 3.6

For required packages, please review requirements.txt

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-req-build-c7s69ktb/requirements.txt'

Since commit 1bfa206, I am seeing the following error when I attempt to build the latest version of monitor-promdiscovery

Processing ./dist/monitor-promdiscovery-0.0.4.post4+git.1bfa2065.dirty.tar.gz
    ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-c7s69ktb/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-c7s69ktb/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(cod
e, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-ofunht7m
         cwd: /tmp/pip-req-build-c7s69ktb/
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-c7s69ktb/setup.py", line 30, in <module>
        install_requires=read('requirements.txt').split(),
      File "/tmp/pip-req-build-c7s69ktb/setup.py", line 6, in read
        return open(join(dirname(__file__), fname)).read()
    FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-req-build-c7s69ktb/requirements.txt'

To reproduce, you can launch a Docker instance of Python:

docker run -it --rm python:3.9 /bin/sh

and attempt to build using these steps:

git clone https://github.com/opsdis/monitor-promdiscovery.git /opt/monitor-promdiscovery
cd /opt/monitor-promdiscovery
pip3 install -r requirements.txt
python3 setup.py sdist
pip3 install dist/monitor-promdiscovery*

opsdis / monitor-promdiscovery Goto Github PK

monitor-promdiscovery's Introduction

monitor-promdiscovery

Overview

Flow

Running

Only use configuration file

Override system property in the configuration file

Installing

Configuration

Prometheus configuration

System requirement

monitor-promdiscovery's People

Contributors

Stargazers

Watchers

Forkers

monitor-promdiscovery's Issues

Recommend Projects

Recommend Topics

Recommend Org