A Docker, LXC, database-free, lightweight container performance monitoring solution, perfect for ephemeral containers (e.g. containers used for DevOps automatic testing). Can also be used with InfluxDB and Grafana to monitor long-lived containers in real-time.
The project is composed by 2 tools:
- a lightweight agent (80KB, native binary; no JVM, Python or other interpreters needed) to collect actual CPU/memory/disk statistics (Linux-only) and store them in a JSON file;
- a simple Python script to convert the generated JSON to a self-contained HTML page.
The agent is actually a cgroup-aware statistics collector; cgroups (i.e. Linux Control Groups) are the basic technology used to create containers (you can read more on them here); this project is thus aimed at monitoring your LXC/Docker container performances but can equally monitor physical servers.
This project supports only Linux x86_64 architectures.
Table of contents of this README:
- Features
- Yet-Another-Monitoring-Project?
- How to install
- How to use
- Project History
- License
This project collects performance data about:
- per-CPU-core usage;
- memory usage;
- network traffic (PPS and MB/s or Mbps);
- disk load;
- average Linux load;
- CPU usage as reported by the
cpuacct(CPU accounting) cgroup; - memory usage as reported by the
memorycgroup; - disk usage as reported by the
blkiocgroup;
Moreover the project allows you to easily post-process collected data and produce a self-contained HTML page which allows to visualize all the performance data easily using Google Charts.
You may be thinking "yet another monitoring project" for containers. Indeed there are already quite a few open source solutions, e.g.:
All these are very complete solutions that allow you to monitor swarms of containers, in real time. The downside is that all these projects require you to setup an infrastructure (usually a time-series database) that collects in real-time all the statistics and then have some powerful web platform (e.g. Graphana) to render those time-series. All this is fantastic for persistent containers.
This project instead is focused on providing a database-free, lightweight container performance monitoring solution, perfect for ephemeral containers (e.g. containers used for DevOps automatic testing). The idea is much simpler:
- you collect data for your container (or, well, your physical server) using a small collector software (written in C++ to avoid Java virtual machines, Python interpreters or the like!) that saves data on disk in JSON format;
- you save the JSON file, convert it to a self-contained HTML page;
- you can archive that HTML file, send it by email, put in a tarball or whatever you like the most: no dependencies at all are required to visualize it later!
If you use an LXC/Docker container based on a Centos/RHEL/Fedora distribution you can log into the container (or change its Dockerfile) and just install the RPM right away from the COPR repository:
yum install -y yum-plugin-copr
yum copr enable -y f18m/cmonitor
yum install -y cmonitor
If you use an LXC/Docker container based on a Ubuntu distribution you can similarly install from my Ubuntu PPA using the following commands:
add-apt-repository ppa:francesco-montorsi/cmonitor
apt-get install cmonitor
If you want to simply use a out-of-the-box Docker container to monitor your baremetal performances you can run:
docker pull f18m/cmonitor
which downloads the Docker image for this project from Docker Hub. See below for examples on how to run the Docker image.
The RPM installs a single utility, cmonitor_collector inside your container; launch it like that:
cmonitor_collector -s 3 -m /home
to produce in the /home folder a JSON with CPU/memory/disk/network stats for the container
sampling all supported performance statistics every 3 seconds.
Whenever you want you can either:
- inject that JSON inside InfluxDB (mostly useful for persistent containers that you want to monitor in real-time); see section "Connecting with InfluxDB and Grafana" below;
- or use the
cmonitor_chartutility to convert that JSON into a self-contained HTML file (mostly useful for ephemeral containers); see below for practical examples.
To plot the JSON containing the collected statistics, simply launch the cmonitor_chart utility installed together
with the RPM/Debian package, with the JSON collected from cmonitor_collector:
cmonitor_chart --input=/path/to/json-stats.json --output=<optional HTML output filename>
Note that to save space/bandwidth you can also gzip the JSON file and pass it gzipped directly to cmonitor_chart.
In this case you can simply install cmonitor as RPM or APT package following instructions in How to install and then launch the cmonitor collector as any other Linux daemon. Example results:
-
baremetal1: example of graph generated with the performance stats collected from a physical (baremetal) server running Ubuntu 18.04; note that despite the absence of any kind of container, the
cmonitor_collectorutility (like just any other software in modern Linux distributions) was running inside the default "user.slice" cgroup and collected both the stats of that cgroup and all baremetal stats (which in this case mostly coincide since the "user.slice" cgroup contains almost all running processes of the server); -
baremetal2: This is a longer example of collected statistics (results in a larger file, may take some time to download) generated with 9 hours of performance stats collected from a physical server running Centos7 and with 56 CPUs (!!); the
cmonitor_collectorutility was running inside the default "user.slice" cgroup so both "CGroup" and "Baremetal" graphs are present;
In this case you can install cmonitor Docker using the official DockerHub image, see Docker; the Docker container will collect all performance stats of the baremetal. Just run it
docker run -d \
--name=cmonitor-baremetal-collector
-v /root:/perf \
f18m/cmonitor
Example results:
- docker_collecting_baremetal_stats: example of graph generated with the performance stats collected from a physical server from inside a Docker container; in this case cgroup stat collection was explicitely disabled so that only baremetal performance graphs are present;
If you can easily modify the Dockerfile of your container, you can embed cmonitor so that it runs inside your container and monitor your dockerized-application. Example of the Dockerfile modified for this purpose:
...
COPY cmonitor_collector /usr/bin/ # first you need to grabthe cmonitor binary for your Docker base image
CMD /usr/bin/cmonitor_collector \
--sampling-interval=3 \
--output-filename=mycontainer.json \
--output-directory /perf ; \
myapplication
...
Example results:
- docker_userapp_with_embedded_collector:
example of graph generated with the performance stats collected from inside a Docker container using Ubuntu as base image; this is a practical example
where the Docker container is actually deploying something that simulates your target application, together with an embedded
cmonitor_collectorinstance that monitors the performance of the Docker container itself; in this case both cgroup stats and baremetal performance graphs are present.
In this case you can simply install cmonitor as RPM or APT package following instructions in How to install and then launch the cmonitor collector as any other Linux daemon, specifying the name of the container it should monitor, e.g.
docker run --name userapp myuserapp-docker-image
# here we exploit the following fact: the cgroup of each Docker container
# is always named 'docker/container-ID'
cmonitor_collector \
--sampling-interval=3 \
--output-filename docker-userapp.json \
--cgroup-name=docker/$(docker ps --no-trunc -aqf name=userapp)
Example results:
- docker_userapp: example of the chart generated by monitoring from the baremetal a simple docker simulating your application, doing some CPU and I/O
The cmonitor_collector can be connected to an InfluxDB deployment to store collected data (this can happen
in parallel to the JSON default storage). This can be done by simply providing the IP and port of the InfluxDB instance when launching
the collector:
cmonitor_collector --remote-ip=1.2.3.4 --remote-port=8086
The InfluxDB instance can then be used as data source for graphing tools like Grafana which allow you to create nice interactive dashboards like the following one:
You can also play with the live dashboard example
To setup easily and quickly the chain "cmonitor_collector-InfluxDB-Grafana" you can checkout the repo of this project and run:
make -C examples regen_grafana_screenshots
which uses Docker files to deploy a temporary setup and fill the InfluxDB with 10minutes of data collected from the baremetal.
This project started as a fork of Nigel's performance Monitor for Linux, adding cgroup-awareness;
but it has quickly evolved to a point where it shares very little code pieces with the original njmon tool.
Some key differences now include:
- cgroup-aware: several performance cgroup stats are collected by
cmonitor_collectorand plotted bycmonitor_chart - more command-line options for
cmonitor_collector; - HTML page generated by
cmonitor_chartdifferently organized; cmonitor_collectoris able to connect to InfluxDB directly and does not need intermediate Python scripts to transform from JSON streamed data to InfluxDB-compatible stream.
This fork supports only Linux x86_64 architectures; support for AIX/PowerPC (present in original nmon) has been dropped.
- Original project: http://nmon.sourceforge.net
- Other forks: https://github.com/axibase/nmon
Just like the original project, this project is licensed under GNU GPL 2.0.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent