kumina / libvirt_exporter Goto Github PK

Prometheus metrics exporter for libvirt.

License: Apache License 2.0

Go 89.07% Shell 4.97% Dockerfile 5.95%

libvirt_exporter's Introduction

Prometheus libvirt exporter

Please note: This repository is currently unmaintained. Due to insufficient time and not using the exporter anymore we decided to archive this project.

This repository provides code for a Prometheus metrics exporter for libvirt. This exporter connects to any libvirt daemon and exports per-domain metrics related to CPU, memory, disk and network usage. By default, this exporter listens on TCP port 9177.

This exporter makes use of libvirt-go, the official Go bindings for libvirt. Ideally, this exporter should make use of the GetAllDomainStats() API call to extract all relevant metrics. Unfortunately, we at Kumina still need this exporter to be compatible with older versions of libvirt that don't support this API call.

The following metrics/labels are being exported:

libvirt_domain_block_stats_read_bytes_total{domain="...",uuid="...",source_file="...",target_device="..."}
libvirt_domain_block_stats_read_requests_total{domain="...",uuid="...",source_file="...",target_device="..."}
libvirt_domain_block_stats_write_bytes_total{domain="...",uuid="...",source_file="...",target_device="..."}
libvirt_domain_block_stats_write_requests_total{domain="...",uuid="...",source_file="...",target_device="..."}
libvirt_domain_info_cpu_time_seconds_total{domain="...",uuid="..."}
libvirt_domain_info_maximum_memory_bytes{domain="...",uuid="..."}
libvirt_domain_info_memory_usage_bytes{domain="...",uuid="..."}
libvirt_domain_info_virtual_cpus{domain="...",uuid="..."}
libvirt_domain_interface_stats_receive_bytes_total{domain="...",uuid="...",source_bridge="...",target_device="..."}
libvirt_domain_interface_stats_receive_drops_total{domain="...",uuid="...",source_bridge="...",target_device="..."}
libvirt_domain_interface_stats_receive_errors_total{domain="...",uuid="...",source_bridge="...",target_device="..."}
libvirt_domain_interface_stats_receive_packets_total{domain="...",uuid="...",source_bridge="...",target_device="..."}
libvirt_domain_interface_stats_transmit_bytes_total{domain="...",uuid="...",source_bridge="...",target_device="..."}
libvirt_domain_interface_stats_transmit_drops_total{domain="...",uuid="...",source_bridge="...",target_device="..."}
libvirt_domain_interface_stats_transmit_errors_total{domain="...",uuid="...",source_bridge="...",target_device="..."}
libvirt_domain_interface_stats_transmit_packets_total{domain="...",uuid="...",source_bridge="...",target_device="..."}
libvirt_up

With the --libvirt.export-nova-metadata flag, it will export the following additional OpenStack-specific labels for every domain:

name
flavor
project_name

At Kumina we want to perform a single build of this exporter, deploying it to a variety of Linux distribution versions. This is why this repository contains a shell script, build_static.sh, that builds a statically linked copy of this exporter in an Alpine Linux based container.

libvirt_exporter's People

Contributors

Stargazers

Watchers

libvirt_exporter's Issues

The memory usage capture incorrect

I am trying to get the memory usage by "libvirt_domain_info_memory_usage_bytes" metric. But the value is constant.
I guess you are returning the actual memory from libvirt API which I can see using virsh dommemstat.

cannot use s.objectives (type map[float64]float64) as type float64 in argument to quantile.NewTargeted

Get that error when running the exporter:

libvirt_exporter]# go run libvirt_exporter.go

github.com/prometheus/client_golang/prometheus

../../prometheus/client_golang/prometheus/summary.go:330:31: cannot use s.objectives (type map[float64]float64) as type float64 in argument to quantile.NewTargeted

Causes HTTP requests to fail:

curl http://127.0.0.1:9177

curl: (7) Failed connect to 127.0.0.1:9177; Connection refused

Ability to configure additional labels

Nice project!

For us I'm looking for the ability to add more labels based on the <metadata> or <sysinfo> parts of the dom xml.

This is in my case to extract additional (openstack) information like instance uuid, flavor, name, creation date, project + user uuid etc.

Maybe I will implement this if the patch is welcome.

The captured memory data is inaccurate？

GitHub Releases Proposal

Hi there!

I'm working on getting this tool running in a Linux distribution called Clear Linux. It's a very lightweight distribution that has a package manager based around "bundles". I'd like to add this project to one of the supported bundles of that OS, but they prefer that projects included in their packages support GitHub Releases as a way of signifying stable releases of the software.

Is this something that you would all be open to? Here is an example of how releases can be made on GitHub.

Thank you for the consideration! 🙇

How should I run the docker image?

Hi all,

I just created a docker image from Dockerfile, but I don't know how should I run the docker image.
Would you please help me to run the docker image?

GetAllDomainStats build tag?

Hi! I'm need to expose libvirt metrics and want to use get all domain stats becasue i have more than 100 domains on each host and this greatly speedup things.
What do you think about providing build tag to utilize never functions?

expose domain state?

I'm use handmade script to detect crushed domains, does it possible to expose domain state with this exporter? So i can watch libvirt domain state ?

multi-second response times

Hi,

we noticed that libvirt_exporter will need more than 20 seconds for some requests.

When getting some of the stats via virsh directly I noticed some requests taking more time as well, so we decided to reduce our scrape rate because if this, worrying about the impact on libvirtd (unknown to me)

This is probably normal, just wanted to know if you experience the same?

http_request_duration_microseconds{job="libvirt_exporter",quantile="0.9"}

rate(http_requests_total{job="libvirt_exporter"}[5m])

Build error

Hi! While building from build_static.sh I'm receiving error

+ sed -i 's/^Libs:.*/& -lnl -ltirpc -lxml2/' /usr/local/lib/pkgconfig/libvirt.pc
+ cd /gopath/src/github.com/kumina/libvirt_exporter
+ export 'GOPATH=/gopath'
+ go get -d ./...
package github.com/kumina/libvirt_exporter/...: /gopath/src/github.com/kumina/libvirt_exporter exists but /gopath/src/github.com/kumina/libvirt_exporter/.git does not - stale checkout?

How can I solve it?

When a VM live migration is occuring the source hypervisor's libvirt-exporter errors on querying libvirt

While a live-migration is occuring The monitoring state is locked by remoteDispatchDomainMigratePrepare3Params.

The log below is shown:

Failed to scrape metrics: virError(Code=68, Domain=10, Message='Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainMigratePrepare3Params)')

I would expect that when a domain's state is locked due to migration, that the /metrics endpoint would stay available on the host and serve up cached data for that instance and update the remaining domains.

libvirt_domain_info_cpu_time_seconds_total not increasing

Between subsequent scrapes, values of this metric are not often increasing (at least for some VMs). For example, I have a VM where over 3 hours the counter value only changed 3 times!!

I notice the same thing when I run virst dominfo. CPU time doesn't change very often.
However, if I use virsh domstats, cpu.time increases every time.

Would it be possible to use the stats API instead of the info API
Thanks

libvirt_domain_info_memory_usage_bytes

Hi. I don't know 'go', but use your exporter.
Collecting information about the used memory of the virtual machine is more correct to take from the " domain.memorystats.useMemory". When i use virsh dominfo, max and used memory is equal,
but I have truthful stats in virsh dommemstats.

Latency disk for Domain

Some metric about latency of disks not exported:
RdTotalTimes
WrTotalTimes
FlushTotalTimes

I checked in libvirt-go in function BlockStats, these above variable not returned.

Please help me to export these metrics!

Support for "hostdev" / VFIO devices

Hi,

great work with this exporter!

Unfortunately, I've hit a bug. When the exporter finds in domain XML definition of a device without target - for example, VFIO interfaces which can be detected as "hostdev" devices - it fails with error:

libvirt_exporter[20034]: 2017/11/24 10:22:11 Failed to scrape metrics: virError(Code=8, Domain=10, Message='invalid argument: invalid path, '' is not a known interface')

and the exporter stops the processing.

Expected behavior - warn about lack of device and ignore the interface (or gather stats in some different way if possible).

I believe this is the fragment of our domain xml file that causes this issue:

<interface type='hostdev' managed='yes'>
  <mac address='AA:BB:CC:DD:EE:FF'/>
  <driver name='vfio'/>
  <source>
    <address type='pci' domain='0x0000' bus='0x04' slot='0x10' function='0x4'/>
  </source>
  <model type='virtio'/>
  <alias name='hostdev0'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</interface>

[request] Grafana dashboard

Might I ask you to share some working Grafana dashboard for this exporter?

Add binary release

It would make sense to have the binary artifact directly in the release.

Additional metrics that could be exported

There are some more metrics that are interesting to us:

Maybe we create a config format for this?

https://libvirt.org/html/libvirt-libvirt-domain.html#VIR_DOMAIN_STATS_BLOCK

  "block.<num>.rd.times" - total time (ns) spent on reads as
                           unsigned long long.
  "block.<num>.wr.times" - total time (ns) spent on writes as
                           unsigned long long.
  "block.<num>.fl.reqs" - total flush requests as unsigned long long.
  "block.<num>.fl.times" - total time (ns) spent on cache flushing as
                           unsigned long long.
  "block.<num>.capacity" - logical size in bytes of the block device
                           backing image as unsigned long long.
  "block.<num>.physical" - physical size in bytes of the container of the
                           backing image as unsigned long long.

also some of the VIR_DOMAIN_STATS_PERF look quite interesting to us.

Still maintained?

Is this project still maintained?

Exclude or merge values

Hi,

I'm monitoring a VM which has during this time been stopped and restarted, in the process it has got a new UUID and now my queries are ending up with two sets of values. See below:
https://pastebin.com/raw/bkmgWvfY

Do you know of a way to make Prometheus combine or ignore the uuid values, so it actually gives me the combined added values from all entries (since it's basically the same VM)?

Thanks

error: '_PATH_MOUNTED' undeclared

I'm trying to build the Docker image but it fails when compiling libvirt 3.2.0

$ docker build -t kumina-libvirt-exporter .
...
storage/storage_backend_fs.c: In function 'virStorageBackendFileSystemIsMounted':
storage/storage_backend_fs.c:306:23: error: '_PATH_MOUNTED' undeclared (first use in this function); did you mean 'XPATH_POINT'?
  306 |     if ((mtab = fopen(_PATH_MOUNTED, "r")) == NULL) {
      |                       ^~~~~~~~~~~~~
      |                       XPATH_POINT
storage/storage_backend_fs.c:306:23: note: each undeclared identifier is reported only once for each function it appears in
make[3]: *** [Makefile:10174: storage/libvirt_storage_backend_fs_la-storage_backend_fs.lo] Error 1
make[3]: *** Waiting for unfinished jobs....
gcc: warning: switch '-Wchkp' is no longer supported
make[3]: Leaving directory '/tmp/libvirt-3.2.0/src'
make[2]: *** [Makefile:5941: all] Error 2
make[2]: Leaving directory '/tmp/libvirt-3.2.0/src'
make[1]: Leaving directory '/tmp/libvirt-3.2.0'
make[1]: *** [Makefile:2084: all-recursive] Error 1
make: *** [Makefile:1981: all] Error 2
The command '/bin/sh -c ./configure --disable-shared --enable-static --localstatedir=/var --without-storage-mpath &&     make -j2 &&     make install &&     sed -i 's/^Libs:.*/& -lnl -ltirpc -lxml2/' /usr/local/lib/pkgconfig/libvirt.pc' returned a non-zero code: 2
$

Any idea how to fix this?

Please tag a new release

I'm trying to build this. It seems I need:

c774db1

Which is not currently in a tagged release.

ERROR:kolla.common.utils.prometheus-libvirt-exporter:Error'd with the following message
ERROR:kolla.common.utils.prometheus-libvirt-exporter:The command '/bin/sh -c curl -sSL -o /tmp/libvirt_exporter.tar.gz ${prometheus_libvirt_exporter_url}     && mkdir /tmp/l
ibvirt_exporter     && tar --strip 1 -xvf /tmp/libvirt_exporter.tar.gz -C /tmp/libvirt_exporter     && cd /tmp/libvirt_exporter     && go get -d ./...     && go build     &&
 install -m 0755 libvirt_exporter /opt/     && rm -rf /tmp/libvirt_exporter*' returned a non-zero code: 2

INFO:kolla.common.utils.prometheus-libvirt-exporter:ESC[91m# _/tmp/libvirt_exporter
INFO:kolla.common.utils.prometheus-libvirt-exporter:./libvirt_exporter.go:451:28: undefined: prometheus.Handler

Dealing with unsupported API calls

Not all libvirt drivers expose the API calls used by the exporter, and the exporter should handle those cases gracefully.

Steps to reproduce

Using libvirt_exporter with the bhyve driver using libvirt_exporter -libvirt.uri bhyve:///system.
visit http://hostname:9177

Expected behaviour

Metrics not exposed by libvirt due to non-supported API calls are not exported.

Actual behaviour

Only some metrics of only the first domain are exposed.
The exporter's stdout states the following error message:

2018/08/27 20:23:22 Failed to scrape metrics: virError(Code=3, Domain=20, Message='this function is not supported by the connection driver: virDomainBlockStats')