Git Product home page Git Product logo

conntrack-stats-exporter's Introduction

Conntrack Statistics Exporter

The well known prometheus node exporter exports conntrack metrics off the /proc pseudo file system. The conntrack module developers consider that surface as deprecated and provide a CLI tool conntrack that shows some interesting metrics.

Motivation for this exporter was to survey insert_failed statistics due to a race condition in the Linux ipfilter conntrack kernel code. This is a subtle bug that in some circumstances escalates in high workload scenarios in Kubernetes clusters and causes drop of initial packets of NATted connections (both UDP, TCP.) The insert_failed statistic correlates with dropped connections due to this bug.

Helm Chart

See Prometheus Community Charts. Kudos to @monotek!

Further information about the conntrack race bug and its effect on Kubernetes

conntrack-stats-exporter's People

Contributors

dependabot[bot] avatar fdomain avatar jwkohnen avatar monotek avatar platinumthinker avatar samartinucci avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar

Forkers

cmattoon platinumthinker travelping raffo delta50 matthope nanorobocop gaffneyd4 thatva samartinucci mchumakov monotek criteo-forks evertonsa eletenkov

conntrack-stats-exporter's Issues

v0.4.4 has no metrics in k8s

In kubernetes (azure aks 1.23) with image v0.4.4 i get the following erros in the pod logs:

k logs -f conntrack-stats-exporter-wtg6p
listening on :9371 with endpoint "/metrics"
error gathering metrics for netns "": op(tool_output_no_match): <nil>
....

Metrics are missing too:

# HELP conntrack_stats_scrape_error Total of error when calling/parsing conntrack command
# TYPE conntrack_stats_scrape_error counter
conntrack_stats_scrape_error{netns="",cause="timeout"} 0
conntrack_stats_scrape_error{netns="",cause="client_gone"} 0
conntrack_stats_scrape_error{netns="",cause="netns_restore"} 0
conntrack_stats_scrape_error{netns="",cause="netns_enter"} 0
conntrack_stats_scrape_error{netns="",cause="netns_cleanup"} 0
conntrack_stats_scrape_error{netns="",cause="netns_prepare"} 0
conntrack_stats_scrape_error{netns="",cause="tool_exec"} 8
conntrack_stats_scrape_error{netns="",cause="tool_output_no_match"} 0

Last working image in k8s is v0.3.2 for me.

Locally the newer v0.4.4 image works via:

docker run -it --net=host --privileged --rm jwkohnen/conntrack-stats-exporter:v0.4.4

imho it broke when the support for network namespaces was introduced.

[Feature Request] show remote endpoint for each connection on the server.

Hello! Thanks for conntrack-stats-exporter.

For example:

# HELP conntrack_open_connections How many open connections are there to the remote host
# TYPE conntrack_open_connections gauge
conntrack_open_connections{host="10.0.1.5:3306"} 49
conntrack_open_connections{host="10.0.1.12:8080"} 19

error gathering metrics for netns "": op(tool_exec): failed to exec conntrack tool: error running the conntrack command with the --stats flag: exit status 1

Hello! Thanks for conntrack-stats-exporter!
I builb conntrack-stats-exporter in ubuntu 22.04 and simple run.
Get error

./conntrack-stats-exporter 
listening on :9371 with endpoint "/metrics"
error gathering metrics for netns "": op(tool_exec): failed to exec conntrack tool: error running the conntrack command with the --stats flag: exit status 1

Kernel

uname -r
5.15.0-56-generic

Golang

go version
go version go1.18.1 linux/amd64

Rely on a Netlink go implementation instead of conntrack command calls

This is a proposal to replace usage of conntrack tool to fetch statistics on the conntrack table, with a golang lib able to bring this information directly over the Netlink socket. The main benefit would be to get rid of the conntrack's output parsing.

There is already a lib allowing to do that: https://github.com/ti-mo/conntrack
I have used it for another project and it seems pretty easy to get stats and have the result in a proper golang struct.

I can work on this but I'd like to have your opinion before rushing into it.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.