This is a fork from https://github.com/neptune-networks/flow-exporter. It adds support for an address_family label, as well as two extra metrics showing source and destination mac addresses.
Flow exporter is a tool that can take flow data (Netflow, sFlow, IPFIX) from Kafka and export it to Prometheus. These flow records can be helpful to visualize which autonomous systems traffic is coming from and going to.
Grafana is a great tool to visualize Prometheus data, and can be used to take the flow data and visualized as so:
An in depth guide on setting this up on a Linux-based router can be found here.
The exporter can be started with:
./flow-exporter --brokers=kafka.fqdn.com:9092 --topic=pmacct.acct --asn=15169
--addr
: Listening address (default = :9590)--brokers
: A comma separated list of Kafka brokers (with their corresponding ports) to consume flows from--topic
: The Kafka topic to consume flows from--asn
: The autonomous system number that the flows are being monitored from
Once running, you can view the data by visiting http://localhost:9590/metrics.
An example of the Prometheus metrics you can find are:
# HELP flow_receive_bytes_total Bytes received.
# TYPE flow_receive_bytes_total counter
flow_receive_bytes_total{destination_as="397143",destination_as_name="NEPTUNE-NETWORKS - Neptune Networks",hostname="border.neptunenetworks.org",source_as="10318",source_as_name="CABLEVISION S.A."} 663
# HELP flow_transmit_bytes_total Bytes transferred.
# TYPE flow_transmit_bytes_total counter
flow_transmit_bytes_total{destination_as="10318",destination_as_name="CABLEVISION S.A.",hostname="border.neptunenetworks.org",source_as="397143",source_as_name="NEPTUNE-NETWORKS - Neptune Networks"} 1137
Flow Exporter automatically finds the name of the ASN and adds it to the metric.
Flow exporter requires a Kafka topic that has events which contain the following JSON attributes:
{
"label": "bdr1.fqdn.com",
"as_src": 15169,
"as_dst": 6939,
"bytes": 52,
}
label
: The hostname of the device that the flow came fromas_src
: The ASN that originated the flowas_dst
: The ASN that the flow was destined forbytes
: The number of bytes contained in the flow
pmacct Integration
Flow Exporter works well with pmacct, a series of tools for monitoring flows in Linux. The following pmacctd configuration can be used to collect flows on Linux, enrich them with BGP ASN data, and publish them to Kafka:
!
! pmacctd configuration example
!
! Did you know CONFIG-KEYS contains the detailed list of all configuration keys
! supported by 'nfacctd' and 'pmacctd' ?
!
! debug: true
daemonize: false
pcap_interfaces_map: /etc/pmacct/interfaces.map
pmacctd_as: longest
pmacctd_net: longest
sampling_rate: 1
!
bgp_daemon: true
bgp_daemon_ip: 127.0.0.2
bgp_daemon_port: 180
bgp_daemon_max_peers: 10
bgp_agent_map: /etc/pmacct/peering_agent.map
networks_file: /etc/pmacct/networks.lst
networks_file_no_lpm: true
!
aggregate: src_host, dst_host, src_port, dst_port, src_as, dst_as, label
!
plugins: kafka
kafka_output: json
kafka_broker_host: kafka.fqdn.com
kafka_topic: pmacct.acct
kafka_refresh_time: 5
kafka_history: 5m
kafka_history_roundoff: m
And the associated configurations referenced in that file:
ifindex=100 ifname=<INTERFACE>
bgp_ip=<BGP_ROUTER_ID> ip=0.0.0.0/0
More information on configuring pmacct can be found here.
A docker image is available at https://gitlab.com/lama-corp/infra/tools/flow-exporter/container_registry
A Dockerfile is provided for convenience. It will build the source and then run the exporter. You can use the Docker command line like so:
docker run -p 9590:9590 flow-exporter:latest --brokers=kafka.fqdn.com:9092 --topic=pmacct.acct --asn=15169
Or if you prefer Docker Compose:
flow-exporter:
image: flow-exporter:latest
command: --brokers=kafka.fqdn.com:9092 --topic=pmacct.acct --asn=15169
expose:
- 9590
Ideally in the same docker-compose.yml
file as your Prometheus server to make communication easy.
The application can be compiled by running:
git clone https://github.com/neptune-networks/flow-exporter
cd flow-exporter/cmd/flow-exporter
make