High Speed PCAP2JSON conversion utility for importing PCAP network data into Elastic Search / ELK
Example implementation
Full description is here https://fmad.io/blog-network-flow-monitoring.html
fmadio@fmadio20v2-149:/mnt/store0/git/pcap2json$ ./pcap2json --help
fmad engineering all rights reserved
http://www.fmad.io
pcap2json is a high speed PCAP meta data extraction utility
example converting a pcap to json:
cat /tmp/test.pcap | pcap2json > test.json
Command Line Arguments:
--capture-name <name> : capture name to use for ES Index data
--verbose : verbose output
--config <confrig file> : read from config file
--json-packet : write JSON packet data
--json-flow : write JSON flow data
Output Mode
--output-stdout : writes output to STDOUT
--output-espush : writes output directly to ES HTTP POST
--output-lineflush <line cnt> : number of lines before flushing output (default 100e3)
--output-timeflush <time ns> : maximum amount of time since last flush (default 1e9(
--output-cpu <gen1|gen2> : cpu mapping list to run on
Flow specific options
--flow-samplerate <nanos> : scientific notation flow sample rate. default 100e6 (100msec)
JSON Output Control
--disable-mac : disable JSON MAC output
--disable-vlan : disable JSON VLAN output
--disable-mpls : disable JSON MPLS output
--disable-ipv4 : disable JSON IPv4 output
--disable-udp : disable JSON UDP output
--disable-tcp : disable JSON TCP output
Elastic Stack options
--es-host <hostname:port> : Sets the ES Hostname
--es-compress : enables gzip compressed POST
fmadio@fmadio20v2-149:/mnt/store0/git/pcap2json_20181103_rc1$
/usr/local/bin/curl -H "Content-Type: application/json" -XPUT "192.168.2.115:9200/interop17?pretty" --data-binary "@mappings.json"
$ cat /mnt/store1/tmp/interop17_hotstage_20170609_133953.717.953.280.pcap | ./pcap2json --json-packet --capture-name interop17 --output-espush --es-compress --es-host 192.168.2.115
This uses a high performance multithreaded direct C socket to to push the JSON data directly into ES. Multiple ES hosts can be specified to load balance the ingress queue.
$ cat /mnt/store1/tmp/interop17_hotstage_20170609_133953.717.953.280.pcap | ./pcap2json --json-packet --capture-name interop17 --output-stdout
Outputs the JSON data on stdout, which is usually piped to a file. This is helpful if debugging is required
Please see the config.* files for other examples. This these are specified using the --config option
{"index":{"_index":"interop17","_type":"flow_record","_score":null}}
{"timestamp":1497015814284.541992,"TS":"13:43:34.284.542.042","FlowCnt":0,"Device":"fmadio20v2-149","hash":"d80b04ebb1a14bdc72ed17cde664cda755b39d8d","MACSrc":"7c:e2:ca:bd:97:d9","MACDst":"00:0e:52:80:00:16","MACProto":"IPv4","IPv4.Src":"150.100.29.14","IPv4.Dst":"130.128.19.30" ,"IPv4.Proto":"UDP","UDP.Port.Src":10662,"UDP.Port.Dst":5004,"TotalPkt":0,"TotalByte":0}
{"index":{"_index":"interop17","_type":"flow_record","_score":null}}
{"timestamp":1497015814284.543213,"TS":"13:43:34.284.543.223","FlowCnt":0,"Device":"fmadio20v2-149","hash":"d80b04ebb1a14bdc72ed17cde664cda755b39d8d","MACSrc":"7c:e2:ca:bd:97:d9","MACDst":"00:0e:52:80:00:16","MACProto":"IPv4","IPv4.Src":"150.100.29.14","IPv4.Dst":"130.128.19.30" ,"IPv4.Proto":"UDP","UDP.Port.Src":10662,"UDP.Port.Dst":5004,"TotalPkt":0,"TotalByte":0}
{"index":{"_index":"interop17","_type":"flow_record","_score":null}}
{"timestamp":1497015814284.544189,"TS":"13:43:34.284.544.362","FlowCnt":0,"Device":"fmadio20v2-149","hash":"d80b04ebb1a14bdc72ed17cde664cda755b39d8d","MACSrc":"7c:e2:ca:bd:97:d9","MACDst":"00:0e:52:80:00:16","MACProto":"IPv4","IPv4.Src":"150.100.29.14","IPv4.Dst":"130.128.19.30" ,"IPv4.Proto":"UDP","UDP.Port.Src":10662,"UDP.Port.Dst":5004,"TotalPkt":0,"TotalByte":0}
{"index":{"_index":"interop17","_type":"flow_record","_score":null}}
{"timestamp":1497015814284.549072,"TS":"13:43:34.284.548.992","FlowCnt":0,"Device":"fmadio20v2-149","hash":"d80b04ebb1a14bdc72ed17cde664cda755b39d8d","MACSrc":"7c:e2:ca:bd:97:d9","MACDst":"00:0e:52:80:00:16","MACProto":"IPv4","IPv4.Src":"150.100.29.14","IPv4.Dst":"130.128.19.30" ,"IPv4.Proto":"UDP","UDP.Port.Src":10662,"UDP.Port.Dst":5004,"TotalPkt":0,"TotalByte":0}
{"index":{"_index":"interop17","_type":"flow_record","_score":null}}
{"timestamp":1497015814284.549316,"TS":"13:43:34.284.549.405","FlowCnt":0,"Device":"fmadio20v2-149","hash":"b6183e3af206ac1c43eefb261f4ec03811ff1a45","MACSrc":"7c:e2:ca:bd:97:d9","MACDst":"00:0e:52:80:00:16","MACProto":"IPv4","IPv4.Src":"45.0.191.123","IPv4.Dst":"205.177.226.213" ,"IPv4.Proto":"UDP","UDP.Port.Src":10500,"UDP.Port.Dst":20986,"TotalPkt":0,"TotalByte":0}Using FMADIO 20Gv2 Packet Capture system uses 12 CPUs
100K Flows @ 64B per packet ~5.1Mpps
100K Flows @ 1500B per packet 15.48Gbps
1M Flows @ 64B per packet 36Mpps. Using a 96 CPU + 384GB RAM m5.metal machine (Blue is reference running on FMADIO 20G Gen2 system)
Generating a profile snapshot as follows. 5 minute (300e9 nanosec) profile sample
sudo stream_cat -v --time-start 20:00:00 --time-stop 20:05:00 --ignore_fcs mycapture_20200313_1712 | ./pcap2json --json-flow --flow-samplerate 300e9 --output-null --output-histogram mycapture_20200313_2000.profile
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent