calmh / ipfix Goto Github PK
View Code? Open in Web Editor NEWIPFIX parser package for Go
License: MIT License
IPFIX parser package for Go
License: MIT License
The boolean values in IPFIX do not use 0 as false. See http://tools.ietf.org/html/rfc5101#section-6.1.5
1 means true, 2 means false
Readme states, that: An input stream in the form of an io.Reader, net.PacketConn, or a []byte is read and chunked into messages.
However, there are no Session methods to read from PacketConn, and, obviously,
cannot use pc (type net.PacketConn) as type io.Reader in argument to s.ParseReader:
net.PacketConn does not implement io.Reader (missing Read method)
Currently, the builtin dictionary is generated by downloading and parsing the RFC.
The better way would be to generate the dictionary using the ipfix.xml file provided by iana ( http://www.iana.org/assignments/ipfix/ipfix.xml ).
This file is being updated on changes and defines a lot more elements than the RFC.
If you wish, I could write a script which generates builtin-dictionary.go from this xml file.
For example, openvswitch uses the same file to generate some C macros, see https://github.com/openvswitch/ovs/blob/master/ofproto/ipfix-gen-entities .
I see here https://github.com/calmh/ipfix/blob/master/parser.go#L296 that option templates are not handled at all.
I am not very familiar with the IPFIX format. How hard would it be to implement support for Option Templates?
I am trying to grab statistics from yaf flow files. Relevant docs embedded below in this post, see https://tools.netsa.cert.org/yaf/yaf.html and search for "Statistics Option Template" for the source.
I tried contacting you on the gophers slack to talk, but I figured I'd post here as well :)
Statistics Option Template
yaf will export information about its process periodically using IPFIX Options Template Record. This record gives information about the status of the flow and fragment table, as well as decoding information. This can be turned off using the --no-stats option. The following Information Elements will be exported:
systemInitTimeMilliseconds IE 161, 8 octets, unsigned
The time in milliseconds of the last (re-)initialization of yaf.
exportedFlowRecordTotalCount IE 42, 8 octets, unsigned
Total amount of exported flows from yaf start time.
packetTotalCount IE 86, 8 octets, unsigned
Total amount of packets processed by yaf from yaf start time.
droppedPacketTotalCount IE 135, 8 octets, unsigned
Total amount of dropped packets according to statistics given by libpcap, libdag, or the Napatech or Netronome APIs.
ignoredPacketTotalCount IE 164, 8 octets, unsigned
Total amount of packets ignored by the yaf packet decoder, such as unsupported packet types and incomplete headers, from yaf start time.
notSentPacketTotalCount IE 167, 8 octets, unsigned
Total amount of packets rejected by yaf because they were received out of sequence.
expiredFragmentCount CERT (PEN 6871) IE 100, 4 octets, unsigned
Total amount of fragments that have been expired since yaf start time.
assembledFragmentCount CERT (PEN 6871) IE 101, 4 octets, unsigned
Total number of packets that been assembled from a series of fragments since yaf start time.
flowTableFlushEventCount CERT (PEN 6871) IE 104, 4 octets, unsigned
Total number of times the yaf flow table has been flushed since yaf start time.
flowTablePeakCount CERT (PEN 6871) IE 105, 4 octets, unsigned
The maximum number of flows in the yaf flow table at any one time since yaf start time.
exporterIPv4Address IE 130, 4 octets, unsigned
The IPv4 Address of the yaf flow sensor.
exportingProcessId IE 144, 4 octets, unsigned
Set the ID of the yaf flow sensor by giving a value to --observation-domain. The default is 0.
meanFlowRate CERT (PEN 6871) IE 102, 4 octets, unsigned
The mean flow rate of the yaf flow sensor since yaf start time, rounded to the nearest integer.
meanPacketRate CERT (PEN 6871) IE 103, 4 octets, unsigned
The mean packet rate of the yaf flow sensor since yaf start time, rounded to the nearest integer.
Hi,
the IPFIX protocoll allows to mix template and data record sets (see section 3 of RFC7011). Unfortunately the parser fails in this case. The following code uses the example given in the wikipedia article on IPFIX to trigger the bug:
package ipfix_test
import (
"encoding/hex"
"testing"
"github.com/calmh/ipfix"
)
func TestParsingTemplateAndDataRecords(t *testing.T) {
packet, _ := hex.DecodeString("000a00405685b3700000000000bc614e000200140100000300080004000c0004000200040100001cc0a800c9c0a80001000000ebc0a800cac0a800010000002a")
p := ipfix.NewSession()
msg, err := p.ParseBuffer(packet)
if err != nil {
t.Fatal("ParseBuffer failed", err)
}
if len(msg.DataRecords) != 1 {
t.Error("Incorrect number of data records", len(msg.DataRecords))
}
if len(msg.TemplateRecords) != 1 {
t.Error("Incorrect number of template records", len(msg.TemplateRecords))
}
}
Cheers,
tpltnt
I'm not sure why, but uint64 fields (like octetDeltaCount/packetDeltaCount) are most of the time come as 4 bytes instead of 8, so interpreter thinks that the field is corrupted and returns them as array of bytes.
This also happens with uint32 data types like ingressInterface/egressInterface where there's two bytes instead of four.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.