vbajpai / nfql Goto Github PK

Network Flow Query Language (NFQL) is our in-house stream-based flow-query language. The query domain specific langauge (DSL) language allows to describe patterns in flow records in a declarative and orthogonal fashion.

• Filter flows.
• Combine flows into groups.
• Aggregate flows on flow-keys as one grouped flow aggregate.
• Merge grouped flows, supporting temporal relations between groups.
• Apply absolute or relative filters when grouping or merging.
• Unfold grouped flows back into individual flows.

nfql is a reference implementation of NFQL. nfql is composed of an execution engine and a query parser. The execution engine is the brain of nfql where the flows are processed. nfql reads the flow-query in an intermediate JSON format and reads in the trace files in memory for efficient processing. The query parser can be used to read a custom query DSL and generate the JSON intermediate representation required by the execution engine.

• Supports reading and writing NetFlow v5 flows in flow-tools format.
• The default query DSL can be mathematically expressed as a DNF expression.
• The query is read in a JSON format allowing M2M communication.
• The query parser and execution engine are decoupled from one another.
• The flow processing is modular adhering to a five-stage processing pipeline.
• Each stage of the processing pipeline can be enabled/disabled at runtime.
• Compression levels for flow-traces can be configured at runtime.

The JSON representation need not be manually generated by the user. A NFQL parser must read, syntactically check and parse the query DSL to generate an equivalent JSON representation. Currently we are missing such a parser module. For now, we provide custom Python scripts that can generate JSON representations. A new query and its corresponding unit test can be prepared by adapting to one of the respective example/ and tests/ scripts provided in the package.

Tried on Debian Wheezy x86_64

Install Dependencies

$ sudo apt-get install cmake flow-tools-dev zlib1g-dev libjson0-dev

Install IPFIX protocol library

$ sudo apt-get install libglib2.0-dev  
$ wget http://tools.netsa.cert.org/releases/libfixbuf-1.4.0.tar.gz
$ tar -zxvf libfixbuf-1.4.0.tar.gz
[libfixbuf-1.4.0] $ ./configure
[libfixbuf-1.4.0] $ make
[libfixbuf-1.4.0] $ sudo make install 

Build and install nfql

[nfql] $ make
[nfql] $ sudo make install
$ sudo ldconfig

Read the man page

$ sudo apt-get install most
$ man nfql | most

Generate Documentation (optional)

$ sudo apt-get install doxygen
$ sudo apt-get install graphviz
[nfql] $ make doc

Cleanup

[nfql] $ make clean

Tried on Mac OS X 10.9.1

Install Homebrew →

$ ruby -e "$(curl -fsSL https://raw.github.com/Homebrew/homebrew/go/install)"

Install Dependencies

$ brew install cmake
$ brew install flow-tools

NOTE: json-c starting from v0.11 has renamed the library from libjson to libjson-c. Debian-based systems are currently providing v0.10 in the repository. In order to avoid complexity in the makefile, we have decided to freeze the library dependency to v0.10. Install json-c v0.10:

$  brew install https://raw.github.com/mxcl/homebrew/9f78662acff4c3cb1bff3587c8ead7f5935925a9/Library/Formula/json-c.rb

Install IPFIX protocol library

$ brew install libfixbuf
$ brew install gettext
$ brew link --force gettext

Build and install nfql

[nfql] $ make 
[nfql] $ make install

Read the man page

$ brew install most 
$ man nfql | most

Generate Documentation (optional)

$ brew install doxygen
$ brew install graphviz
[nfql] $ make doc

Cleanup

[nfql] $ make clean

WARNING: This documentation is obsolete.

Tried on Mac OS X 10.7.

Install MacPorts →

Install Dependencies:

$ sudo port install cmake flow-tools json-c

Build nfql:

[nfql] $ make CMAKE_PREFIX_PATH=/opt/local
[nfql] $ make install

Read the man page:

$ man nfql

Generate Documentation (optional)

$ sudo port install doxygen graphviz
[nfql] $ make doc

Cleanup

[nfql] $ make clean

Tried on Fedora 18.

Install Dependencies

$ sudo yum install cmake flow-tools-devel json-c-devel

Install IPFIX protocol library

$ sudo yum install glib2-devel
$ wget http://tools.netsa.cert.org/releases/libfixbuf-1.4.0.tar.gz
$ tar -zxvf libfixbuf-1.4.0.tar.gz
[libfixbuf-1.4.0] $ ./configure
[libfixbuf-1.4.0] $ make
[libfixbuf-1.4.0] $ sudo make install 

Build and install nfql

[nfql] $ make 
[nfql] $ sudo make install
# echo "/usr/local/lib" >> /etc/ld.so.conf
# ldconfig

Read the man page

$ sudo yum install most
$ man nfql | most

Generate Documentation (optional)

$ sudo yum install doxygen graphviz
[nfql] $ make doc

Cleanup

[nfql] $ make clean

WARNING: This documentation is obsolete.

Tried on FreeBSD 9.2

Install Dependencies

$ cd /usr/ports/devel/cmake
[cmake] $ sudo make install 

$ cd /usr/ports/net-mgmt/flow-tools
[flow-tools] $ sudo make install 

$ cd /usr/ports/devel/json-c
[json-c] $ sudo make install 

$ cd /usr/ports/devel/glib20
[glib20] $ sudo make install

$ cd /usr/ports/net/libfixbuf
[libfixbuf] $ sudo make install

$ cd /usr/ports/devel/libexecinfo
[libexecinfo] $ sudo make install

Build and install nfql

[nfql] $ make CMAKE_PREFIX_PATH=/usr/local
[nfql] $ make install

Read the man page

$ cd /usr/ports/sysutils/most
[most] $ sudo make install
$ man nfql | most

Generate Documentation (optional)

$ cd /usr/ports/devel/doxygen
[doxygen] $ sudo make install 

$ cd /usr/ports/graphics/graphviz
[graphviz] $ sudo make install 

[nfql] $ make doc

Cleanup

[nfql] $ make clean

Some example queries are provided in examples/ along with a sample trace.

[nfql] $ nfql --ipfix examples/query-http-tcp-session.json examples/trace-2009.ipfix  
[nfql] $ nfql examples/query-http-tcp-session.json examples/trace-2009.ft

The sample queries can also be run on your own NetFlow v5 records

[nfql] $ flow-cat tracefile[s] | nfql examples/query-http-tcp-session.json

Requirements: Python 2.7+

To run the complete regression test-suite:

[nfql] $ tests/regression.py [-v]

Regression tests can also be run individually on a specific example query type. For instance:

[nfql] $ tests/test-query-http-tcp-session.py [-v]

Requirements: Python 2.7+

To run the nfql benchmarks:

[nfql] $ make
[nfql] $ sudo benchmarks/nfql.py nfql tracefile[s]/ querie[s]/

Example nfql traces and queries are provided in examples/

To run the SiLK benchmarks:

[nfql] $ sudo benchmarks/silk.py tracefile[s]/ querie[s]/

Example SiLK traces and queries are provided in examples/silk/

• Vaibhav Bajpai [email protected]
• Johannes Schauer [email protected]
• Corneliu Claudiu Prodescu [email protected]
• Jürgen Schönwälder [email protected]