Git Product home page Git Product logo

nfql's Introduction

NFQL


Network Flow Query Language (NFQL) is our in-house stream-based flow-query language. The query domain specific langauge (DSL) language allows to describe patterns in flow records in a declarative and orthogonal fashion.

Features


  • Filter flows.
  • Combine flows into groups.
  • Aggregate flows on flow-keys as one grouped flow aggregate.
  • Merge grouped flows, supporting temporal relations between groups.
  • Apply absolute or relative filters when grouping or merging.
  • Unfold grouped flows back into individual flows.

nfql


nfql is a reference implementation of NFQL. nfql is composed of an execution engine and a query parser. The execution engine is the brain of nfql where the flows are processed. nfql reads the flow-query in an intermediate JSON format and reads in the trace files in memory for efficient processing. The query parser can be used to read a custom query DSL and generate the JSON intermediate representation required by the execution engine.

Architecture


  • Supports reading and writing NetFlow v5 flows in flow-tools format.
  • The default query DSL can be mathematically expressed as a DNF expression.
  • The query is read in a JSON format allowing M2M communication.
  • The query parser and execution engine are decoupled from one another.
  • The flow processing is modular adhering to a five-stage processing pipeline.
  • Each stage of the processing pipeline can be enabled/disabled at runtime.
  • Compression levels for flow-traces can be configured at runtime.

The JSON representation need not be manually generated by the user. A NFQL parser must read, syntactically check and parse the query DSL to generate an equivalent JSON representation. Currently we are missing such a parser module. For now, we provide custom Python scripts that can generate JSON representations. A new query and its corresponding unit test can be prepared by adapting to one of the respective example/ and tests/ scripts provided in the package.

Installing on Debian-based Linux


Tried on Debian Wheezy x86_64

Install Dependencies

$ sudo apt-get install cmake flow-tools-dev zlib1g-dev libjson0-dev

Install IPFIX protocol library

$ sudo apt-get install libglib2.0-dev  
$ wget http://tools.netsa.cert.org/releases/libfixbuf-1.4.0.tar.gz
$ tar -zxvf libfixbuf-1.4.0.tar.gz
[libfixbuf-1.4.0] $ ./configure
[libfixbuf-1.4.0] $ make
[libfixbuf-1.4.0] $ sudo make install 

Build and install nfql

[nfql] $ make
[nfql] $ sudo make install
$ sudo ldconfig

Read the man page

$ sudo apt-get install most
$ man nfql | most

Generate Documentation (optional)

$ sudo apt-get install doxygen
$ sudo apt-get install graphviz
[nfql] $ make doc

Cleanup

[nfql] $ make clean

Installing on Mac OS X using Homebrew


Tried on Mac OS X 10.9.1

Install Homebrew →

$ ruby -e "$(curl -fsSL https://raw.github.com/Homebrew/homebrew/go/install)"

Install Dependencies

$ brew install cmake
$ brew install flow-tools

NOTE: json-c starting from v0.11 has renamed the library from libjson to libjson-c. Debian-based systems are currently providing v0.10 in the repository. In order to avoid complexity in the makefile, we have decided to freeze the library dependency to v0.10. Install json-c v0.10:

$  brew install https://raw.github.com/mxcl/homebrew/9f78662acff4c3cb1bff3587c8ead7f5935925a9/Library/Formula/json-c.rb

Install IPFIX protocol library

$ brew install libfixbuf
$ brew install gettext
$ brew link --force gettext

Build and install nfql

[nfql] $ make 
[nfql] $ make install

Read the man page

$ brew install most 
$ man nfql | most

Generate Documentation (optional)

$ brew install doxygen
$ brew install graphviz
[nfql] $ make doc

Cleanup

[nfql] $ make clean

Installing on Mac OS X using MacPorts


WARNING: This documentation is obsolete.

Tried on Mac OS X 10.7.

Install MacPorts →

Install Dependencies:

$ sudo port install cmake flow-tools json-c

Build nfql:

[nfql] $ make CMAKE_PREFIX_PATH=/opt/local
[nfql] $ make install

Read the man page:

$ man nfql

Generate Documentation (optional)

$ sudo port install doxygen graphviz
[nfql] $ make doc

Cleanup

[nfql] $ make clean

Installing on Fedora Linux


Tried on Fedora 18.

Install Dependencies

$ sudo yum install cmake flow-tools-devel json-c-devel

Install IPFIX protocol library

$ sudo yum install glib2-devel
$ wget http://tools.netsa.cert.org/releases/libfixbuf-1.4.0.tar.gz
$ tar -zxvf libfixbuf-1.4.0.tar.gz
[libfixbuf-1.4.0] $ ./configure
[libfixbuf-1.4.0] $ make
[libfixbuf-1.4.0] $ sudo make install 

Build and install nfql

[nfql] $ make 
[nfql] $ sudo make install
# echo "/usr/local/lib" >> /etc/ld.so.conf
# ldconfig

Read the man page

$ sudo yum install most
$ man nfql | most

Generate Documentation (optional)

$ sudo yum install doxygen graphviz
[nfql] $ make doc

Cleanup

[nfql] $ make clean

Installing on FreeBSD


WARNING: This documentation is obsolete.

Tried on FreeBSD 9.2

Install Dependencies

$ cd /usr/ports/devel/cmake
[cmake] $ sudo make install 

$ cd /usr/ports/net-mgmt/flow-tools
[flow-tools] $ sudo make install 

$ cd /usr/ports/devel/json-c
[json-c] $ sudo make install 

$ cd /usr/ports/devel/glib20
[glib20] $ sudo make install

$ cd /usr/ports/net/libfixbuf
[libfixbuf] $ sudo make install

$ cd /usr/ports/devel/libexecinfo
[libexecinfo] $ sudo make install

Build and install nfql

[nfql] $ make CMAKE_PREFIX_PATH=/usr/local
[nfql] $ make install

Read the man page

$ cd /usr/ports/sysutils/most
[most] $ sudo make install
$ man nfql | most

Generate Documentation (optional)

$ cd /usr/ports/devel/doxygen
[doxygen] $ sudo make install 

$ cd /usr/ports/graphics/graphviz
[graphviz] $ sudo make install 

[nfql] $ make doc

Cleanup

[nfql] $ make clean

Running nfql


Some example queries are provided in examples/ along with a sample trace.

[nfql] $ nfql --ipfix examples/query-http-tcp-session.json examples/trace-2009.ipfix  
[nfql] $ nfql examples/query-http-tcp-session.json examples/trace-2009.ft

The sample queries can also be run on your own NetFlow v5 records

[nfql] $ flow-cat tracefile[s] | nfql examples/query-http-tcp-session.json

Running the Test Suite


Requirements: Python 2.7+

To run the complete regression test-suite:

[nfql] $ tests/regression.py [-v]

Regression tests can also be run individually on a specific example query type. For instance:

[nfql] $ tests/test-query-http-tcp-session.py [-v]

Running the Benchmarks


Requirements: Python 2.7+

To run the nfql benchmarks:

[nfql] $ make
[nfql] $ sudo benchmarks/nfql.py nfql tracefile[s]/ querie[s]/

Example nfql traces and queries are provided in examples/

To run the SiLK benchmarks:

[nfql] $ sudo benchmarks/silk.py tracefile[s]/ querie[s]/

Example SiLK traces and queries are provided in examples/silk/

Authors


nfql's People

Contributors

cprodescu avatar vbajpai avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

Forkers

blakme

nfql's Issues

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.