Git Product home page Git Product logo

nfql's Introduction

NFQL


Network Flow Query Language (NFQL) is our in-house stream-based flow-query language. The query domain specific langauge (DSL) language allows to describe patterns in flow records in a declarative and orthogonal fashion.

Features


  • Filter flows.
  • Combine flows into groups.
  • Aggregate flows on flow-keys as one grouped flow aggregate.
  • Merge grouped flows, supporting temporal relations between groups.
  • Apply absolute or relative filters when grouping or merging.
  • Unfold grouped flows back into individual flows.

nfql


nfql is a reference implementation of NFQL. nfql is composed of an execution engine and a query parser. The execution engine is the brain of nfql where the flows are processed. nfql reads the flow-query in an intermediate JSON format and reads in the trace files in memory for efficient processing. The query parser can be used to read a custom query DSL and generate the JSON intermediate representation required by the execution engine.

Architecture


  • Supports reading and writing NetFlow v5 flows in flow-tools format.
  • The default query DSL can be mathematically expressed as a DNF expression.
  • The query is read in a JSON format allowing M2M communication.
  • The query parser and execution engine are decoupled from one another.
  • The flow processing is modular adhering to a five-stage processing pipeline.
  • Each stage of the processing pipeline can be enabled/disabled at runtime.
  • Compression levels for flow-traces can be configured at runtime.

The JSON representation need not be manually generated by the user. A NFQL parser must read, syntactically check and parse the query DSL to generate an equivalent JSON representation. Currently we are missing such a parser module. For now, we provide custom Python scripts that can generate JSON representations. A new query and its corresponding unit test can be prepared by adapting to one of the respective example/ and tests/ scripts provided in the package.

Installing on Debian-based Linux


Tried on Debian Wheezy x86_64

Install Dependencies

$ sudo apt-get install cmake flow-tools-dev zlib1g-dev libjson0-dev

Install IPFIX protocol library

$ sudo apt-get install libglib2.0-dev  
$ wget http://tools.netsa.cert.org/releases/libfixbuf-1.4.0.tar.gz
$ tar -zxvf libfixbuf-1.4.0.tar.gz
[libfixbuf-1.4.0] $ ./configure
[libfixbuf-1.4.0] $ make
[libfixbuf-1.4.0] $ sudo make install 

Build and install nfql

[nfql] $ make
[nfql] $ sudo make install
$ sudo ldconfig

Read the man page

$ sudo apt-get install most
$ man nfql | most

Generate Documentation (optional)

$ sudo apt-get install doxygen
$ sudo apt-get install graphviz
[nfql] $ make doc

Cleanup

[nfql] $ make clean

Installing on Mac OS X using Homebrew


Tried on Mac OS X 10.9.1

Install Homebrew →

$ ruby -e "$(curl -fsSL https://raw.github.com/Homebrew/homebrew/go/install)"

Install Dependencies

$ brew install cmake
$ brew install flow-tools

NOTE: json-c starting from v0.11 has renamed the library from libjson to libjson-c. Debian-based systems are currently providing v0.10 in the repository. In order to avoid complexity in the makefile, we have decided to freeze the library dependency to v0.10. Install json-c v0.10:

$  brew install https://raw.github.com/mxcl/homebrew/9f78662acff4c3cb1bff3587c8ead7f5935925a9/Library/Formula/json-c.rb

Install IPFIX protocol library

$ brew install libfixbuf
$ brew install gettext
$ brew link --force gettext

Build and install nfql

[nfql] $ make 
[nfql] $ make install

Read the man page

$ brew install most 
$ man nfql | most

Generate Documentation (optional)

$ brew install doxygen
$ brew install graphviz
[nfql] $ make doc

Cleanup

[nfql] $ make clean

Installing on Mac OS X using MacPorts


WARNING: This documentation is obsolete.

Tried on Mac OS X 10.7.

Install MacPorts →

Install Dependencies:

$ sudo port install cmake flow-tools json-c

Build nfql:

[nfql] $ make CMAKE_PREFIX_PATH=/opt/local
[nfql] $ make install

Read the man page:

$ man nfql

Generate Documentation (optional)

$ sudo port install doxygen graphviz
[nfql] $ make doc

Cleanup

[nfql] $ make clean

Installing on Fedora Linux


Tried on Fedora 18.

Install Dependencies

$ sudo yum install cmake flow-tools-devel json-c-devel

Install IPFIX protocol library

$ sudo yum install glib2-devel
$ wget http://tools.netsa.cert.org/releases/libfixbuf-1.4.0.tar.gz
$ tar -zxvf libfixbuf-1.4.0.tar.gz
[libfixbuf-1.4.0] $ ./configure
[libfixbuf-1.4.0] $ make
[libfixbuf-1.4.0] $ sudo make install 

Build and install nfql

[nfql] $ make 
[nfql] $ sudo make install
# echo "/usr/local/lib" >> /etc/ld.so.conf
# ldconfig

Read the man page

$ sudo yum install most
$ man nfql | most

Generate Documentation (optional)

$ sudo yum install doxygen graphviz
[nfql] $ make doc

Cleanup

[nfql] $ make clean

Installing on FreeBSD


WARNING: This documentation is obsolete.

Tried on FreeBSD 9.2

Install Dependencies

$ cd /usr/ports/devel/cmake
[cmake] $ sudo make install 

$ cd /usr/ports/net-mgmt/flow-tools
[flow-tools] $ sudo make install 

$ cd /usr/ports/devel/json-c
[json-c] $ sudo make install 

$ cd /usr/ports/devel/glib20
[glib20] $ sudo make install

$ cd /usr/ports/net/libfixbuf
[libfixbuf] $ sudo make install

$ cd /usr/ports/devel/libexecinfo
[libexecinfo] $ sudo make install

Build and install nfql

[nfql] $ make CMAKE_PREFIX_PATH=/usr/local
[nfql] $ make install

Read the man page

$ cd /usr/ports/sysutils/most
[most] $ sudo make install
$ man nfql | most

Generate Documentation (optional)

$ cd /usr/ports/devel/doxygen
[doxygen] $ sudo make install 

$ cd /usr/ports/graphics/graphviz
[graphviz] $ sudo make install 

[nfql] $ make doc

Cleanup

[nfql] $ make clean

Running nfql


Some example queries are provided in examples/ along with a sample trace.

[nfql] $ nfql --ipfix examples/query-http-tcp-session.json examples/trace-2009.ipfix  
[nfql] $ nfql examples/query-http-tcp-session.json examples/trace-2009.ft

The sample queries can also be run on your own NetFlow v5 records

[nfql] $ flow-cat tracefile[s] | nfql examples/query-http-tcp-session.json

Running the Test Suite


Requirements: Python 2.7+

To run the complete regression test-suite:

[nfql] $ tests/regression.py [-v]

Regression tests can also be run individually on a specific example query type. For instance:

[nfql] $ tests/test-query-http-tcp-session.py [-v]

Running the Benchmarks


Requirements: Python 2.7+

To run the nfql benchmarks:

[nfql] $ make
[nfql] $ sudo benchmarks/nfql.py nfql tracefile[s]/ querie[s]/

Example nfql traces and queries are provided in examples/

To run the SiLK benchmarks:

[nfql] $ sudo benchmarks/silk.py tracefile[s]/ querie[s]/

Example SiLK traces and queries are provided in examples/silk/

Authors


nfql's People

Contributors

vbajpai avatar cprodescu avatar

Stargazers

YQ avatar George Erickson avatar adjohnson avatar  avatar  avatar Changling Zhou avatar

Watchers

 avatar Changling Zhou avatar  avatar James Cloos avatar  avatar

Forkers

blakme

nfql's Issues

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.