cesnet / nemea-modules Goto Github PK

View Code? Open in Web Editor NEW

14.0 15.0 20.0 5.6 MB

Base modules of the Nemea system. This repository contains modules for export, replay, filtering, merging etc.

License: Other

Shell 1.51% C 27.05% C++ 59.27% Makefile 0.60% Yacc 0.12% Lex 0.14% Python 5.58% M4 2.70% HTML 2.55% Lua 0.49%

nemea liberouter

nemea-modules's Introduction

NEMEA Modules

This repository contains basic modules of the NEMEA system. The modules and their functionality/purposes are:

logger: export messages into CSV
logreplay: replay stored CSV
nfreader: replay stored nfdump file(s)
flow_meter: simple flow exporter from network interface controller (NIC) or PCAP file
unirecfilter: filtering module
anonymizer: module for anonymization of flow records
traffic_repeater: flow repeater module (e.g. for interconnection of modules using different types of communication interfaces)
flowcounter: simple example of flow counting module
report2idea: reporting modules that receive alerts from detectors and store them into database (MongoDB), file or send them into Warden
resolver: amend flow records with resolved fields
luamodule: module to filter or modify input stream of messages into one output stream of messages
merger: module for joining multiple input streams of messages into one output stream of messages
debug_sender: interactive tool (in Python) for creation and sending own messages/flow records

nemea-modules's People

Stargazers

Watchers

nemea-modules's Issues

anonymizer: unchecked rv of fgets!

Here: https://github.com/CESNET/Nemea-Modules/blob/master/anonymizer/anonymizer.c#L151
Wrong usage of fgets, check if it succeeded.

Also should revisit this old code and check for other possible problems.

Warning shebang in python scripts/modules

The path to python(2) interpreter will be deprecated because of implicit move to python3 in a near future.

Therefore, we should rewrite #!/usr/bin/env python resp. #!/usr/bin/python to #!/usr/bin/env python2 resp. #!/usr/bin/python2.

Improvement of README (merger)

Usage example doesn't work by default. In source code is requested different template format.

flowcounter - documentation update

Flowcounter needs to update its README and help.

remove "-u" parameter
explain in help, that "-o" enables output interface
describe output format completely, "@Volume" is not specified anywhere now

Move sup files from supervisor to modules

According to the mailing list (the following messages), sup files will be moved from the Nemea-Supervisor repo into this repo.

https://random.cesnet.cz/pipermail/nemea/2016-November/000016.html
https://random.cesnet.cz/pipermail/nemea/2016-November/000017.html
https://random.cesnet.cz/pipermail/nemea/2016-November/000018.html

A macro NEMEASUPDIR must be defined to specify default path for installation of the sup files.

@thorgrin suggested to use /etc/nemea/sup-available/ as a default value for NEMEASUPDIR.

Remove libnfdump

Since nobody protested via mailinglist (https://random.cesnet.cz/pipermail/nemea/2016-October/000006.html), libnfdump will be completely removed from Modules.

It is necessary to change configure.ac, Makefile.am files and, finally, clean the source codes of nfreader.

Run tests before PR (at least make distcheck), additionally, the functionality should be tested as well, any sample test dataset will be appreciated.

flow_meter: incompatible http template

Our (CESNET's) exporters and collector operate with the following template of HTTP fields:

ipaddr DST_IP,ipaddr SRC_IP,uint64 BYTES,uint64 LINK_BIT_FIELD,time TIME_FIRST,time TIME_LAST,uint32 HTTP_REQUEST_AGENT_ID,uint32 HTTP_REQUEST_METHOD_ID,uint32 HTTP_RESPONSE_STATUS_CODE,uint32 PACKETS,uint16 DST_PORT,uint16 SRC_PORT,uint8 DIR_BIT_FIELD,uint8 PROTOCOL,uint8 TCP_FLAGS,uint8 TOS,uint8 TTL,string HTTP_REQUEST_AGENT,string HTTP_REQUEST_HOST,string HTTP_REQUEST_REFERER,string HTTP_REQUEST_URL,string HTTP_RESPONSE_CONTENT_TYPE

Is there any reason why flow_meter is not compatible in this naming "convention"?

The flow_meter should be fixed.

vportscan aggregation logger does not work

When I try to run vportscan aggregation logger, it fails:

vportscan_agg_logger_stderr:

Fri Jul 13 16:07:46 2018 [INFO] Supervisor - executed command: /usr/bin/nemea/logger   -t   -T   -a   /data/vportscan/alerts.log   -i   t:12005   (null)
trap_recv() error: output and input interfaces data formats or data specifiers mismatch.

I have verified that the port 12005 belongs to the vportscan_aggregator. Is there anything else that I can check or do to make it work? Thanks

Fix distcheck broken by flow_meter

There are missing files and bad paths in flow_meter. Therefore, distcheck fails.

debug_sender: behaviour when SIGINT occurs

README claims that the module does not exit after SIGINT.
This should be revisited and fixed, since I think debug_sender currently exits.

The source code of debug_sender should be updated (if needed) in order to prevent module exit after SIGINT without exit/quit command.

Remove OpenMP dependency in logger

Currently, logger is linked with OpenMP. It makes the work with parallel receiving easier.
The task is to replace OpenMP with just pthread, that means to replace all #pragma and #include <omp.h> from ./logger/logger.c and write an equivalent source code using pthread mutex and thread.

The motivation of this task is an easier compilation for OpenWrt.

Unstable if flow_meter uses with many plugins in one command line

@cejkato2,
After testing flow_meter, logger many times (all modules installed in Centos 6.3), I have some comments as below

1. Use many plugins in one command line (eg: basic,http,dns)
flow_meter -i t:11111,t:11112,t:11113 -p basic,http,dns -I eth1
ex: logger -i t:11113 -t
it is unstable: sometimes I can get flows, sometimes can not

But if I use with many command lines as below
flow_meter -i t:11111 -p basic -I eth1
flow_meter -i t:11112 -p http -I eth1
flow_meter -i t:11113 -p dns -I eth1
ex: logger -i t:11113 -t
it is stable, I can get flows

2. Can not auto-kill logger if use unix socket process
flow_meter -i u:c -p dns -I eth1
Then I use flow_meter -i u:c -p http -I eth1, with the same socket u:c
ex: logger -i u:c -t

When changes socket, i think Nemea should also kill or refresh 'names of fields' logger socket
I tested with TCP socket. If I kill flow_meter, then Nemea will auto-kill logger

Flow_meter export application layers on big-endian architecture

Some plugins use uint32_t constants with content that is searched for.
Unfortunately, these constants don't work on different architectures.
The issues is related mainly to openwrt compilation.

Feature request: MAC Address in plugins (flow_meter)

Dear Nemea Team,
I can't find any option for getting SRC_MAC, DST_MAC in plugins eg. basic, http
Because when client gets IP via DHCP, we need MAC address for identifying client.
So I think you should export SRC_MAC, DST_MAC for plugins
Thanks,

Module `sni_dataset_saver` is missing a `<cstdint>` include

When building on Fedora Workstation 38, the <cstdint> seems not to be included in sampler.h, leading to compile errors.

After adding #include <cstdint> to modules/sni_dataset_saver/sampler.h, the compilation proceeds without further errors.

gcc version 13.2.1 20230728 (Red Hat 13.2.1-1) (GCC)

make[4]: Entering directory '/home/bpuser/sources/Nemea/modules/sni_dataset_saver'
  CXX      sni_dataset_saver-main.o
  CC       fields.o
  CXX      sni_dataset_saver-sampler.o
  CXX      sni_dataset_saver-sni.o
  GEN      sni_dataset_saver-nemea.1
In file included from sampler.cpp:48:
sampler.h:55:5: error: ‘uint16_t’ does not name a type
   55 |     uint16_t _rate; /* reverse sampling rate. Every 1:_rate flows should not be sampled. */
      |     ^~~~~~~~
sampler.h:48:1: note: ‘uint16_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
   47 | #include <ctime>
  +++ |+#include <cstdint>
   48 | #include <iostream>
sampler.h:59:5: error: ‘uint64_t’ does not name a type
   59 |     uint64_t _counter; /* of seen flows. */
      |     ^~~~~~~~
...
...
make[4]: *** [Makefile:601: sni_dataset_saver-sampler.o] Error 1

Improvement of documentation of unirecfilter

README of UnirecFilter tells no information about PROTOCOL field.
This should be documented.

In addition, deeper analysis of this module would be helpful. The goal is to improve and complete docs for this module and add examples.

Add missing IDEA messages into report2idea examples

SIP bruteforce alert is missing, horizontal address scan alert is missing.

Unirecfilter: Support for "IN" operator for IP prefix

This issue is for tracking work on new operator "IN" for IP prefix filtering in unirecfilter module.

Example usage:

.. -F "DST_IP IN 192.168.0.0/24" ...

This would filter out all records, with destination IP address not in 192.168.0.0/24.

ipblacklist does not list dst IP address in idea message

Is there a reason why the ipblacklist2idea.py does not fill in dst IP address from the blacklist as dst address? I think that both dst and src ports as well to increase the information value of the incident report.

Revisit README files

Some modules contain README with obsoleted info such as shared memory IFC.

unirecfilter: Unable to filter fields containing number

Unirec filter is unable to filter L4 fields from ipfixprobe. Using any fields, which contains number will results in filter SYNTAX ERROR.

Example:
field: "L4_TCP_SYN_SIZE"
Error message: "[URFilter] Syntax error in filter: L4_TCP_SYN_SIZE != 0"
Using "L_TCP_SYN_SIZE" or "TCP_SYN_SIZE" will results in warning "[field] is not present in input format"

Category name typo

Category name should be Availability.DDoS not Availibility.DDoS. I've already provided a pull request to fix this some time ago. Is there any reason to keep the misspelled name?

vportscan AggrWin is fixed

I'm not really sure whether it's a bug or not, but the AggrWin of vportscan2idea reporter is fixed to 10 minutes (https://github.com/CESNET/Nemea-Modules/blob/master/report2idea/vportscan/vportscan2idea.py#L52). But this window can be set by the vportscan aggregator, can't it? Therefore, I think that this value should be based on the aggregator configuration. Can you please explain?

Flow records aggregation

The topic of flow aggregation was opened by @whitetiger21022014 as a PR...
However, it is highly relevant so let's move the discussion here.

The question: #27 (comment)

My response: #27 (comment)

BPF support in flow_meter

@whitetiger21022014 mentioned an idea about filtering support in flow_meter in #28.
It shouldn't be hard to add it, since we currently use libpcap...
http://www.tcpdump.org/pcap.html -> "Filtering traffic"

So, what about adding -F parameter and something like this:

if (filter_exp != NULL) {
   if (pcap_compile(handle, &fp, filter_exp, 0, net) == -1) {
      fprintf(stderr, "Couldn't parse filter %s: %s\n", filter_exp, pcap_geterr(handle));
   }
   if (pcap_setfilter(handle, &fp) == -1) {
      fprintf(stderr, "Couldn't install filter %s: %s\n", filter_exp, pcap_geterr(handle));
      return(2);
   }
}

into flow_meter?
Any comments? @thorgrin @havraji6 @vaclavbartos
Can this work on OpenWrt? Do we have any plans about dpdk or pf_ring? I currently don't have idea if the filtering can cause any troubles...

Unirecfilter infinite loop

I just found my unirecfilter stuck att 99% cpu and not responding to attempts to get data from it (/usr/bin/nemea/logger -t -T -i u:egress_flow_data_source -c3 would just hang).

Attaching with strace to the unirecfilter process produced an endless sequence of this:
sendto(12, "\0\0\0O\0M\0\0\0\0\0\0\0\0A4\213\250\377\377\377\377\0\0\0\0\0\0\0\0\202\354"..., 83, MSG_NOSIGNAL, NULL, 0) = -1 ENOTCONN (Transport endpoint is not connected)
sendto(12, "\0\0\0O\0M\0\0\0\0\0\0\0\0A4\213\250\377\377\377\377\0\0\0\0\0\0\0\0\202\354"..., 83, MSG_NOSIGNAL, NULL, 0) = -1 ENOTCONN (Transport endpoint is not connected)
sendto(12, "\0\0\0O\0M\0\0\0\0\0\0\0\0A4\213\250\377\377\377\377\0\0\0\0\0\0\0\0\202\354"..., 83, MSG_NOSIGNAL, NULL, 0) = -1 ENOTCONN (Transport endpoint is not connected)
sendto(12, "\0\0\0O\0M\0\0\0\0\0\0\0\0A4\213\250\377\377\377\377\0\0\0\0\0\0\0\0\202\354"..., 83, MSG_NOSIGNAL, NULL, 0) = -1 ENOTCONN (Transport endpoint is not connected)
sendto(12, "\0\0\0O\0M\0\0\0\0\0\0\0\0A4\213\250\377\377\377\377\0\0\0\0\0\0\0\0\202\354"..., 83, MSG_NOSIGNAL, NULL, 0) = -1 ENOTCONN (Transport endpoint is not connected)
sendto(12, "\0\0\0O\0M\0\0\0\0\0\0\0\0A4\213\250\377\377\377\377\0\0\0\0\0\0\0\0\202\354"..., 83, MSG_NOSIGNAL, NULL, 0) = -1 ENOTCONN (Transport endpoint is not connected)
sendto(12, "\0\0\0O\0M\0\0\0\0\0\0\0\0A4\213\250\377\377\377\377\0\0\0\0\0\0\0\0\202\354"..., 83, MSG_NOSIGNAL, NULL, 0) = -1 ENOTCONN (Transport endpoint is not connected)
sendto(12, "\0\0\0O\0M\0\0\0\0\0\0\0\0A4\213\250\377\377\377\377\0\0\0\0\0\0\0\0\202\354"..., 83, MSG_NOSIGNAL, NULL, 0) = -1 ENOTCONN (Transport endpoint is not connected)
sendto(12, "\0\0\0O\0M\0\0\0\0\0\0\0\0A4\213\250\377\377\377\377\0\0\0\0\0\0\0\0\202\354"..., 83, MSG_NOSIGNAL, NULL, 0) = -1 ENOTCONN (Transport endpoint is not connected)
sendto(12, "\0\0\0O\0M\0\0\0\0\0\0\0\0A4\213\250\377\377\377\377\0\0\0\0\0\0\0\0\202\354"..., 83, MSG_NOSIGNAL, NULL, 0) = -1 ENOTCONN (Transport endpoint is not connected)
sendto(12, "\0\0\0O\0M\0\0\0\0\0\0\0\0A4\213\250\377\377\377\377\0\0\0\0\0\0\0\0\202\354"..., 83, MSG_NOSIGNAL, NULL, 0) = -1 ENOTCONN (Transport endpoint is not connected)

Attaching with gdb and asking for a backtrace produced this:
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Attaching to process 17666
Reading symbols from /usr/bin/nemea/unirecfilter...Reading symbols from /usr/bin/nemea/unirecfilter...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Reading symbols from /lib64/liburfilter.so.0...Reading symbols from /lib64/liburfilter.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/liburfilter.so.0
Reading symbols from /lib64/libnemea-common.so.3...Reading symbols from /lib64/libnemea-common.so.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libnemea-common.so.3
Reading symbols from /lib64/libxml2.so.2...Reading symbols from /lib64/libxml2.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libxml2.so.2
Reading symbols from /lib64/libz.so.1...Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libtrap.so.1...Reading symbols from /lib64/libtrap.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libtrap.so.1
Reading symbols from /lib64/libssl.so.10...Reading symbols from /lib64/libssl.so.10...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libssl.so.10
Reading symbols from /lib64/libcrypto.so.10...Reading symbols from /lib64/libcrypto.so.10...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypto.so.10
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[New LWP 17675]
[New LWP 17674]
[New LWP 17673]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/liblzma.so.5...Reading symbols from /lib64/liblzma.so.5...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/liblzma.so.5
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libgssapi_krb5.so.2...Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libkrb5support.so.0...Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libselinux.so.1...Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libpcre.so.1...Reading symbols from /lib64/libpcre.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libpcre.so.1
0x00007f52e2262cfb in send () from /lib64/libpthread.so.0
Missing separate debuginfos, use: debuginfo-install nemea-modules-2.6.3-1.x86_64
(gdb) bt
#0 0x00007f52e2262cfb in send () from /lib64/libpthread.so.0
#1 0x00007f52e2d5df03 in tcpip_sender_send () from /lib64/libtrap.so.1
#2 0x00007f52e2d59740 in trap_ctx_send_flush () from /lib64/libtrap.so.1
#3 0x000000000040429d in main ()

Unfortunately by the time i had installed the debuginfo package the process had terminated, probably as a result of having the debugger attached.

I will try to produce a better backtrace if it happens again.

Please let me know if there is anything else i can provide!

Feature request: Add DNS A record for dns plugin (flow_meter)

Dear Nemea Team,
With dns plugin (flow_meter) it shows as below ( test and capture external interface of OpenWRT)
Can't get mapping between domain name <---> A record

8.8.8.8,192.168.2.51,0,0,2017-02-16T02:31:12.070,2017-02-16T02:31:12.070,0,1,0,1,16817,0,1,0,53,28110,0,0,0,17,0,0,64,"checkip.org",
8.8.8.8,192.168.2.51,0,0,2017-02-16T02:31:12.070,2017-02-16T02:31:12.070,0,1,0,1,16817,0,1,0,53,28110,0,0,0,17,0,0,64,"checkip.org",

Could you add A-record field like below for passiveDNS purpose ?
Example tcpdump -i eth1 dst port 53
09:32:00.505589 IP 192.168.111.138.45145 > 192.168.111.2.domain: 6718+ A? checkip.org. (29)
09:32:00.505759 IP 192.168.111.138.45145 > 192.168.111.2.domain: 51489+ AAAA? checkip.org. (29)
09:32:00.506371 IP 192.168.111.138.39505 > 192.168.111.2.domain: 60537+ PTR? 2.111.168.192.in-addr.arpa. (44)
09:32:00.506832 IP 192.168.111.2.domain > 192.168.111.138.45145: 6718 1/0/0 A 52.202.215.126 (45)
09:32:00.507176 IP 192.168.111.2.domain > 192.168.111.138.45145: 51489 0/0/0 (29)
09:32:00.508841 IP 192.168.111.2.domain > 192.168.111.138.39505: 60537 NXDomain 0/0/0 (44)
09:32:00.509250 IP 192.168.111.138.50612 > 192.168.111.2.domain: 43979+ PTR? 138.111.168.192.in-addr.arpa. (46)

Thanks,

flow_meter: export of ICMP flows

For ICMP flows, the Source Port should be zero, and the Destination Port number field codes ICMP message Type and Code (port = ICMP-Type * 256 + ICMP-Code).

Chevk the current state and fix if needed.

Build fails on openSUSE 15

When building on opensuse-leap 15, I'm getting the following error:

$ make
make  all-recursive
make[1]: Vstupuje se do adresáře „/home/krejci/workspace/nemea-modules“
Making all in aggregator
make[2]: Vstupuje se do adresáře „/home/krejci/workspace/nemea-modules/aggregator“
/usr/local/bin/ur_processor.sh -i ./ -o ./
make  all-am
make[3]: Vstupuje se do adresáře „/home/krejci/workspace/nemea-modules/aggregator“
  CXX      agg-aggregator.o
In file included from /usr/include/c++/7/stdlib.h:36:0,
                 from /usr/local/include/libtrap/jansson.h:12,
                 from /usr/local/include/libtrap/trap.h:57,
                 from aggregator.cpp:53:
/usr/include/c++/7/cstdlib:151:11: error: ‘::malloc’ has not been declared
   using ::malloc;
           ^~~~~~
/usr/include/c++/7/cstdlib:164:11: error: ‘::realloc’ has not been declared
   using ::realloc;
           ^~~~~~~
In file included from /usr/local/include/libtrap/jansson.h:12:0,
                 from /usr/local/include/libtrap/trap.h:57,
                 from aggregator.cpp:53:
/usr/include/c++/7/stdlib.h:65:12: error: ‘std::malloc’ has not been declared
 using std::malloc;
            ^~~~~~
/usr/include/c++/7/stdlib.h:73:12: error: ‘std::realloc’ has not been declared
 using std::realloc;
            ^~~~~~~
In file included from /usr/local/include/libtrap/trap.h:56:0,
                 from aggregator.cpp:53:
aggregator.cpp: In function ‘int main(int, char**)’:
aggregator.cpp:473:4: error: ‘realloc’ was not declared in this scope
    INIT_MODULE_INFO_STRUCT(MODULE_BASIC_INFO, MODULE_PARAMS)
    ^
/usr/local/include/libtrap/trap_module_info.h:328:10: note: in expansion of macro ‘MODULE_PARAMS’
          PARAMS(GENERATE_GETOPT_STRING) \
          ^~~~~~
aggregator.cpp:473:4: note: suggested alternative: ‘valloc’
    INIT_MODULE_INFO_STRUCT(MODULE_BASIC_INFO, MODULE_PARAMS)
    ^
/usr/local/include/libtrap/trap_module_info.h:328:10: note: in expansion of macro ‘MODULE_PARAMS’
          PARAMS(GENERATE_GETOPT_STRING) \
          ^~~~~~
aggregator.cpp:473:4: error: ‘realloc’ was not declared in this scope
    INIT_MODULE_INFO_STRUCT(MODULE_BASIC_INFO, MODULE_PARAMS)
    ^
/usr/local/include/libtrap/trap_module_info.h:328:10: note: in expansion of macro ‘MODULE_PARAMS’
          PARAMS(GENERATE_GETOPT_STRING) \
          ^~~~~~
aggregator.cpp:473:4: note: suggested alternative: ‘valloc’
    INIT_MODULE_INFO_STRUCT(MODULE_BASIC_INFO, MODULE_PARAMS)
    ^
/usr/local/include/libtrap/trap_module_info.h:328:10: note: in expansion of macro ‘MODULE_PARAMS’
          PARAMS(GENERATE_GETOPT_STRING) \
          ^~~~~~
aggregator.cpp:473:4: error: ‘realloc’ was not declared in this scope
    INIT_MODULE_INFO_STRUCT(MODULE_BASIC_INFO, MODULE_PARAMS)
    ^
/usr/local/include/libtrap/trap_module_info.h:328:10: note: in expansion of macro ‘MODULE_PARAMS’
          PARAMS(GENERATE_GETOPT_STRING) \
          ^~~~~~
aggregator.cpp:473:4: note: suggested alternative: ‘valloc’
    INIT_MODULE_INFO_STRUCT(MODULE_BASIC_INFO, MODULE_PARAMS)
    ^
/usr/local/include/libtrap/trap_module_info.h:328:10: note: in expansion of macro ‘MODULE_PARAMS’
          PARAMS(GENERATE_GETOPT_STRING) \
          ^~~~~~
aggregator.cpp:473:4: error: ‘realloc’ was not declared in this scope
    INIT_MODULE_INFO_STRUCT(MODULE_BASIC_INFO, MODULE_PARAMS)
    ^
/usr/local/include/libtrap/trap_module_info.h:328:10: note: in expansion of macro ‘MODULE_PARAMS’
          PARAMS(GENERATE_GETOPT_STRING) \
          ^~~~~~
aggregator.cpp:473:4: note: suggested alternative: ‘valloc’
    INIT_MODULE_INFO_STRUCT(MODULE_BASIC_INFO, MODULE_PARAMS)
    ^
/usr/local/include/libtrap/trap_module_info.h:328:10: note: in expansion of macro ‘MODULE_PARAMS’
          PARAMS(GENERATE_GETOPT_STRING) \
          ^~~~~~
aggregator.cpp:473:4: error: ‘realloc’ was not declared in this scope
    INIT_MODULE_INFO_STRUCT(MODULE_BASIC_INFO, MODULE_PARAMS)
    ^
/usr/local/include/libtrap/trap_module_info.h:328:10: note: in expansion of macro ‘MODULE_PARAMS’
          PARAMS(GENERATE_GETOPT_STRING) \
          ^~~~~~
aggregator.cpp:473:4: note: suggested alternative: ‘valloc’
    INIT_MODULE_INFO_STRUCT(MODULE_BASIC_INFO, MODULE_PARAMS)
    ^
/usr/local/include/libtrap/trap_module_info.h:328:10: note: in expansion of macro ‘MODULE_PARAMS’
          PARAMS(GENERATE_GETOPT_STRING) \
          ^~~~~~
aggregator.cpp:473:4: error: ‘realloc’ was not declared in this scope
    INIT_MODULE_INFO_STRUCT(MODULE_BASIC_INFO, MODULE_PARAMS)
    ^
/usr/local/include/libtrap/trap_module_info.h:328:10: note: in expansion of macro ‘MODULE_PARAMS’
          PARAMS(GENERATE_GETOPT_STRING) \
          ^~~~~~
aggregator.cpp:473:4: note: suggested alternative: ‘valloc’
    INIT_MODULE_INFO_STRUCT(MODULE_BASIC_INFO, MODULE_PARAMS)
    ^
/usr/local/include/libtrap/trap_module_info.h:328:10: note: in expansion of macro ‘MODULE_PARAMS’
          PARAMS(GENERATE_GETOPT_STRING) \
          ^~~~~~
aggregator.cpp:473:4: error: ‘realloc’ was not declared in this scope
    INIT_MODULE_INFO_STRUCT(MODULE_BASIC_INFO, MODULE_PARAMS)
    ^
/usr/local/include/libtrap/trap_module_info.h:328:10: note: in expansion of macro ‘MODULE_PARAMS’
          PARAMS(GENERATE_GETOPT_STRING) \
          ^~~~~~
aggregator.cpp:473:4: note: suggested alternative: ‘valloc’
    INIT_MODULE_INFO_STRUCT(MODULE_BASIC_INFO, MODULE_PARAMS)
    ^
/usr/local/include/libtrap/trap_module_info.h:328:10: note: in expansion of macro ‘MODULE_PARAMS’
          PARAMS(GENERATE_GETOPT_STRING) \
          ^~~~~~
aggregator.cpp:473:4: error: ‘realloc’ was not declared in this scope
    INIT_MODULE_INFO_STRUCT(MODULE_BASIC_INFO, MODULE_PARAMS)
    ^
/usr/local/include/libtrap/trap_module_info.h:328:10: note: in expansion of macro ‘MODULE_PARAMS’
          PARAMS(GENERATE_GETOPT_STRING) \
          ^~~~~~
aggregator.cpp:473:4: note: suggested alternative: ‘valloc’
    INIT_MODULE_INFO_STRUCT(MODULE_BASIC_INFO, MODULE_PARAMS)
    ^
/usr/local/include/libtrap/trap_module_info.h:328:10: note: in expansion of macro ‘MODULE_PARAMS’
          PARAMS(GENERATE_GETOPT_STRING) \
          ^~~~~~
aggregator.cpp:473:4: error: ‘realloc’ was not declared in this scope
    INIT_MODULE_INFO_STRUCT(MODULE_BASIC_INFO, MODULE_PARAMS)
    ^
/usr/local/include/libtrap/trap_module_info.h:328:10: note: in expansion of macro ‘MODULE_PARAMS’
          PARAMS(GENERATE_GETOPT_STRING) \
          ^~~~~~
aggregator.cpp:473:4: note: suggested alternative: ‘valloc’
    INIT_MODULE_INFO_STRUCT(MODULE_BASIC_INFO, MODULE_PARAMS)
    ^
/usr/local/include/libtrap/trap_module_info.h:328:10: note: in expansion of macro ‘MODULE_PARAMS’
          PARAMS(GENERATE_GETOPT_STRING) \
          ^~~~~~
aggregator.cpp:473:4: error: ‘realloc’ was not declared in this scope
    INIT_MODULE_INFO_STRUCT(MODULE_BASIC_INFO, MODULE_PARAMS)
    ^
/usr/local/include/libtrap/trap_module_info.h:328:10: note: in expansion of macro ‘MODULE_PARAMS’
          PARAMS(GENERATE_GETOPT_STRING) \
          ^~~~~~
aggregator.cpp:473:4: note: suggested alternative: ‘valloc’
    INIT_MODULE_INFO_STRUCT(MODULE_BASIC_INFO, MODULE_PARAMS)
    ^
/usr/local/include/libtrap/trap_module_info.h:328:10: note: in expansion of macro ‘MODULE_PARAMS’
          PARAMS(GENERATE_GETOPT_STRING) \
          ^~~~~~
make[3]: *** [Makefile:557: agg-aggregator.o] Chyba 1
make[3]: Opouští se adresář „/home/krejci/workspace/nemea-modules/aggregator“
make[2]: *** [Makefile:407: all] Chyba 2
make[2]: Opouští se adresář „/home/krejci/workspace/nemea-modules/aggregator“
make[1]: *** [Makefile:479: all-recursive] Chyba 1
make[1]: Opouští se adresář „/home/krejci/workspace/nemea-modules“
make: *** [Makefile:409: all] Chyba 2

Log from the configure script is here: config.log

Unirecfilter startup race

Unirecfilter sometimes gets itself into a bad state, i think it is when it is started at about the same time as all other Nemea modules.

When it gets into this state it keeps answering output ifc negotiations with trap_fmt_unknown.

A workaround that succeeds most of the time is to enter the supervisor_cli and restart just the unirecfiltermodule we run and then reloading the configuration causing downstream modules that have been automatically disabled to be reenabled and restarted.

Log files for the unirecfiltermodule we run are attached in egressfilter-logs.zip.

I might try again to produce a pull request for this issue but my initial attempts have failed.

Flow_meter TTL and TOS are 0

Check if these fields are exported.
It seems to me that they are not, this should be fixed.

json_dump.py flushing

I'm trying to use json_dump.py instead of logger (there's no other way with vportscan aggregator) and I've found that is does not flush messages at all, so I have to wait for a lot of messages to arrive before anything is written to the log file.

Could you add a file.flush() call (optional if necessary) to json_dump.py so that it can be used as a logger?

Debian package

Add necessary files and make neccessary changes to generate the deb package with NEMEA modules.

For inspiration, have a look into https://github.com/CESNET/Nemea-Framework/tree/master/libtrap/debian and https://github.com/CESNET/Nemea-Framework/blob/master/libtrap/Makefile.am#L47

Question: DIR_BIT_FIELD of interface network

Dear Nemea Team,
I used logger command for show logging from flow_meter and DIR_BIT_FIELD always is 0

May I get the direction value of incoming and outgoing traffic of interface network, like ingress and egress of netflow v9?
Thanks,

logger bug: `-o` parameter doesn't work

After the last change of logger (b0114eb), specification of output template (the -o parameter) is ignored, it always print all the fields.

The recommended way to fix this is to update the urcsv code to allow to specify two different templates - one specifying fields that should be printed out (passed to urcsv_init) and potentially different one specifying the fields of the record to print (passed to urcsv_record together with rec).

Reporter modules crash on db connection error

All reporting modules crash when I stop MongoDB (to which they are configured to write).

I get the following error:

Traceback (most recent call last):
  File "/usr/bin/nemea/hoststats2idea.py", line 195, in <module>
    arg_parser = None # (optional) Instance of argparse.ArgumentParser with some arguments defined can be passed - these arguments are then parsed from command line and passed to the conv_func)
  File "/usr/lib/python2.7/site-packages/report2idea.py", line 141, in Run
    config = Config.Config(args.config, trap = trap, warden = wardenclient)
  File "/usr/lib/python2.7/site-packages/reporter_config/Config.py", line 59, in __init__
    self.actions[i["id"]] =  MongoAction(i)
  File "/usr/lib/python2.7/site-packages/reporter_config/actions/Mongo.py", line 34, in __init__
    self.client = pymongo.MongoClient(self.host, self.port)
  File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 337, in __init__
    raise ConnectionFailure(str(e))
pymongo.errors.ConnectionFailure: could not connect to localhost:27017: [Errno 111] Connection refused
Exception AttributeError: "MongoAction instance has no attribute 'client'" in <bound method MongoAction.__del__ of <reporter_config.actions.Mongo.MongoAction instance at 0x128f830>> ignored

The expected behavior is to log the problem or be quiet about it, not crash.

unirecfilter: Allow "PROTOCOL==number" in filter

In filter specification, a value of PROTOCOL field must contain a name of the protocol (UDP, TCP and ICMP as a token, others must be encoled in quotes as a string). Numbers are not allowed ("syntax error" without any meanigful explanation is generated).

A change is needed to allow number as a PROTOCOL value.

It would also be nice to remove the special handling of UDP,TCP,ICMP and treat all protocol names the same.

nfreader: malformed IPv6 addresses in output UniRec

When logger is connected to nfreader, we get 0:2:fc00:2::1 instead of fc00:2:0:2::1.

'uint64 BYTES' in openWRT always has zero value

Dear Nemea team,
I have problem with OpenWRT as below
When I tested with commit: e115bd7

Run flow_meter in OpenWRT

Run logger in OpenWRT, it will show 192.168.11.11,10.10.10.10**,0,**0
value of 'uint64 BYTES; always zero
But if I use endiverter on Centos and connect to openWRT, then use logger in Centos, the BYTE will have value
192.168.11.11,10.10.10.10,222,0
BRs

Create a simple README.md for json_dump

json_dump.py is used to connect to trap IFC sending JSONs and to dump it into a file. Simple README should be added for better documentation.

cesnet / nemea-modules Goto Github PK

nemea-modules's Introduction

NEMEA Modules

nemea-modules's People

Stargazers

Watchers

Forkers

nemea-modules's Issues

Recommend Projects

Recommend Topics

Recommend Org