arunk-s / mig Goto Github PK

Currently the messages are written to jsonBuffChan which is a buffered channel, so if the dispatch function is busy, it'll fill the buffer, and if the buffer is filled the messages will be skipped from dispatching.
The ideal condition will be to find a good buffer limit so that if the dispatch function fail to send a message, it can take time to retry.

It should be possible that the module forwards the audit events to MIG which stores them in a queue and relays them further to any agent.

It should be configurable such that it is possible to dispatch audit events to a rabbitmq agent.

Currently the module is using gozdef library to send audit events to a mozdef url. But it should be feasible to send audit events to any HTTP API. The decision of using gozdef should be indicated properly in docs and it should be made clear what other options can work.

Design the buffer in mig audit module which are used to dispatch events to outputs in such a fashion that it drops the least recent message if the buffer is full.
[currently] the buffer is a channel buffer and if the buffer is full incoming messages will be dropped.

See previous discussion here: mozilla@25c6d6a#r71168607
Quoting the docs:

When a scan stops, the reader may have advanced arbitrarily far past the last token. Programs that need more control over error handling or large tokens, or must run sequential scans on a reader, should use bufio.Reader instead

If any issue appears while using scanner we should consider using reader. Also scanner includes a caveat that by defaults it splits the input only after a \n symbol. ( \n is the default token, unless we provide our own filter function). So the persistent module should always end their messages by a \n.

Fixing and testing libaudit-go: https://github.com/mozilla/libaudit-go/commits?author=arunk-s

Addition of mig-audit module: mozilla#253

Run the module directly: https://gist.github.com/arunk-s/a5814d7faded57420e35338514ce38d2

It should be configurable such that it is possible to dispatch audit events to a https server that accepts post response (mozdef format). Further extended to dispatch to an existing mozdef server.

One that could be desirable before shutting down the module is to properly clean up queues, wait for go-routines to finish up and then close. Currently we are directly crashing the module on receiving the termination signal.