corelight / json-streaming-logs Goto Github PK

looks like like 58 needs to be "zeek_init()"

Running the default zeek image and the json-streaming-logs package results in an error logged. I do not know if this is a docker issue, zeek issue, or something that can be fixed in json-streaming-logs. Please close this out if it's not something that should be posted here.

Error:

{
  "_path": "reporter",
  "_system_name": "ubuntu",
  "_write_ts": "2022-10-23T14:40:00.006577Z",
  "level": "Reporter::ERROR",
  "location": "/usr/local/zeek/share/zeek/site/json-streaming-logs/./main.zeek, line 73",
  "message": "cannot rename file '/opt/logs//json_streaming_files-22-10-23_14.39.20.log' to 'json_streaming_files.1.log': Invalid cross-device link (rename(JSONStreaming::info$fname, JSONStreaming::info$path + .1.log))",
  "ts": "2022-10-23T14:40:00.006577Z"
}

Image Name: zeekurity/zeek:latest

I'm finding an error where the process watching the logs is not smart enough to understand the file was renamed and rather re-reads the entire file when it is renamed from .log to .1.log.

It might be better to initially start the file as json_streaming_conn_{ts}.log and on roll over a new file is created with the new timestamp. After N files, it starts to delete the older timestamps(in order).

This way the newest file is always the most recent timestamp, and the files themselves are never actually renamed but rather just stop being written to.

I hope that makes sense. I'll try to hack it in and if it works properly will submit a pull request.

warning in /usr/local/zeek/spool/installed-scripts-do-not-touch/site/packages/./json-streaming-logs/./main.zeek, line 135: use of out-of-scope local JSONStreaming::filt deprecated; move declaration to outer scope

		{
		Log::add_filter(stream, filt);
		}

Not sure what this issue or error is, but when running zeek with this plugin I get a single line entry in stderr.log that says this is deprecated.

Looks like the issue is related to how filt is defined here but I can't really see anything wrong with it.

for ( [stream, filt] in new_filters )

From the README:

Logs are named in such a way that a glob in your log shipper configuration should be able to easily match all of the logs. Each log will have a prefix of json_streaming_ so that the http log would have the full name of json_streaming_http.log

In some configurations the typical wildcard match on the json_streaming_ logs results in duplicate event collection by the shipper. For a given event, the log is read once in the current output file, and then one time additionally for each rotated log file. It seems like by default rotate_logs will create at least one rotated file (.1.log), even if JSONStreaming::extra_files is set to 0.

This shows a unique event that only occurred once but is picked up repeatedly at each rotation interval by the shipper (in this case Fluentd/td-agent):

The Fluentd configuration, with the path option wildcard matching the log files:

<source>
    @type tail
    @id zeek_json
    @label @zeek
    tag zeek.*
    path /nsm/zeek/logs/current/json_streaming_*.log
    pos_file /var/log/td-agent/tmp/zeek_json.pos
    <parse>
        @type json
    </parse>
</source>

What is the best course of action here? Is this a case where json-streaming-logs should support giving the output file a different file name that can be uniquely matched by the given glob (and not the extra files), or is this a situation that a log shipper should be able to handle better? Fluentd documentation does state:

You should not use * with log rotation because it may cause the log duplication. In this case, you should separate in_tail plugin configuration.

...but at the same time, Zeek outputs several logs and it's not clear up front which log file may appear at some point, so a glob approach is probably the most correct. If the plugin could be guaranteed to output a single file that wouldn't be picked up again at rotation, that seems the most predictable.

When running this script on a PCAP, the value for _write_ts will be the timestamp from the time of the packet in the PCAP and not the current time the log data is being written. According to the docs, network_time should be used to get the time from the last processed packet. Instead, because the log file write time is not related to the time of the traffic, current_time should be used.

Any interest in a PR for an optional binary format like https://amzn.github.io/ion-docs or JSONB to reduce bandwidth?

Zstandard/Parquet aggregation is probably out of scope because that would be the job of the log shipper.