Request for more Zeek JSON log support about sof-elk HOT 6 OPEN

Progress on these noted below (comment will be updated as more are added).

These are all PR'ed and merged onto the develop branch as they are added to the list, so testing can occur against develop.

• files.log: 730472d
• ssl.log: 8762560
• x509.log: 98fb81e
• ftp.log: b8860df
• notice.log: 8683955
• weird.log: 49c4225

from sof-elk.

ok, @tileo this should be ready for testing. of particular note, be aware that I have removed TSV support. It's just too fragile and both labor- and resource- intensive to support. JSON is the way.

I've tested this with a decent variety of log files here but your eyes and inputs are most certainly appreciated.

I've added the "awaiting-validation" label here, so give this a try on develop and let me know what needs to change. if you find it's good to go, let me know here and I'll remove that label, close the issue, and stage all these updates for deployment with the new public VM in a few weeks.

from sof-elk.

I only had time for a quick check.

Test case:

• Used the FOR572 VM, so I had the Lab 3.1 netflow indexed
• Ran Zeek against the nitroba.pcap file
• Copied the supported files to /logstash/zeek

[elk_user@sof-elk zeek]$ sof-elk_clear.py -i list
The following indices are currently active in Elasticsearch:
- netflow (8,959,232 documents)

$ wc -l conn.log
4676 conn.log

[elk_user@sof-elk zeek]$ sof-elk_clear.py -i list
The following indices are currently active in Elasticsearch:
- netflow (8,963,908 documents)

8959232 + 4676 = 8,963,908

$ wc -l http.log 
4564 http.log

[elk_user@sof-elk zeek]$ sof-elk_clear.py -i list
The following indices are currently active in Elasticsearch:
- httpdlog (4,564 documents)
- netflow (8,963,908 documents)

$ wc -l files.log 
4713 files.log

[elk_user@sof-elk zeek]$ sof-elk_clear.py -i list
The following indices are currently active in Elasticsearch:
- httpdlog (4,564 documents)
- netflow (8,963,908 documents)
- zeek (4,713 documents)

$ wc -l dns.log 
1492 dns.log

[elk_user@sof-elk zeek]$ sof-elk_clear.py -i list
The following indices are currently active in Elasticsearch:
- httpdlog (4,564 documents)
- logstash (1,492 documents)
- netflow (8,963,908 documents)
- zeek (4,713 documents)

$ wc -l x509.log
27 x509.log

[elk_user@sof-elk zeek]$ sof-elk_clear.py -i list
The following indices are currently active in Elasticsearch:
- httpdlog (4,564 documents)
- logstash (1,492 documents)
- netflow (8,963,908 documents)
- zeek (4,740 documents)

4713 + 27 = 4740

$ wc -l ssl.log
136 ssl.log

[elk_user@sof-elk zeek]$ sof-elk_clear.py -i list
The following indices are currently active in Elasticsearch:
- httpdlog (4,564 documents)
- logstash (1,492 documents)
- netflow (8,963,908 documents)
- zeek (4,876 documents)

4740 + 136 = 4876

$ wc -l ftp.log 
13 ftp.log

[elk_user@sof-elk ~]$ sof-elk_clear.py -i list
The following indices are currently active in Elasticsearch:
- httpdlog (4,564 documents)
- logstash (1,492 documents)
- netflow (8,963,908 documents)
- zeek (4,889 documents)

4876 + 13 = 4889

$ wc -l notice.log 
38 notice.log

[elk_user@sof-elk zeek]$ sof-elk_clear.py -i list
The following indices are currently active in Elasticsearch:
- httpdlog (4,564 documents)
- logstash (1,492 documents)
- netflow (8,963,908 documents)
- zeek (4,927 documents)

4889 + 38 = 4927

$ wc -l weird.log 
64 weird.log

[elk_user@sof-elk zeek]$ sof-elk_clear.py -i list
The following indices are currently active in Elasticsearch:
- httpdlog (4,564 documents)
- logstash (1,492 documents)
- netflow (8,963,908 documents)
- zeek (4,991 documents)

4927 + 64 = 4991

So record counts look good, but the design decision of loading http.log to the httpdlog index, conn.log to the netflow index and dns.log to the logstash index means that using the uid, and fuid values for correlation will be harder, and you can't see all zeek logs associated to a uid, or fuid value together in Discovery.

Pivot on the uid works (although there is a warning message on the pinned filter) to the netflow dashboard:

The httpd dashboard have some issues:

And I am not sure why I can't create filters on the dashboards (this is probably not related to the Zeek JSON log parsing):

from sof-elk.

hey! not forgetting this, was just sidelined for a bit. thanks for the testing so far!! always good to get extra eyes on this. a few thoughts/points so far:

• the zeek_uid field should most certainly still be there. Do you maybe need to do the shift-browser_reload thing to re-sync Kibana to the new indices that have the additional field?
• I'm curious about the httpd dashboard you showed - that should also be all settled. I'll need to dig into that
• I think the lack of the filter creation problem might just be that you need to scroll to the top of the dashboard to make those. I think it's a somewhat new Kibana thing.
• The different indices issue is acknowledged, and there's no simple approach (except for full-on duplication of data in multiple indices, which I am REALLY opposed to doing). The http.log data is just too similar to all the other http-based evidence NOT to include alongside it, IMO. Same for conn.log and NetFlow. However the expected workflows would be to either create a new dashboard that uses two indices (like we ship in the SSH dashboard that SEC501 uses/used), or to pivot between dashboards (or indices in Discover) using the zeek_uid field pinned as a filter (when that's all ironed out, of course).

I'll get to more testing with this tomorrow as well. I've been mostly working on ECS conversion, so I'll need to re-tool to test this on develop again.

from sof-elk.

follow-up after some testing:

• zeek_uid is present in both NetFlow from conn.log and HTTP records from http.log. I am pretty sure you ran into the shift-reload browser caching problem.
  • I was able to create a filter using the zeek_uid field on both the NetFlow and HTTPD dashboards, as well as pin it across applications
• all visualization panels on the http dashboard were working fine

from sof-elk.

I did find some errors in the dns dashboard during testing/validation on these. those should be fixed as of e54693f

from sof-elk.