for syslog data, create an array of IPs so a search of "ip:a.b.c.d" will return records of any type that contain that IP address

Change all [foo] =~ "regex" strings to [foo] =~ /regex/

add attrib header to each file as well

move inputs to separate file
use integer prefixes to order files

elasticsearch (and maybe logstash/kibana?) logs automatically rotate - remove them from logrotate.d

Hi,

I trying downloading, the VM, on

The latest version of the VM itself is available here: http://for572.com/sof-elk-vm

but no respond

https://github.com/logstash-plugins/logstash-filter-tld

use post-merge to run the load_all_dashboards.sh script after pulling an update.

Request to have logstash-filter-elasticsearch plugin installed into the SOF-ELK vm. It allows for querying Elasticsearch within the filter section of Logstash config files.

all relevant fields need to have IP addresses pushed to the [ips] string for later array-ifying

when loading asn in post-processing via geoip, push the ASN number directly into the original netflow fields and remove from the geo-specific fields

Would it be possible to include the logstash-translate plugin into SOF-ELK? I use this for many of my configuration files. It also requires the dictionaries folder in /etc/logstash although we can change the location of this. To install use:

sudo /opt/logstash/bin/plugin install logstash-filter-translate

sudo mkdir /etc/logstash/dictionaries
cd /etc/logstash/dictionaries
sudo wget https://github.com/SMAPPER/Logstash-Configs/raw/master/dictionaries/iana_protocols.yaml
sudo wget https://github.com/SMAPPER/Logstash-Configs/raw/master/dictionaries/iana_services.yaml
sudo wget https://github.com/SMAPPER/Logstash-Configs/raw/master/dictionaries/services.yaml
sudo wget https://github.com/SMAPPER/Logstash-Configs/blob/master/dictionaries/tcp_flags.yaml

I use it for a handful of my configfiles. I am testing the current SOF-ELK with my configfiles and noticed Logstash failed to run because of this.

eg /configfiles-UNSUPPORTED/foousername/
(update the readme file(s) accordingly.)

https://www.centos.org/forums/viewtopic.php?t=49942

"sudo loadkeys uk"

0xxx = inputs
0[0-4]xx = forensics
0[5-9]xx = cyber defense
9xxx = output
0[0-4]xx = forensics
0[5-9]xx = cyber defense

• What are the classification buckets we want to use for filter/parse/enrich segments? Below is an option, but does this make sense?

1xxx = network forensic
2xxx = disk forensic
3xxx = endpoint
4xxx = service logs
6xxx = siem

create a supporting script that will delete items in ES via the path (possibly via other field name/values)

https://github.com/rough007/CDQR

should be removed if empty

can we pull TLD from domain data? Would be useful for pivoting to other intel sources, etc.

use the 'useragent' filter to do this

store all IP addresses as IPs instead of strings

site plugins are not supported in ES 5.

I see that sof-elk can accept netflow data. Do you think it will ever be able to accept Argus flows as well?

possibly use bulk insert instead of logstash parser?

How do you feel about this proposal for field names?

Field name standards (always follow):

1. Only use lower case characters (“first_name” instead of “FirstName”)
2. Avoid special characters except underscores (“first_name” instead of “first name”)
3. Use underscores to separate words in a field name (“destination_port” instead of “destinationport”)
4. Whenever possible, rename field names with the same purpose to one field name (IPAddress, IP, ip, should be consolidated to ip)
5. Due to individuals abbreviating differently, do not use appreviations (“source_port” instead of “src_port”)
6. Always use singular forms not plural (“message” instead of “messages”)
7. Use proper spelling of words

Field name guidelines (try to follow):

1. Use present tense unless field describes historical information (Example: end of connection recording “bytes_received”)
2. Always use singular forms not plural (“message” instead of “messages”)
   a. Exception: When describing something that is past tense and the expectation is for multiple values (“bytes_received” instead of “byte_received”)
3. Where possible rename fields to match consistent names so long as renaming the field does not cause the event to lose context (Example: “unauthorized_user” may be able to be renamed to “user” if the only event that contains the field “unauthorized_user” has another field that provides the context of a failed login)

the following doesn't parse with the syslog input. possibly related to () in the process name?
<22>Dec 30 20:21:32 systemname deliver(username): msgid=<[email protected]>: saved mail to mailbox.submailbox
<77>Dec 30 20:01:01 systemname run-parts(/etc/cron.hourly)[2284 starting 0yum-hourly.cron

not sure what's up with the following line, though - possibly the space in the "programname" position?
<22>Dec 30 20:08:41 systemname2 spf filter[15569]: SPF status: PASS

https://github.com/elasticfence/kiBrand

create a tag to indicate whether an http log entry is a "page" or a "non-page" resource. possibly use awstats.pl's list of extensions as a default?

before parsing anything, keep original message - probably in a non-analyzed string

define how each branch is used. oh yeah - also create some docs

HTTPD log entries such as:
domain.org 1.2.3.4 - [email protected] [02/Jan/2015:03:05:30 +0000] "POST /autodiscover/autodiscover.xml HTTP/1.1" 302 - "-" "RIM-Passport-SQW100-1/10.3.0.1418"

instead of hitting raw -* indices, use saved search for all visualizations

input:
Feb 10 01:42:32 quaff fail2ban.filter[3631]: WARNING Determined IP using DNS Lookup: ec2-54-93-46-228.eu-central-1.compute.amazonaws.com = ['54.93.46.228']

results:
syslog_hostname: fail2ban.filter,WARNING Determined IP using DNS Lookup
message: ec2-54-93-46-228.eu-central-1.compute.amazonaws.com = ['54.93.46.228']

Need to handle this input better- the (may be forged) gets caught in the "smtp_relay" field from the kv {} filter.
Feb 17 22:56:38 quaff sendmail[10971]: t1HMubqj010971: from=<[email protected]>, size=0, class=0, nrcpts=0, bodytype=8BITMIME, proto=ESMTP, daemon=MTA, relay=z-y-x-w.static.reverse.lstn.net [w.x.y.z] (may be forged)

does not allow underscores in query/response due to the IPORHOST semantic
also seems there are some replies including the "/" character, e.g. 114.178.112.212.in-addr.arpa. -> 114.96/27.178.112.212.in-addr.arpa.

parse inputs via syslog and filebeat (maybe via raw JSON socket?)

send to separate index (carbonblack-*), so needs index-mapping, kibana field template, etc. (NOTE: Need a complete(ish) list of all fields CB can output, data types, etc)
add the new index to es_nuke, etc.
probably needs dashboards

what about logstash ingest enrichments for IP address, etc?

Import nfdump lines without the exporter IP set. Force to 0.0.0.0

create a visualization for HTTP response codes on a time series (like the current one for HTTP request methods)