Getting Fortidragon to fly with rsyslog about fortinet-2-elasticsearch HOT 13 OPEN

I am giving it another try, as time allows now, however still can't get it to work. I think there is some documentation missing for the Elastic/Kibana Setup.

from fortinet-2-elasticsearch.

I think we should follow datastreams index strategy

instead of

searchIndex="logstash-index"

I think we can use

searchIndex="logs-fortinet.fortigate"

Also, that could reuse the index templates configs.

Take into account that I split it even further, by type. if you can accomplish that with Rsyslog, that would make it completely transparent for current index templates.

from fortinet-2-elasticsearch.

I think i followed https://chabik.com/2019/03/rsyslog-to-elasticsearch/ any idea on what to change in the config? I mean still if i understand it corretly it should at least find some feeded values from the elastic index, as rsyslog seems to send its stuff into when i activate it.

from fortinet-2-elasticsearch.

Ok, i must be doing something completely wrong on the rsyslog rule. i am trying to figure out how to change the rule to not use logstash but do directly omelasticsearch

from fortinet-2-elasticsearch.

looks like https://gist.github.com/pando85/75f06fb9a3b67788342e700fa8365674 could help

from fortinet-2-elasticsearch.

I am giving it another try, as time allows now, however still can't get it to work. I think there is some documentation missing for the Elastic/Kibana Setup.

I am also checking the connection to Grafana Loki, so I am considering using Promtail for that, which might be more efficient because it is not java. We will see how that goes, I will keep eveybody updated.

from fortinet-2-elasticsearch.

My main problem at the moment (as i am not yet experienced at Elastic/Kibana) is to get all the component templates inside of Elastic, is there a "nice" way to import them by the files in the repo?

best Regards

from fortinet-2-elasticsearch.

mmmm nice is manual, it takes a while (30 min aprox) but you only got to do it once.

Just go into devtools inside kibana and load component templates both from Elastic ECS and FortiDragon specific. Do it manually one by one:

https://github.com/enotspe/fortinet-2-elasticsearch#on-kibana

from fortinet-2-elasticsearch.

Tried already, if i try to add /master/index%20templates/component%20templates/logs-fortinet.forticlient%40ilm.json for example it seems to be incomplete and it claims about missing PUT or similar.

from fortinet-2-elasticsearch.

You have to load the ilm policies first

from fortinet-2-elasticsearch.

Tried already, if i try to add /master/index%20templates/component%20templates/logs-fortinet.forticlient%40ilm.json for example it seems to be incomplete and it claims about missing PUT or similar.

thx for your help, i created for example the "ilm policy fortigate.traffic" however still as mentioned above, cant do that example in the dev console:

{
"template": {
"mappings": {
"properties": {
"vcluster": {
"type": "long"
},
"iaid": {
"ignore_above": 1024,
"type": "keyword"
},
"infectedfilelevel": {
"type": "long"
},
"domainctrlip": {
"type": "ip"
},
"sentdelta": {
"type": "long"
},
"dstport": {
"type": "long"
},
"channel": {
"type": "long"
},
"mgmtcnt": {
"type": "long"
},
"sessionid": {
"ignore_above": 1024,
"type": "keyword"
},
"remote": {
"type": "ip"
},
"type": {
"type": "constant_keyword"
},
"ha_group": {
"type": "long"
},
"aptype": {
"type": "long"
},
"remport": {
"type": "long"
},
"virusid": {
"ignore_above": 1024,
"type": "keyword"
},
"translationid": {
"ignore_above": 1024,
"type": "keyword"
},
"cfgtid": {
"ignore_above": 1024,
"type": "keyword"
},
"scanned": {
"type": "long"
},
"countff": {
"type": "long"
},
"eapolcnt": {
"type": "long"
},
"portend": {
"type": "long"
},
"dstip": {
"type": "ip"
},
"nat": {
"type": "ip"
},
"identifier": {
"type": "long"
},
"vwpvlanid": {
"ignore_above": 1024,
"type": "keyword"
},
"newchannel": {
"type": "long"
},
"ip": {
"type": "ip"
},
"srcserver": {
"type": "long"
},
"fams_pause": {
"type": "long"
},
"ovrdid": {
"ignore_above": 1024,
"type": "keyword"
},
"received": {
"type": "long"
},
"botnetip": {
"type": "ip"
},
"vrf": {
"type": "long"
},
"centralnatid": {
"ignore_above": 1024,
"type": "keyword"
},
"scantime": {
"type": "long"
},
"chassisid": {
"ignore_above": 1024,
"type": "keyword"
},
"filteridx": {
"type": "long"
},
"nextstat": {
"type": "long"
},
"setuprate": {
"type": "long"
},
"countweb": {
"type": "long"
},
"remip": {
"type": "ip"
},
"alarmid": {
"ignore_above": 1024,
"type": "keyword"
},
"apstatus": {
"type": "long"
},
"assigned": {
"type": "ip"
},
"ha-prio": {
"type": "long"
},
"locip": {
"type": "ip"
},
"highcount": {
"type": "long"
},
"domainctrlauthstate": {
"type": "long"
},
"quotaused": {
"type": "long"
},
"stacount": {
"type": "long"
},
"countssh": {
"type": "long"
},
"epoch": {
"type": "long"
},
"filesize": {
"type": "long"
},
"oldchannel": {
"type": "long"
},
"sentpkt": {
"type": "long"
},
"countssl": {
"type": "long"
},
"local": {
"type": "ip"
},
"from_vcluster": {
"type": "long"
},
"ddnsserver": {
"type": "ip"
},
"policyid": {
"ignore_above": 1024,
"type": "keyword"
},
"sess_duration": {
"type": "long"
},
"sysuptime": {
"type": "long"
},
"encrypt": {
"type": "long"
},
"noise": {
"type": "long"
},
"oldchassisid": {
"ignore_above": 1024,
"type": "keyword"
},
"disklograte": {
"type": "long"
},
"countapp": {
"type": "long"
},
"infected": {
"type": "long"
},
"lanin": {
"type": "long"
},
"rssi": {
"type": "long"
},
"radioidclosest": {
"type": "long"
},
"column": {
"type": "long"
},
"portbegin": {
"type": "long"
},
"countdns": {
"type": "long"
},
"session_id": {
"ignore_above": 1024,
"type": "keyword"
},
"domainctrlauthtype": {
"type": "long"
},
"shapingpolicyid": {
"ignore_above": 1024,
"type": "keyword"
},
"countips": {
"type": "long"
},
"wanout": {
"type": "long"
},
"countemail": {
"type": "long"
},
"event_id": {
"ignore_above": 1024,
"type": "keyword"
},
"oldslot": {
"type": "long"
},
"quotamax": {
"type": "long"
},
"stage": {
"type": "long"
},
"appid": {
"ignore_above": 1024,
"type": "keyword"
},
"passedcount": {
"type": "long"
},
"dst_port": {
"type": "long"
},
"lease": {
"type": "long"
},
"rcvddelta": {
"type": "long"
},
"tranport": {
"type": "long"
},
"rcvdbyte": {
"type": "long"
},
"attackid": {
"ignore_above": 1024,
"type": "keyword"
},
"dstserver": {
"type": "long"
},
"shaperdroprcvdbyte": {
"type": "long"
},
"srcip": {
"type": "ip"
},
"newchassisid": {
"ignore_above": 1024,
"type": "keyword"
},
"eventtime": {
"type": "long"
},
"pid": {
"ignore_above": 1024,
"type": "keyword"
},
"used": {
"type": "long"
},
"vwlid": {
"ignore_above": 1024,
"type": "keyword"
},
"radioiddetected": {
"type": "long"
},
"xid": {
"ignore_above": 1024,
"type": "keyword"
},
"countav": {
"type": "long"
},
"crscore": {
"type": "long"
},
"newslot": {
"type": "long"
},
"limit": {
"type": "long"
},
"infectedfilesize": {
"type": "long"
},
"qtypeval": {
"type": "long"
},
"processtime": {
"type": "long"
},
"signal": {
"type": "long"
},
"auditid": {
"ignore_above": 1024,
"type": "keyword"
},
"assignip": {
"type": "ip"
},
"vulnid": {
"ignore_above": 1024,
"type": "keyword"
},
"audittime": {
"type": "long"
},
"transid": {
"ignore_above": 1024,
"type": "keyword"
},
"cfgtxpower": {
"type": "long"
},
"groupid": {
"ignore_above": 1024,
"type": "keyword"
},
"count": {
"type": "long"
},
"cpu": {
"type": "long"
},
"countdlp": {
"type": "long"
},
"transip": {
"type": "ip"
},
"freediskstorage": {
"type": "long"
},
"suspicious": {
"type": "long"
},
"to_vcluster": {
"type": "long"
},
"vcluster_member": {
"type": "long"
},
"craction": {
"type": "long"
},
"mtu": {
"type": "long"
},
"lowcount": {
"type": "long"
},
"disk": {
"type": "long"
},
"unit": {
"type": "long"
},
"wanin": {
"type": "long"
},
"port": {
"type": "long"
},
"used_for_type": {
"type": "long"
},
"proto": {
"type": "long"
},
"tunnelid": {
"ignore_above": 1024,
"type": "keyword"
},
"tunnelip": {
"type": "ip"
},
"countwaf": {
"type": "long"
},
"lanout": {
"type": "long"
},
"eventid": {
"ignore_above": 1024,
"type": "keyword"
},
"policy_id": {
"ignore_above": 1024,
"type": "keyword"
},
"fazlograte": {
"type": "long"
},
"slot": {
"type": "long"
},
"domainfilteridx": {
"type": "long"
},
"urlfilteridx": {
"type": "long"
},
"incidentserialno": {
"type": "long"
},
"duration": {
"type": "long"
},
"total": {
"type": "long"
},
"mem": {
"type": "long"
},
"rate": {
"type": "long"
},
"cat": {
"type": "long"
},
"checksum": {
"ignore_above": 1024,
"type": "keyword"
},
"countcifs": {
"type": "long"
},
"live": {
"type": "long"
},
"criticalcount": {
"type": "long"
},
"domainctrlprotocoltype": {
"type": "long"
},
"sentbyte": {
"type": "long"
},
"receivedsignature": {
"type": "long"
},
"radioid": {
"ignore_above": 1024,
"type": "keyword"
},
"mediumcount": {
"type": "long"
},
"transport": {
"type": "long"
},
"shaperdropsentbyte": {
"type": "long"
},
"malform_data": {
"type": "long"
},
"src_port": {
"type": "long"
},
"trueclntip": {
"type": "ip"
},
"locport": {
"type": "long"
},
"opertxpower": {
"type": "long"
},
"tranip": {
"type": "ip"
},
"serial": {
"type": "long"
},
"shaperperipdropbyte": {
"type": "long"
},
"expectedsignature": {
"type": "long"
},
"srcport": {
"type": "long"
},
"category": {
"type": "long"
},
"age": {
"type": "long"
},
"gateway": {
"type": "ip"
},
"rcvdpkt": {
"type": "long"
},
"totalsession": {
"type": "long"
},
"jitter": {
"type": "long"
},
"latency": {
"type": "long"
}
}
}
},
"_meta": {
"fortigate_version": "6.2.2"
}
}

from fortinet-2-elasticsearch.

We can schedule a time for to help you online. Let me know if that would fine

from fortinet-2-elasticsearch.