Need plugin for Fortinet FortiGate logs about enterprise-log-search-and-archive HOT 10 CLOSED

Here are the patterndb.xml entries which will parse fields from your log 
entries, assuming that the field order is always the same:

<ruleset name="fortinet_url" id='21'>
        <pattern>kernel</pattern>
        <rules>
            <rule provider="ELSA" class='21' id='21'>
                <patterns>
                    <pattern>date=@ESTRING:: @time=@ESTRING:: @devname=@ESTRING:: @device_id=@ESTRING:: @log_id=@ESTRING:: @subtype=@ESTRING:: @type=webfilter pri=@ESTRING:: @vd=@ESTRING:: @policyid=@ESTRING:: @identidx=@ESTRING:: @serial=@ESTRING:: @user=@ESTRING:s0: @group=@ESTRING:s1: @src=@IPv4:i0:@ sport=@ESTRING:i1: @src_port=@ESTRING:: @src_int=@ESTRING:: @dst=@IPv4:i2:@ dport=@ESTRING:i3: @dst_port=@ESTRING:: @dst_int=@ESTRING:: @service=@ESTRING:s2: @hostname=@ESTRING:s3: @profiletype=@ESTRING:: @profile=@ESTRING:: @status=@ESTRING:s4: @req_type=@ESTRING:: @url=@ESTRING:s5: @method=@ESTRING:: @class=@ESTRING:: @cat=@ESTRING:i4: @cat_desc=@QSTRING::""@ carrier_ep=@ESTRING:: @msg=@QSTRING::""@ class_desc=@ESTRING:: @profilegroup=</pattern>
                </patterns>
                <examples>
                    <example>
                        <test_message program="kernel">date=2012-02-10 time=11:27:01 devname=CUSTID01-SITEID-FW device_id=FG100C999999999 log_id=13312 subtype=ftgd_allow type=webfilter pri=notice vd=VDOM policyid=44 identidx=1 serial=369298248 user=USER group=AD/GROUP src=10.1.2.3 sport=2163 src_port=2163 src_int=INT dst=4.3.2.1 dport=80 dst_port=80 dst_int=WAN service=http hostname=col.stb.s-msn.com profiletype=Webfilter_Profile profile=PROFILE status=passthrough req_type=referral url=/i/79/65F987C952BDA0E84AE52464ADD59.jpg method=domain class=0 cat=41 cat_desc="Search Engines and Portals" carrier_ep=N/A msg="URL belongs to an allowed category in policy" class_desc=N/A profilegroup=N/A</test_message>
                        <test_values>
                            <test_value name="i0">10.1.2.3</test_value>
                            <test_value name="i1">2163</test_value>
                            <test_value name="i2">4.3.2.1</test_value>
                            <test_value name="i3">80</test_value>
                            <test_value name="s0">USER</test_value>
                            <test_value name="s1">AD/GROUP</test_value>
                            <test_value name="s2">http</test_value>
                            <test_value name="s3">col.stb.s-msn.com</test_value>
                            <test_value name="s4">passthrough</test_value>
                            <test_value name="s5">/i/79/65F987C952BDA0E84AE52464ADD59.jpg</test_value>
                            <test_value name="i4">41</test_value>
                        </test_values>
                    </example>
                </examples>
            </rule>
        </rules>
    </ruleset>
    <ruleset name="fortinet_traffic" id='22'>
        <pattern>kernel</pattern>
        <rules>
            <rule provider="ELSA" class='22' id='22'>
                <patterns>
                    <pattern>date=@ESTRING:: @time=@ESTRING:: @devname=@ESTRING:: @device_id=@ESTRING:: @log_id=@ESTRING:: @subtype=@ESTRING:: @type=traffic pri=@ESTRING:: @vd=@ESTRING:: @dir_disp=@ESTRING:: @tran_disp=@ESTRING:: @src=@IPv4:i0:@ srcname=@ESTRING:: @src_port=@NUMBER:i1:@ dst=@IPv4:i2:@ dstname=@ESTRING:: @dst_port=@NUMBER:i3:@ tran_ip=@ESTRING:: @tran_port=@ESTRING:: @service=@ESTRING:: @proto=@NUMBER:i4@ app_type=@ESTRING:: @duration=@NUMBER:i5@ rule=@ESTRING:: @policyid=@ESTRING:: @identidx=@ESTRING:: @sent=@ESTRING:: @rcvd=@ESTRING:: @shaper_drop_sent=@ESTRING:: @shaper_drop_rcvd=@ESTRING:: @perip_drop=@ESTRING:: @sent_pkt=@ESTRING:: @rcvd_pkt=@ESTRING:: @src_int=@ESTRING:: @dst_int=@ESTRING:: @SN=@ESTRING:: @app=@ESTRING:: @app_cat=@ESTRING:: @carrier_ep=@ESTRING:: @vpn=@ESTRING:: @status=@ESTRING:: @user=@ESTRING:: @group=@ESTRING:: @shaper_sent_name=@ESTRING:: @shaper_rcvd_name=@ESTRING:: @perip_name</pattern>
                </patterns>
                <examples>
                    <example>
                        <test_message program="kernel">date=2012-02-10 time=11:27:01 devname=CUSTID01-SITEID-FW device_id=FGT80C9999999999 log_id=2 subtype=allowed type=traffic pri=notice vd=VDOM dir_disp=org tran_disp=snat src=10.1.2.3 srcname=10.1.2.3 src_port=53624 dst=4.3.2.2 dstname=4.3.2.2 dst_port=80 tran_ip=5.4.3.2 tran_port=49648 service=80/tcp proto=6 app_type=N/A duration=120 rule=49 policyid=49 identidx=0 sent=1221 rcvd=2062 shaper_drop_sent=0 shaper_drop_rcvd=0 perip_drop=0 sent_pkt=7 rcvd_pkt=6 src_int=INT dst_int=WAN SN=16349534 app=N/A app_cat=N/A carrier_ep=N/A vpn=N/A status=accept user=N/A group=N/A shaper_sent_name=N/A shaper_rcvd_name=N/A perip_name=N/A</test_message>
                        <test_values>
                            <test_value name="i0">10.1.2.3</test_value>
                            <test_value name="i1">53624</test_value>
                            <test_value name="i2">4.3.2.2</test_value>
                            <test_value name="i3">80</test_value>
                            <test_value name="i4">6</test_value>
                            <test_value name="i5">120</test_value>
                        </test_values>
                    </example>
                </examples>
            </rule>
        </rules>
    </ruleset>

I will add it to the distributed patterndb.xml, so it should be ready to 
download from SVN.

Before updating the patterndb.xml, you will need to add classes to the database 
on each node.  I have included this code in the schema.sql, but it is commented 
out so as not to clutter the config for those who don't use Fortinet.  

First, we add the class:
INSERT INTO classes (id, class, parent_id) VALUES(21, "FORTINET_URL", 0);
INSERT INTO classes (id, class, parent_id) VALUES(22, "FORTINET_TRAFFIC", 0);

Then we add the fields not already present (this is also in the schema.sql now):
INSERT INTO fields (field, field_type, pattern_type) VALUES ("group", "string", 
"QSTRING");
INSERT INTO fields (field, field_type, pattern_type) VALUES ("status", 
"string", "QSTRING");

Then we map the fields to the class and the "i0/s0" column names by field_order 
(in the schema, but commented out);

INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_URL"), (SELECT id FROM fields 
WHERE field="srcip"), 5);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_URL"), (SELECT id FROM fields 
WHERE field="srcport"), 6);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_URL"), (SELECT id FROM fields 
WHERE field="dstip"), 7);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_URL"), (SELECT id FROM fields 
WHERE field="dstport"), 8);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_URL"), (SELECT id FROM fields 
WHERE field="user"), 11);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_URL"), (SELECT id FROM fields 
WHERE field="group"), 12);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_URL"), (SELECT id FROM fields 
WHERE field="service"), 13);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_URL"), (SELECT id FROM fields 
WHERE field="site"), 14);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_URL"), (SELECT id FROM fields 
WHERE field="status"), 15);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_URL"), (SELECT id FROM fields 
WHERE field="uri"), 16);

INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_TRAFFIC"), (SELECT id FROM 
fields WHERE field="srcip"), 5);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_TRAFFIC"), (SELECT id FROM 
fields WHERE field="srcport"), 6);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_TRAFFIC"), (SELECT id FROM 
fields WHERE field="dstip"), 7);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_TRAFFIC"), (SELECT id FROM 
fields WHERE field="dstport"), 8);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_TRAFFIC"), (SELECT id FROM 
fields WHERE field="proto"), 9);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_TRAFFIC"), (SELECT id FROM 
fields WHERE field="conn_duration"), 10);

Restart syslog-ng, update patterndb.xml, and your logs should now be parsed in 
the correct fields.  They should show up in the "Add Term" menu on the web 
interface.

Thanks!  As soon as I resolve my other issue, I will let you know how this 
works out.

I have this somewhat working and the FORTINET_URL filter produces results but I 
see nothing for the FORTINET_TRAFFIC class.

I do get an error when starting syslog-ng though:

Starting syslog-ng
Unknown parser type specified; type='subtype='

I checked and both FORTINET_URL and FORTINET_TRAFFIC have @subtype=@ESTRING:: 
so do not know which one it does not like.

Also, what if the field order changes slightly?  Does it completely break the 
pattern recognition?

Hm, I can't reproduce the issue you're seeing.  Are you sure there are no line 
breaks in the patterndb.xml file and that it is exactly as it is in the 
repository?

If the field order changes, the pattern will probably stop working.  At best, 
the wrong fields will be extracted.  If you don't care about fields, then there 
are ways to make the parser more resilient to changing patterns.

I just did a update-from-cvs.sh and now when I restart syslog-ng I do not get 
the error.

It seems that different versions of firmware have the field order slightly 
different so we may miss on a few log matching.  I will check the logs in a few 
minutes after the restart to see if both of these are now matching like they 
should.

Ok, let me know if it changes as there are lots of ways we can account for it, 
but I'll need specific examples.

I have FORTINET_TRAFFIC and FORTINET_URL now after the update from CVS.  I will 
watch for mismatched log entries and let you know.

Hi, i have the changes in the database and patterndb.xml but the logs of my 
fortinet is not parsed, when i try the test with pdbtool i receive this message:

PROGRAM=fortinet_traffic
.classifier.class=unknown
TAGS=.classifier.unknown

the pattern is:

<pattern>fortinet_traffic</pattern>
                <rules>
                        <rule class='22' id='22'>
                                <patterns>
                                        <pattern>time=@ESTRING:: @devname=@ESTRING:: @device_id=@ESTRING:: @log_id=@ESTRING:: @type=traffic subtype=@ESTRING:: @pri=@ESTRING:: @vd=@ESTRING:: @src=@IPv4:i0:@ src_port=@NUMBER:i1:@ src_int=@QSTRING::""@ dst=@IPv4:i2:@ dst_port=@NUMBER:i3:@ dst_int=@QSTRING:: @SN=@ESTRING:: @status=@ESTRING:: @policyid=@ESTRING:: @dst_country=@QSTRING::""@ src_country=@QSTRING::"" @service=@ESTRING:: @proto=@NUMBER:i4:@ duration=@NUMBER:i5: @sent=@ESTRING:: @rcvd=@ESTRING:: @msg</pattern>

This is the test:

pdbtool match -p /opt/elsa/node/conf/patterndb.xml -P fortinet_traffic -M 
"10:42:43 devname=FORTIVM02 device_id=FGVM01XXXXXXX log_id=0038000007 
type=traffic subtype=other pri=warning vd=X_VDOM src=1.1.1.1 src_port=1037 
src_int="X_MPLS" dst=2.2.2.2 dst_port=514 dst_int="X_LAN" SN=XXXXXX status=deny 
policyid=0 dst_country="Reserved" src_country="Reserved" service=SYSLOG 
proto=17 duration=0 sent=0 rcvd=0 msg="Denied by forward policy check""

This is the result:

PROGRAM=fortinet_traffic
.classifier.class=unknown
TAGS=.classifier.unknown


Any ideas?

Your messages are coming in without a program tag on them, so the first
part of the date is getting interpreted as the program name.  You need to
change your syslog-ng config to add a special source for your Fortinet
firewalls and set the flags(no-parse)
and program_override("fortinet_traffic") setting for them.


On Wed, Aug 28, 2013 at 2:27 PM, <
[email protected]> wrote: