Comments (10)
duanehoward
commented on May 2, 2024
@satta
I'm failing to find good references in the current Suricata documentation around support for keywords icmpv6, ftp-data etc for protocols.
Can you provide a quick pointer in the source to where these other protocols are defined (probably an issue should be opened with OISF to update the docs to make them complete as well).
Additionally can you give me an example of where noalert isn't working as expected? Should be simple enough to add a test and iterate on that.
from gonids.
satta
commented on May 2, 2024
I don't think they are listed in the documentation itself. The last time I needed this information, I just looked them up in the source code.
App-layer protocols:
$ grep 'proto_name = ' src/app-layer-protos.c | cut -f2 -d= | egrep -v 'NULL|template|failed'
"http";
"ftp";
"smtp";
"tls";
"ssh";
"imap";
"jabber";
"smb";
"dcerpc";
"irc";
"dns";
"modbus";
"enip";
"dnp3";
"nfs";
"ntp";
"ftp-data";
"tftp";
"ikev2";
"krb5";
"dhcp";
"snmp";
"sip";
"rfb";
"mqtt";
"rdp";
Lower layer protocols are mentioned in src/detect-engine-proto.c:
$ grep strcasecmp src/detect-engine-proto.c | cut -f2 -d\"
tcp
tcp-pkt
tcp-stream
udp
icmpv4
icmpv6
icmp
sctp
ipv4
ip4
ipv6
ip6
ip
pkthdr
from gonids.
satta
commented on May 2, 2024
Here's an example for the noalert issue (ET OPEN):
alert http any any -> $HOME_NET any (msg:"ET EXPLOIT IBM Data Risk Manager Authentication Bypass - Session ID Assignment (set)"; flow:established,to_server; xbits:set,ET.IBMDRM1,track ip_dst, expire 10; noalert; http.method; content:"GET"; http.uri; content:"/albatross/saml/idpSelection"; startswith; fast_pattern; content:"id="; content:"userName="; reference:url,github.com/pedrib/PoC/blob/master/advisories/IBM/ibm_drm/ibm_drm_rce.md; classtype:attempted-admin; sid:2029986; rev:2; metadata:attack_target Server, created_at 2020_04_21, deployment Perimeter, deployment Datacenter, deployment SSLDecrypt, former_category EXPLOIT, performance_impact Low, signature_severity Major, updated_at 2020_04_21;)
which does not parse giving no valid value for noalert tag as the error.
from gonids.
satta
commented on May 2, 2024
You can test the parser with https://gist.github.com/satta/a19397fd91ce182df7570efbd2a14b92 and the ET OPEN ruleset (https://rules.emergingthreats.net/open/suricata-5.0/emerging-all.rules.tar.gz). For example, here are the issues still seen with the current master:
$ egrep -v '^#' emerging-all.rules | ./test-gonids | egrep -o ': .*' | sort | uniq -c
1 : could not parse base64Decode: offset must be positive, non-zero values only
1 : no valid value for app-layer-protocol tag
1 : no valid value for noalert tag
38 : rule contains unsupported option(s): dotprefix
1 : some or all destination ips are invalid: [!134.170.0.0/16,$EXTERNAL_NET]
1 : some or all destination ips are invalid: [!208.87.232.0/21,!216.115.208.0/20,!216.219.112.0/20,!66.151.158.0/24,!66.151.150.160/27,!66.151.115.128/26,!64.74.80.0/24,!202.173.24.0/21,!67.217.64.0/19,!78.108.112.0/20,!68.64.0.0/19,!206.183.100.0/22,!173.199.0.0/18,!103.15.16.0/22,!180.153.30.0/23,!140.207.108.0/23,!23.239.224.0/19,!185.36.20.0/22,!8.28.150.0/24,!54.208.0.0/15,!54.248.0.0/15,!70.42.29.0/27,!72.5.190.0/24,!104.129.194.0/24,!104.129.200.0/24,!199.168.148.0/24,!199.168.151.0/24,!216.52.207.64/26,$EXTERNAL_NET]
1 : some or all destination ips are invalid: [!224.0.0.1,$EXTERNAL_NET]
1 : some or all destination ips are invalid: [82.163.143.135,82.163.142.137]
1 : some or all destination ips are invalid: [92.63.0.0/16,91.218.114.0/24,149.56.245.196]
1 : some or all destination ports are invalid: [!$HTTP_PORTS,1024:]
8 : some or all destination ports are invalid: [0:1023]
1 : some or all destination ports are invalid: [:1024]
54 : some or all destination ports are invalid: [1024:]
1 : some or all destination ports are invalid: [104,2104,22104]
1 : some or all destination ports are invalid: [!11000,!11001,!12000]
1 : some or all destination ports are invalid: [135,139,445,1024:2048]
2 : some or all destination ports are invalid: [137,138,139,445]
42 : some or all destination ports are invalid: [139,445]
1 : some or all destination ports are invalid: [16992,16993,623,664]
1 : some or all destination ports are invalid: [20000:]
1 : some or all destination ports are invalid: [21,25,110,143,443,465,587,636,989:995,5061,5222]
26 : some or all destination ports are invalid: [23,2323]
4 : some or all destination ports are invalid: [23,2323,3323,4323]
3 : some or all destination ports are invalid: [2375,2376]
1 : some or all destination ports are invalid: [25,2525,587]
1 : some or all destination ports are invalid: [25,26,587,2525]
1 : some or all destination ports are invalid: [!25,!445,!1500]
3 : some or all destination ports are invalid: [25,465,587]
26 : some or all destination ports are invalid: [25,587]
1 : some or all destination ports are invalid: [25,587,2525]
1 : some or all destination ports are invalid: [!3478,1023:]
1 : some or all destination ports are invalid: [442,443,446,447,8001]
2 : some or all destination ports are invalid: [443,636,989,990,992,993,994,995,5061,25]
1 : some or all destination ports are invalid: [443,7080,8080]
9 : some or all destination ports are invalid: [445,139]
2 : some or all destination ports are invalid: [5060,5061]
1 : some or all destination ports are invalid: [554,9527]
2 : some or all destination ports are invalid: [5555,7547]
1 : some or all destination ports are invalid: [!5721,!5938]
1 : some or all destination ports are invalid: [!5800,!445]
1 : some or all destination ports are invalid: [!5938,!1433]
1 : some or all destination ports are invalid: [!5938,!1935,!3265,!2394]
6 : some or all destination ports are invalid: [60000:]
1 : some or all destination ports are invalid: [623,664]
16 : some or all destination ports are invalid: [6892,6893]
1 : some or all destination ports are invalid: [7080,8080,443]
5 : some or all destination ports are invalid: [7080,8080,443,80,4143,995,21,50000,20,8090,8443,990,22]
1 : some or all destination ports are invalid: [80,443]
1 : some or all destination ports are invalid: [8080:]
1 : some or all destination ports are invalid: [81:]
1 : some or all destination ports are invalid: [8443,9090]
2 : some or all destination ports are invalid: [9200,9292]
2 : some or all destination ports are invalid: [9530,9527,23]
1 : some or all source ips are invalid: [108.160.162.0/20,162.125.0.0/16,192.189.200.0/23,199.47.216.0/22,205.189.0.0/24,209.99.70.0/24,45.58.64.0/20]
1 : some or all source ips are invalid: [195.22.26.192/26,195.22.28.192/27,195.38.137.100,195.22.4.21,195.157.15.100,212.61.180.100]
1 : some or all source ports are invalid: [0:20,22:24,26:118,120:138,140:444,446:464,466:586,588:901,903:1432,1434:65535]
1 : some or all source ports are invalid: [10000:]
1 : some or all source ports are invalid: [1023:]
36 : some or all source ports are invalid: [1024:]
1 : some or all source ports are invalid: [104,2104,22104]
1 : some or all source ports are invalid: [20000:]
2 : some or all source ports are invalid: [21,1024:]
2 : some or all source ports are invalid: [21,22,23,25,53,80,443,8080]
2 : some or all source ports are invalid: [21,25,110,143,443,465,587,636,989:995,5061,5222]
1 : some or all source ports are invalid: [!3389,1024:65535]
1 : some or all source ports are invalid: [443,$HTTP_PORTS]
5 : some or all source ports are invalid: [443,4443]
3 : some or all source ports are invalid: [443,465,993,995,25]
2 : some or all source ports are invalid: [445,139]
1 : some or all source ports are invalid: [80,443]
1 : some or all source ports are invalid: [81:]
from gonids.
satta
commented on May 2, 2024
Can you provide a quick pointer in the source to where these other protocols are defined (probably an issue should be opened with OISF to update the docs to make them complete as well).
I can do that and open an issue in their Redmine later.
from gonids.
duanehoward
commented on May 2, 2024
I think with the current revisions all of the protocol parsing issues should be resolved.
network parsing issues should be resolved as well, with one caveat: We currently define 0 as an invalid port in our validation, This appears to be used in a few places by ET OPEN this requires a bit of consideration as to whether we should allow this value or not, it probably doesn't hurt to do so unless there is a special meaning of 0 in Suricata.
Feel free to pull the newest revision and let me know. I still haven't addressed the noalert issue yet.
from gonids.
duanehoward
commented on May 2, 2024
note: network parsing issues were addressed in #165
from gonids.
satta
commented on May 2, 2024
I think with the current revisions all of the protocol parsing issues should be resolved.
Thanks! Looks good.
network parsing issues should be resolved as well
Nice, I tried something complicated like
alert tcp [10.0.0.0/24,!10.0.0.5,![10.0.0.10,10.0.0.11]] any <> any [1:1000,!99,![80:90]] (msg: "foo"; sid:42; rev:1;)
and it looks like it was parsed correctly now. However, the parser still seems to disallow spaces between the brackets, which according to the Suricata documentation is OK to do.
I also noticed that apparently invalid source/dest patterns like
[10.0.0.0/24,![10.0.0.10,10.0.0.11] any
(note the disbalanced brackets) is currently successfully parsed to
10.0.0.0/24,![10.0.0.10,10.0.0.11 any
I think this should be an error.
with one caveat: We currently define
0as an invalid port in our validation, This appears to be used in a few places byET OPENthis requires a bit of consideration as to whether we should allow this value or not, it probably doesn't hurt to do so unless there is a special meaning of0in Suricata.
I think since this is not a syntax issue I would be inclined to allow the value 0, tool.
Feel free to pull the newest revision and let me know. I still haven't addressed the
noalertissue yet.
I see. The dotprefix is also not in there yet. But getting there! Many thanks for looking into this! 👍🏻
from gonids.
duanehoward
commented on May 2, 2024
IIRC we removed support for spaces in the past, possibly because it was introducing some odd complexity in the lexer, and we noted that basically all rulesets and examples don't use spaces. (the lexer can't look for a ; when finding the network tokens or similar and keying on [ and ] was problematic as well. I don't believe support for spaces is something we can quickly fix. PR's are welcome for that.
#167 should resolve the unbalanced [ and ] issue and adds support for ports to include 0.
The handling of nested groups appears to have introduced another bug in handling negated network groups where the negation is the first part of the pattern something like:
![$DNS_SERVERS,$SMTP_SERVERS]
This issue is strangely complex to find a solution for, and frankly the more I look at this, I wonder if it would be simpler to move the network definitions into better defined structs and lex them differently. That, however is not something I can handle in the short term, so I'll track that as a distinct issue.
TODO(duane): dotprefix and bare noalert
from gonids.
satta
commented on May 2, 2024
IIRC we removed support for spaces in the past, possibly because it was introducing some odd complexity in the lexer, and we noted that basically all rulesets and examples don't use spaces. (the lexer can't look for a
;when finding the network tokens or similar and keying on[and]was problematic as well. I don't believe support for spaces is something we can quickly fix.
Yes, I suspected that. But I think that's more of a minor issue since we do get an error message if spaces are encountered instead of silently parsing the range incorrectly.
#167 should resolve the unbalanced
[and]issue and adds support for ports to include0.
Great, thanks!
The handling of nested groups appears to have introduced another bug in handling negated network groups where the negation is the first part of the pattern something like:
![$DNS_SERVERS,$SMTP_SERVERS]This issue is strangely complex to find a solution for, and frankly the more I look at this, I wonder if it would be simpler to move the network definitions into better defined structs and lex them differently. That, however is not something I can handle in the short term, so I'll track that as a distinct issue.
Yes, it is more complex than one might think at first glance. When I wrote a rule parser in Python the last time, I used https://github.com/lark-parser/lark to work out a comprehensive grammar, which I found easier to tweak and optimize than having to work with actual parsing code. I didn't research if there is something similar for Go though. Maybe something like https://github.com/alecthomas/participle
Anyway, thanks so far for all the work you put in, it has really improved things!
from gonids.
Related Issues (20)
- Handling escaped characters in content
- Errors parsing network info HOT 1
- PR for additional validations HOT 3
- Nested network info parsing unsupported HOT 4
- cifuzz broken HOT 2
- GolangCI dead HOT 3
- fix lint errors HOT 1
- Open ended port ranges not correctly handled HOT 2
- Add support for `target` keyword HOT 1
- Can't parse Snort rule with error: rpc error: code = Unknown desc = invalid special characters escaping HOT 1
- OSS-Fuzz issue 49328 HOT 1
- OSS-Fuzz issue 49653 HOT 1
- have a question about base64_decode HOT 3
- OSS-Fuzz issue 54669 HOT 1
- OSS-Fuzz issue 54867
- OSS-Fuzz issue 55416 HOT 1
- OSS-Fuzz issue 55592 HOT 1
- OSS-Fuzz issue 56718 HOT 1
- OSS-Fuzz issue 58318 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gonids.