Git Product home page Git Product logo

gonids's Introduction

gonids is a library to parse IDS rules for engines like Snort and Suricata.

Installation

$ go get github.com/google/gonids

Quick Start

Add this import line to the file you're working in:

import "github.com/google/gonids"

To parse a rule:

rule := `alert tcp $HOME_NET any -> $EXTERNAL_NET 80 (msg:"GONIDS TEST hello world"; flow:established,to_server; content:"hello world"; classtype:trojan-activity; sid:1; rev:1;)`
r, err := gonids.ParseRule(rule)
if err != nil {
  // Handle parse error
}
// Do something with your rule.
switch r.Action {
case "alert":
  // This is an 'alert' rule.
case "drop":
  // This is a 'drop' rule.
case "pass":
  // This is a 'pass' rule.
default:
  // I have no idea what this would be. =)
}

To create a rule a DNS rule (using dns_query sticky buffer) and print it:

r := gonids.Rule{
	Action:   "alert",
	Protocol: "dns",
	Source: Network{
		Nets:  []string{"any"},
		Ports: []string{"any"},
	},
	Destination: Network{
		Nets:  []string{"any"},
		Ports: []string{"any"},
	},
	SID:         1234,
	Revision:    1,
}

badDomain := "c2.evil.com"
dnsRule.Description = fmt.Sprintf("DNS query for %s", badDomain)

sb, _ := gonids.StickyBuffer("dns_query")
c := &gonids.Content{
			DataPosition: sb,
			Pattern:      []byte(badDomain),
			Options: []*gonids.ContentOption{
				{"nocase", ""},
			},
		}
}

fmt.Println(r)

To optimize a Snort HTTP rule for Suricata:

rule := `alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"GONIDS TEST hello world"; flow:established,to_server; content:"hello.php"; http_uri; classtype:trojan-activity; sid:1; rev:1;)`
r, err := gonids.ParseRule(rule)
if err != nil {
  // Handle parse error
}
r.OptimizeHTTP()

Miscellaneous

This is not an official Google product.

gonids's People

Contributors

catenacyber avatar chuanjiesun avatar danielpoliakov avatar duanehoward avatar elmeyer avatar jakewarren avatar julienschmidt avatar krzysiekpiasecki avatar leo-neat avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gonids's Issues

Add SidMsg function

It's a relatively trivial thing to add a function to return a string representing a sidmsg.map entry
We should add this as a function to rule.go

Errors parsing network info

Hi,

refering to network examples in these sections of docs:


Negated lists

s := `alert tcp any any -> any ![80,443,9000] (msg:"test"; content:"123"; sid:1; rev:1;)`
r, _ := gonids.ParseRule(s)
fmt.Println(r)

outputs

alert tcp any any -> any [![80,443,9000] (msg:"test"; content:"123"; sid:1; rev:1;)

and

s := `alert tcp any any -> ![1.1.1.1,1.1.1.2] any (msg:"test"; content:"123"; sid:1; rev:1;)`
r, _ := gonids.ParseRule(s)
fmt.Println(r)

outputs

alert tcp any any -> [![1.1.1.1,1.1.1.2] any (msg:"test"; content:"123"; sid:1; rev:1;)

which is invalid.


Spaces in network components

Gonids outputs error when network components contain spaces. Spaces in list of IPs(ranges)/ports are valid based on the examples in docs and my experiments.

s := `alert tcp any any -> [1.1.1.1, 1.1.1.2] any (msg:"test"; content:"123"; sid:1; rev:1;)`

_, err := gonids.ParseRule(s)
if err != nil {
	fmt.Println(err)
}

outputs

network component contains spaces: 1.1.1.2] any

Same it goes for list of ports.


Can you please give me any hints where in gonids these issues can be addressed?

Support for lexing \ at end of content

PR #29 added support for lexing escaped quotes in option values, however this broke lexing a specific case in contents where the content ends in a \ like:
content:"foo\";

design question: lexer in goroutine?

While looking through the code, I noticed that the lexer runs its main loop its own goroutine:

func lex(input string) (*lexer, error) {
	...
	l := &lexer{
		input: input,
		items: make(chan item),
	}
	go l.run()
	return l, nil
}

Items are then received from the unbuffered channel l.items , thus making the code run fully sequential again.

Is there any other good reason to run it in it's own goroutine? If not, the overhead for synchronization and context switches could be avoided.

Stringer output missing some Tags

Specifically after parsing some rules containing dsize and byte_test values these components are not printed back out.
This may be an issue with the initial parsing of the rule into the struct or with the printing of these values.

Missing keywords for ET OPEN

After processing ET OPEN with trojan.rules web_specific_apps.rules and current _events.rules I've identified support gaps for the following missing keywords:

68 stream_size
10 itype
10 icode
3 flags
2 ssl_version
1 rawbytes
1 icmp_id

So, we're pretty close to complete support for ET OPEN. A few gaps to fille.

Nested network info parsing unsupported

Suricata allows multiple nested levels of network and port grouping, e.g.

[1:80,![2,4]]

which would specify a port range from 1-80, except ports 2 and 4. See https://suricata.readthedocs.io/en/latest/rules/intro.html#ports-source-and-destination for more examples.

I have read the code in parser.go and it looks due to the way the current parser handles these entries (only one level of grouping) this would not be parsable:

some or all destination ports are invalid: [1:80 ![2 4]

There is also a problem with negated groups in general: Parsing

![25,110,143,465,587,2525]

leads to

some or all destination ports are invalid: [![25 110 143 465 587 2525]

since the leading ! is not clipped before checking the validity of the port entries. Port group parsing is currently the biggest problem when trying to parse the entirety of the ETPRO rule set, and while the latter issue is easy to patch for one level of group items, it feels like it should be implemented correctly for all cases.

There are other issues with parsing ETPRO at the moment, such as missing support for noalert without values in some places, transformations (such as dotprefix) and missing protocols (icmpv6, ftp-data, etc.) and more. These seem to be smaller in nature and probably easy to fix.

fix lint errors

After re-enabling basic linting, two functions from lex.go are noted as unused.

PR for additional validations

Hi @duanehoward,

I recently reached out via e-mail in regards to a PR I'd like to know if you'd find useful. Basically some additional validations, such as:

  • action keyword
  • protocol
  • network information (IP and port)

We have the changes at https://github.com/pdr9rc/gonids in case you'd like to have a look. Happy to make any changes to fit gonids requirements.

Also, if you prefer to talk via e-mail feel free to close/ignore this issue.

ByteMatchers should support variables in "NumBytes" probably in other fields.

We currently parse the NumBytes for isdataat, and others as an int.
Some rules seem to use variables here:

... byte_extract:4,0,variable,relative,little; content:"foo"; within:3; distance:4; isdataat:variable,relative; ...

We currently fail to parse "variable" in isdataat because it's not an int.

  1. Validate that this is true for Suricata (documentation doesn't specifically mention this)
  2. Migrate to using string for the NumBytes field for ByteMatchers.

parse rule falied,invalid type '\v' for content type

parse the rule falied:
alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"ET POLICY Java Url Lib User Agent Web Crawl"; flow:established,to_server; content:"Java/"; nocase; http_user_agent; pcre:/Java/\d.\d/Vi"; threshold: type both, track by_src, count 10, seconds 60; reference:url,www.mozilla.org/docs/netlib/seealso/netmods.html; reference:url,doc.emergingthreats.net/2002945; classtype:attempted-recon; sid:2002945; rev:12; metadata:created_at 2010_07_30, updated_at 2010_07_30;)
error is : invalid type '\v' for option content, how to solve this problem?

Revisit lexRuleEnd

lexRuleEnd is called when we encounter the ) that closes a rule, it then consumes some characters \t ;
Probably we should just jump to emit itemEOR instead?
Needs research into what is valid outside of the rule definition in parentheses, I think nothing?

Add support for Suricata 5.0 rules

This may be relatively simple as the core differences seem to center around greater use of sticky buffers and we already have a good mechanism for supporting those.

However it may be useful or necessary to be able to identify different rule versions in order to be able to do things like perform automatic optimizations (4.x -> 5x). This part requires a bit more thought as we'll need to be able to map between equivalent constructs in versions.

For example we'd need to understand that content:"foo"; http_uri; and http.uri; content:"foo"; are equivalent.

It's also not clear to me (needs more research) as to whether it is allowed or valid to mix and match these types for example is it valid to have:
http.uri; content:"/foo"; pkt_data; content:"bar"; http_uri"

While doing this would be incredibly strange, and possibly silly we may or may not need to be able to parse these independently.

Stringer for rule forgets ordering

We currently do something like:
Print all contents, then all PCRE's, etc.

However, PCRE, byte_extract, isdataat, etc. can all have relative positioning.
We need to do something to force that these maintain ordering (or at least they maintain ordering IF the match is relative, but that seems like an overcomplication).

Improve test readability

Lots of the test data is defined on single lines,
add newlines to make modification of tests, and reading of test data simpler.

add enabled/disabled state for parser

Lexer treats all "#" lines as comments, we should be able to handle commented out rules (parse and set a disabled bool) differently than actual comments (ignore).

Handling escaped characters in content

Hi, if I'm not mistaken FormatPattern handles backslashes wrongly. I came to this in numerous ET open rules while firstly parsing them, saving parsed gonids.Rule as string and loading with Suricata with errors. For example:

s := `alert http any any -> any any (msg:"test"; content:"|3a 5c|Windows|5c|"; sid:1; rev:1;)`
r, _ := gonids.ParseRule(s)
fmt.Println(r)

outputs:

alert http any any -> any any (msg:"test"; content:"|3A|\Windows\"; sid:1; rev:1;)

When loading this rule to Suricata (5.0.0) I get following error:

[ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - '\' has to be escaped
[ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert http any any -> any any (msg:"test"; content:"|3a|\Windows\"; sid:1; rev:1;)" from file test.rules at line 1

lexDestinationPort is flawed

Other portions of 'network' lexing lex until a space character. This one lexes until '(' and does a backup. This can lead to destination ports that are invalid:

Probably this could use some additional test cases and tweaks to the logic.
Possibly the parser should handle this.

parse rule falied, invalid content option "byte_extract" with no content match

parse rule failed:
alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"TROJAN Backdoor family PCRat/Gh0st CnC traffic (OUTBOUND) 14"; flow:to_server,established; dsize:>11; byte_extract:4,0,c_size,little; byte_test:4,>,c_size,4,little; content:"|08 01|"; offset:2; depth:2; content:"|79 94|"; offset:13; depth:2; pcre:"/^.{8}[\x20-\x7e]+?\x79\x94/s"; reference:url,www.securelist.com/en/descriptions/10155706/Trojan-GameThief.Win32.Magania.eogz; reference:url,www.microsoft.com/security/portal/Threat/Encyclopedia/Entry.aspx?Name=Backdoor%3AWin32%2FPcClient.ZR&ThreatID=-2147325231; reference:md5,9fae15fa8ab6bb8d78d609bdceafe28e; classtype:trojan-activity; sid:2017944; rev:5; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, deployment Perimeter, tag PCRAT, tag Gh0st, tag RAT, signature_severity Critical, created_at 2014_01_08, malware_family Gh0st, malware_family PCRAT, updated_at 2016_07_01;)
error message is : invalid content option "byte_extract" with no content match, how to solve it?

can't parse pcre in suricata

the parser can't parse pcre in suricata rules,can add the item to parser?
the rule looks like below:
alert http $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"ET CURRENT_EVENTS Request to malicious info.php drive-by landing"; flow:established,to_server; content:"/info.php?n="; http_uri; fast_pattern:only; content:!"&"; http_uri; content:!"|0d 0a|Referer|3a|"; pcre:"//info.php?n=\d/U"; classtype:trojan-activity; sid:2013010; rev:3; metadata:created_at 2011_06_10, updated_at 2011_06_10;)

Can't parse Snort rule with error: rpc error: code = Unknown desc = invalid special characters escaping

I use the version github.com/google/gonids v0.0.0-20211022205232-4d00a2956aaa, and when using function gonids.ParseRule(resp.Snort) to parse our Snort rule, it returns error "rpc error: code = Unknown desc = invalid special characters escaping". The Snort rule has some special characters such as "/(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=|[A-Za-z0-9+/]{4})$/P". Could you help with it? Thanks.

Space normalization breaks parsing of disabled rules

PR #92 normalized all types of unicode space characters, but doesn't replace them with the same number of characters that existed before. So a rule that is disabled and has multiple spaces in sequence will break during parsing.

input:  #alert...(content:"foo     bar";....;)
output: #alert...(content:"foo bar";....;)

Parser has grown too large

Refactor some of the code to split into new files, parser.go has non-parsing related code in it that should be easily moved (all Rule related code to 'rule.go' for example)

support more rule protocols

When parsing suricata rules, the following errors were shown with a parsing error:

2019/12/10 12:06:57 [ERR] invalid character '6' for a rule protocol
alert ipv6 any any -> any any (msg:"SURICATA reserved field in Frag Header not zero"; decode-event:ipv6.fh_non_zero_reserved_field; classtype:protocol-command-decode; sid:2200095; rev:2;)
2019/12/10 12:06:57 [ERR] invalid character '2' for a rule protocol
alert ikev2 any any -> any any (msg:"SURICATA IKEv2 weak cryptographic parameters (Auth)"; flow:to_client; app-layer-event:ikev2.weak_crypto_auth; classtype:protocol-command-decode; sid:2224004; rev:2;)
2019/12/10 12:06:57 [ERR] invalid character '2' for a rule protocol
alert ikev2 any any -> any any (msg:"SURICATA IKEv2 unknown proposal"; flow:to_server; app-layer-event:ikev2.unknown_proposal; classtype:protocol-command-decode; sid:2224011; rev:1;)
2019/12/10 12:06:58 [ERR] invalid character '3' for a rule protocol
alert dnp3 any any -> any any (msg:"SURICATA DNP3 Length too small"; app-layer-event:dnp3.len_too_small; classtype:protocol-command-decode; sid:2270001; rev:3;)

It seems gonids didn't support parsing app protocol in the protocol field of a rule. Will this be supported in the future? Thanks!!

Missing protocols and transformations

From issue #154 @satta notes:

There are other issues with parsing ETPRO at the moment, such as missing support for noalert without values in some places, transformations (such as dotprefix) and missing protocols (icmpv6, ftp-data, etc.) and more. These seem to be smaller in nature and probably easy to fix.

GolangCI dead

GolangCI linter died over a year ago, move to "new" github action.

have a question about base64_decode

hello, i have a question about base64_decode:
when i have a rule such as:
alert http any any -> any any (msg:"this is test", flow:established, to_server; http.request_body; content:"test"; fast_pattern; base64_decode: bytes 1024, offset 0, relative; base64_data; ..........)
it can't be parsed because offset must be positive, non-zero values only. But i think offset can be set to 0 after i look through suricata document and source code.
Could you help with it? Thanks.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.