0xrawsec / gene Goto Github PK

View Code? Open in Web Editor NEW

154.0 14.0 17.0 5.78 MB

Signature engine for all your logs

License: GNU General Public License v3.0

Go 97.58% Makefile 0.32% Shell 0.12% Python 1.98%

dfir threat-hunting detection-engineering

gene's Introduction

Gene(sis)

A long long time ago (in 2017) after doing many responses to incidents, I realized I was always ending up doing the same thing to search inside Windows EVTX logs: writting a custom script to match log entries against our findings ! At that moment I decided to start coding this tool, not only to ease my daily work but also to be able to share detection rules between parties.

Since then, the tool has evolved and it can now be used to match against any kind of log (formatted in JSON) and has native support for Windows EVTX parsing.

Some use cases

Digital forensic analysis
- early compromise information collection
- infected host analysis
- IOC scan on a whole network
(Retro)Hunt on cold storage
- backups
- logs forwarded
Combined with other tools to achieve powerful detection primitives
- Microsoft Sysmon
- Kunai

Additional resources

To convert old rules (prior to 2.0.0 schema) to the new format, use migraterule.py
Where to find rules ?

Changelog

v2.0.0

Code refactoring:
- Changes in package organisation
- Changes in API definitions
- Implementation of an Event interface making APIs more generic
- Default actions to apply on detections
Changes in the rule format:
- New way define events to apply rule on
- Schema field to enforce rule format compatibility with engine
- Removed trace support (not up to date and not used)
Regex templates defined in TOML format

v1.6.0

Indirect Match Support (we can now compare two fields of the same event)
Containers are now case insensitive
New -test command line switch to create easy Gene unit testing

v1.5.0

Support for Mitre ATT&CK framework
Changes in the reducer function

gene's People

Contributors

Stargazers

Watchers

Forkers

5up3rc beerandgin michaelhidalgo advancedwebdeveloper rmusser01 casimkhan blue-infosec trietptm-on-coding-algorithms bigbrobro darsenault hartl3y94 webvul mnaza darksidesfear data-gami binaco fengjixuchui

gene's Issues

Use TOML for regex templates

More readable and takes advantage of syntax parsing/highlighting of modern text editors.

change rule format

remove trace support (not used and not updated)
change event selection (current selection does not make sense because channel and event ids are independents)
create a way to check format in the engine to prevent using incompatible rules

Update documentation

Update documentation in relation with issue #11

Not working on multiple files

Hi guys, thanks for this amazing tool.

i am analyzing multiple vulnerable evtx files (can be downloaded from https://github.com/sans-blue-team/DeepBlueCLI/tree/master/evtx) to test Gene however Gene does not appear to be working when *.evtx is used and is only working on single files.

POC(powershekk):


PS D:\CA\Gene> .\gene-386.exe -r ./gene-rules -progress D:\CA\Gene\test\*.evtx
.\gene-386.exe : 2019/08/30 03:12:21 INFO - Size of whitelist container: 0
At line:1 char:1
+ .\gene-386.exe -r ./gene-rules -progress D:\CA\Gene\test\*.evtx
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (2019/08/30 03:1...st container: 0:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
2019/08/30 03:12:21 INFO - Size of blacklist container: 0
2019/08/30 03:12:21 INFO - Loading regexp templates from file: D:\CA\Gene\gene-rules\rules\regexp.tpl
2019/08/30 03:12:21 Warning - Unknown container "blacklist" used in rule "BlacklistedDomain"
2019/08/30 03:12:21 Warning - Rule "BlacklistedDomain" has been disabled at compile time
2019/08/30 03:12:21 Warning - Unknown container "blacklist" used in rule "BlacklistedHash"
2019/08/30 03:12:21 Warning - Rule "BlacklistedHash" has been disabled at compile time
2019/08/30 03:12:21 Warning - Unknown container "blacklist" used in rule "BlacklistedImphash"
2019/08/30 03:12:21 Warning - Rule "BlacklistedImphash" has been disabled at compile time
2019/08/30 03:12:21 Warning - Unknown container "whitelist" used in rule "NotWhitelisted"
2019/08/30 03:12:21 Warning - Rule "NotWhitelisted" has been disabled at compile time
2019/08/30 03:12:21 Warning - Unknown container "blacklist" used in rule "SysmonBlacklistedDomain"
2019/08/30 03:12:21 Warning - Rule "SysmonBlacklistedDomain" has been disabled at compile time
2019/08/30 03:12:21 ERROR - Error loading D:\CA\Gene\gene-rules\rules\ads.gen: Failed to load rule file "D:\CA\Gene\gene-rules\rules\ads.gen": Rule "NewADS" already 
exists
[bunch of rule already exist errors]
2019/08/30 03:12:21 INFO - Loaded 128 rules
2019/08/30 03:12:21 INFO - Processing: D:\CA\Gene\test\*.evtx
2019/08/30 03:12:21 ERROR - open D:\CA\Gene\test\*.evtx: The filename, directory name, or volume label syntax is incorrect.
2019/08/30 03:12:21 INFO - Count Rules Used (loaded + generated): 128
2019/08/30 03:12:21 INFO - Event Scanned: 0
2019/08/30 03:12:21 INFO - Positives: 0

When i load a single file, it works perfectly:

.\gene-amd64.exe -progress -a -r ./gene-rules D:\CA\Gene\test\metasploit-psexec-pwshpayload.evtx
.\gene-amd64.exe : 2019/08/30 03:13:51 INFO - Size of whitelist container: 0
At line:1 char:1
+ .\gene-amd64.exe -progress -a -r ./gene-rules D:\CA\Gene\test\metaspl ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (2019/08/30 03:1...st container: 0:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
2019/08/30 03:13:51 INFO - Size of blacklist container: 0
2019/08/30 03:13:51 INFO - Loading regexp templates from file: D:\CA\Gene\gene-rules\rules\regexp.tpl
2019/08/30 03:13:51 Warning - Unknown container "blacklist" used in rule "BlacklistedDomain"
2019/08/30 03:13:51 Warning - Rule "BlacklistedDomain" has been disabled at compile time
2019/08/30 03:13:51 Warning - Unknown container "blacklist" used in rule "BlacklistedHash"
2019/08/30 03:13:51 Warning - Rule "BlacklistedHash" has been disabled at compile time
2019/08/30 03:13:51 Warning - Unknown container "blacklist" used in rule "BlacklistedImphash"
2019/08/30 03:13:51 Warning - Rule "BlacklistedImphash" has been disabled at compile time
2019/08/30 03:13:51 Warning - Unknown container "whitelist" used in rule "NotWhitelisted"
2019/08/30 03:13:51 Warning - Rule "NotWhitelisted" has been disabled at compile time
2019/08/30 03:13:51 Warning - Unknown container "blacklist" used in rule "SysmonBlacklistedDomain"
2019/08/30 03:13:51 Warning - Rule "SysmonBlacklistedDomain" has been disabled at compile time
2019/08/30 03:13:51 ERROR - Error loading D:\CA\Gene\gene-rules\rules\ads.gen: Failed to load rule file "D:\CA\Gene\gene-rules\rules\ads.gen": Rule "NewADS" already 
exists
[bunch of rule already exist errors]
2019/08/30 03:13:51 INFO - Loaded 128 rules
2019/08/30 03:13:51 INFO - Processing: D:\CA\Gene\test\metasploit-psexec-pwshpayload.evtx
{"Event":{"GeneInfo":{"ATTACK":[{"ID":"T1070","Tactic":"defense-evasion","Reference":"https://attack.mitre.org/techniques/T1070"}],"Criticality":8,"Signature":["SecurityL
ogClearing"]},"System":{"Channel":"Security","Computer":"SANS-TBT570","Correlation":{},"EventID":"1102","EventRecordID":"23133","Execution":{"ProcessID":"988","ThreadID":
"304"},"Keywords":"0x4020000000000000","Level":"4","Opcode":"0","Provider":{"Guid":"{fc65ddd8-d6ef-4962-83d5-6e5cfe9ce148}","Name":"Microsoft-Windows-Eventlog"},"Security
":{},"Task":"104","TimeCreated":{"SystemTime":"2019-05-03T15:20:20Z"},"Version":"0"},"UserData":{"LogFileCleared":{"SubjectDomainName":"SANS-TBT570","SubjectLogonId":"0x0
0042c3d","SubjectUserName":"student","SubjectUserSid":"S-1-5-21-1552841522-3835366585-4197357653-1001","xmlns":"http://manifests.microsoft.com/win/2004/08/windows/eventlo
g"}}}}
2019/08/30 03:13:51 INFO - Count Rules Used (loaded + generated): 128
2019/08/30 03:13:51 INFO - Event Scanned: 5
2019/08/30 03:13:51 INFO - Positives: 1

Any idea what i might be doing wrong?

Best,

Running unit tests : unknown import path "engine": cannot find module providing package engine

Hey @qjerome , I'm trying to run the gene unit test for the engine but I'm having some trouble.

Am I missing something?

go test engine_test.go


engine_test.go:5:2: unknown import path "engine": cannot find module providing package engine
FAIL    command-line-arguments [setup failed]

[root@mike engine]

sigma and yara

Is it possible to use sigma and yara rules with gene?

regexp

1)How to make register-independent regexp?
I did`t find the command line options.
2) Please write examples of using traces

Implement default actions

Let the engine set some default actions based on criticality.
Will be used to replace the dump setting in WHIDS

Define an Event interface

Goal: make the tool compatible with ETW logs

Implement filter rules

Implement a way to filter events going through the engine.
A filtered event is an event that only match rules defined as filter.
A filter rule is a rule that allows to filter in the events we want to keep for logging.

See: 0xrawsec/whids#18

Add unit testing for compiled rules that check functionality and performance

Hey @qjerome as per my article it would be good to add unittests with mocks of the sysmon events that allow us to not only test that rules matches but also that are performant.

Condition

Hi, thanks for the great utility.
Question 1:
Tell me how to use "traces" and changing "criticality"

Question 2:
Is it possible to create the construction of conditions corresponding to the yara:

Condition:
all of them
1 of them
(all of ($s*)) and !(all of ($x*))
$s>10

Question 3:
Is there a description of the operators both in yara?
https://yara.readthedocs.io/en/v3.4.0/writingrules.html

Ability to tag an alert based on MITRE ATT&CK Framework

opened by @michaelhidalgo in WHIDS project but moved here because more appropriate:

With MITRE ATT&CK becoming a standard way to classify an attacker action (aka technique) across the cyber kill change, it would be good to be able that, when a rule matches, be able to enrich that rule with the actual MITRE ATT&CK Tactic and Technique being used.

This has several benefits opening the door for doing correlation and eventually be able to draw (via some sort of dot notation graphs) the attacker objectives.

For example, this rule:

  "Name": "MaliciousLsassAccess",
  "Tags": ["Mimikatz", "Credentials", "Lsass"],
  "Meta": {
    "EventIDs": [10],
    "Channels": ["Microsoft-Windows-Sysmon/Operational"],
    "Computers": [],
    "Traces": [],
    "Criticality": 10,
    "Author": "0xrawsec"
  },
  "Matches": [
    "$ct: CallTrace ~= 'UNKNOWN'",
    "$lsass: TargetImage ~= '(?i:\\\\lsass\\.exe$)'"
  ],
  "Condition": "$lsass and $ct"
}```


Potentially matches [T1003](https://attack.mitre.org/techniques/T1003/)  Credential Dumping, so it would be nice to have this 

`

{
  "Event": {
    "EventData": {
      "CallTrace": "C:\\Windows\\SYSTEM32\\ntdll.dll+4bf9a|C:\\Windows\\system32\\KERNELBASE.dll+189b7|UNKNOWN(00000000259123BC)",
      "GrantedAccess": "0x1410",
      "SourceImage": "C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\powershell.exe",
      "SourceProcessGUID": "{49F1AF32-DD18-5A72-0000-0010042C0A00}",
      "SourceProcessId": "2248",
      "SourceThreadId": "3308",
      "TargetImage": "C:\\Windows\\system32\\lsass.exe",
      "TargetProcessGUID": "{49F1AF32-DB3B-5A72-0000-001013690000}",
      "TargetProcessId": "492",
      "UtcTime": "2018-02-01 11:24:53.331"
    },
    "GeneInfo": {
      "Criticality": 10,
      "Signature": [
        "MaliciousLsassAccess"
      ]
      "ATT&CK Tactic"   : "Credential Access",
      "ATT&CK Technique : "T1003",
      "ATT&CK Reference": "https://attack.mitre.org/techniques/T1003/"
    },
    "System": {
        "Classical Windows Event System Section": "..."
    }
  }
}`

error

Hello!
When I run the program (windows-v1.2.tar.gz), I get an error: The system can not find the path specified.
What am I doing wrong?

Rule:
`{
"Name": "Mimikatz",
"Tags": ["test"],
"Meta": {
"EventIDs": [4107],
"Channels": ["Microsoft-Windows-PowerShell/Operational"],
"Computers": [],
"Criticality": 0
},

"Matches": [
	"$a: ScriptBlockText ~= 'Mimikatz'"
	],
"Condition": "$a"
}

{
"Name": "sysmon",
"Tags": ["sysmon"],
"Meta": {
"EventIDs": [1],
"Channels": ["Microsoft-Windows-Sysmon/Operational"],
"Computers": [],
"Criticality": 0
},

"Matches": [
	"$a: Hashes ~= 'SHA256=30879AAB3ED2518FF7'"
	],
"Condition": "$a"
}`

Error:

`f:>gene-386.exe -r evtx.txt "c:\windows\System32\Winevt\Logs\Microsoft-Windows-
PowerShell/Operational.evtx"
2018/03/20 19:08:15 INFO - Loaded 2 rules
2018/03/20 19:08:15 INFO - Processing: c:\windows\System32\Winevt\Logs\Microsoft
-Windows-PowerShell/Operational.evtx
2018/03/20 19:08:15 ERROR - open c:\windows\System32\Winevt\Logs\Microsoft-Windo
ws-PowerShell/Operational.evtx: The system cannot find the path specified.
2018/03/20 19:08:15 INFO - Count Event Scanned: 0
2018/03/20 19:08:15 INFO - Count Rules Used (loaded + generated): 2

f:>gene-386.exe -r evtx.txt "%SystemRoot%\System32\Winevt\Logs\Microsoft-Window
s-PowerShell%4Operational.evtx"
2018/03/20 19:10:36 INFO - Loaded 2 rules
2018/03/20 19:10:36 INFO - Processing: C:\Windows\System32\Winevt\Logs\Microsoft
-Windows-PowerShell%4Operational.evtx
2018/03/20 19:10:36 ERROR - open C:\Windows\System32\Winevt\Logs\Microsoft-Windo
ws-PowerShell%4Operational.evtx: The system cannot find the path specified.
2018/03/20 19:10:36 INFO - Count Event Scanned: 0
2018/03/20 19:10:36 INFO - Count Rules Used (loaded + generated): 2

f:>gene-386.exe -r evtx.txt "%SystemRoot%\System32\Winevt\Logs\Microsoft-Window
s-Sysmon%4Operational.evtx"
2018/03/20 19:11:28 INFO - Loaded 2 rules
2018/03/20 19:11:28 INFO - Processing: C:\Windows\System32\Winevt\Logs\Microsoft
-Windows-Sysmon%4Operational.evtx
2018/03/20 19:11:28 ERROR - open C:\Windows\System32\Winevt\Logs\Microsoft-Windo
ws-Sysmon%4Operational.evtx: The system cannot find the path specified.
2018/03/20 19:11:28 INFO - Count Event Scanned: 0
2018/03/20 19:11:28 INFO - Count Rules Used (loaded + generated): 2

f:>gene-386.exe -r evtx.txt "%SystemRoot%\System32\Winevt\Logs\Microsoft-Window
s-Sysmon%4Operational.evtx" -d
2018/03/20 19:12:56 INFO - Loaded 2 rules
2018/03/20 19:12:56 INFO - Processing: C:\Windows\System32\Winevt\Logs\Microsoft
-Windows-Sysmon%4Operational.evtx
2018/03/20 19:12:56 ERROR - open C:\Windows\System32\Winevt\Logs\Microsoft-Windo
ws-Sysmon%4Operational.evtx: The system cannot find the path specified.
2018/03/20 19:12:56 INFO - Processing: -d
2018/03/20 19:12:56 ERROR - open -d: The system cannot find the file specified.
2018/03/20 19:12:56 INFO - Count Event Scanned: 0
2018/03/20 19:12:56 INFO - Count Rules Used (loaded + generated): 2`

Add an OS field to rules

Add an OS field to rules definition to allow match/load only on some OS.

Integrate Kunai's log format to be matched against rules

Request: Test data (.evtx)

Create a bounded score in reducer

The reducer is used to aggregate detection statistics about machines.
Currently a score is attributed to each machine, however this later is not bounded which might be difficult to interpret when looking at it. So the objective is to create a bounded score derived from the normal score.

bounded_score = score / max_score

Make fails when shell doesn't have $GOPATH

gene/gene/makefile

Line 1 in 4401b38

RELEASE=$(GOPATH)/release

go env GOPATH shows the correct value for $GOPATH, however, on shells like mine, echo $GOPATH simply returns a blank line because it's not set. The makefile, if possible, could pull the value from go env GOPATH instead of querying the shell.
I encountered this problem while developing a PKGBUILD. Users of my PKGBUILD will potentially face the same issues.
Please consider fixing it. For the time being, I will export the path to the installing shell.
Here's the output of ./build.sh

./build.sh
go get -u github.com/0xrawsec/gene
go get -u github.com/0xrawsec/golang-evtx
go get -u github.com/0xrawsec/golang-utils
printf "package main\n\nconst(\n    version=\"1.6.13\"\n    commitID=\"4401b38e4e48b366279380f2e39f0fe5f3eecbd9\"\n)\n" > version.go
make clean
make[1]: Entering directory '.../gene/gene'
rm -rf /release/*
make[1]: Leaving directory '.../gene/gene'
make init
make[1]: Entering directory '.../gene/gene'
mkdir -p /release
mkdir: cannot create directory ‘/release’: Permission denied
make[1]: *** [makefile:21: init] Error 1
make[1]: Leaving directory '.../gene/gene'
make: *** [makefile:14: all] Error 2

Implement actionnable rules

Create a new field in the rule format to list actions to execute when such a rule triggers.
This new field has for first intention to be used when Gene is used in real-time.

Document new rule format

Related issues: