Git Product home page Git Product logo

whids's Introduction

IMPORTANT: this README and other documentation is not updated to reflect the changes made to latest beta releases (as a lot of changes happened). All this will be updated in time when a stable version will be released (see Roadmap until next release). Feel free to run beta releases on your own and ask questions if you need help.

logo

Slack Coverage Latest Commit Latest Release Downloads License

What

EDR with artifact collection driven by detection. The detection engine is built on top of a previous project Gene specially designed to match Windows events against user defined rules.

What do you mean by "artifact collection driven by detection" ?

It means that an alert can directly trigger some artifact collection (file, registry, process memory). This way you are sure you collected the artifacts as soon as you could (near real time).

All this work has been done on my free time in the hope it would help other people, I hope you will enjoy it. Unless I get some funding to further develop this project, I will continue developing it on a best effort basis. I will make all I can to fix issues in time and provide updates. Feel free to open issues to improve that project and keep it alive.

Why

  • Provide an Open Source EDR to the community
  • Make transparency on the detection rules to make analysts understand why a rule triggered
  • Offer powerful detection primitives though a flexible rule engine
  • Optimize Incident Response processes by drastically reducing the time between detection and artifact collection

How

big-picture

NB: the EDR agent can be ran standalone (without being connected to an EDR manager)

What you have to know

  • It works only on Windows using ETW logs
  • Relies on Sysmon for all the heavy lifting (kernel component)
  • Very powerful and customizable detection engine (powered by gene)
  • Built by an Incident Responder for all Incident Responders with the aim of making their job less painful
  • Low footprint -> no process injection
  • Can co-exist with any antivirus product (advised to run it along with MS Defender)
  • Designed for high throughput. It can easily enrich and analyze up to 15M events a day per endpoint without performance impact. Good luck to achieve that with a SIEM.
  • Easily integrable with other tools (Splunk, ELK, MISP ...)
  • Integrated with ATT&CK framework
  • There is a powerful administrative API to ease management of large deployments (no GUI yet)

Installation

Requirements

  1. Install Sysmon
  2. Configure Sysmon
    • You can find optimized Sysmon configurations here
    • Logging any ProcessCreate and ProcessTerminate is mandatory
  3. Take note of the path to your Sysmon binary because you will need it later on

NB: event filtering can be done at 100% with Gene rules so do not bother creating a complicated Sysmon configuration.

Pre-Installation Recommendations

In order to get the most of WHIDS you might want to improve your logging policy.

  • Enable Powershell Module Logging
  • Audit Service Creation: gpedit.msc -> Computer Configuration\Windows Settings\Security Settings\Advanced Audit Policy Configuration\System Audit Policies\System\Audit Security System Extension -> Enable
  • Enable File System Audit. Sysmon only provides FileCreate events when new files are created, so if you want/need to log other kind of accesses (Read, Write, ...) you need to enable FS Auditing.
    1. gpedit.msc -> Computer Configuration\Windows Settings\Security Settings\Advanced Audit Policy Configuration\System Audit Policies\Object Access\Audit File System -> Enable
    2. Right Click Any Folder -> Properties -> Security -> Advanced -> Auditing -> Add
      1. Select a principal (put here the name of the user/group you want the audit for). Put group Everyone if you want to log access from any user.
      2. Apply this to is used to select the scope of this audit policy starting from the folder you have selected
      3. Basic permissions select the kinds of accesses you want the logs to be generated for
      4. Validate
    3. File System auditing logs will appear in the Security log channel
  • If you want an antivirus to run on your endpoints, keep Microsoft Defender, first because it is a good AV but also because it logs alerts in a dedicated log channel Microsoft-Windows-Windows Defender/Operational monitored by the EDR.

EDR Endpoint agent (Whids.exe)

This section covers the installation of the agent on the endpoint.

  1. Download and extract the latest WHIDS release https://github.com/0xrawsec/whids/releases
  2. Run manage.bat as administrator
  3. Launch installation by selecting the appropriate option
  4. Verify that files have been created at the installation directory
  5. Edit configuration file by selecting the appropriate option in manage.bat or using your preferred text editor
  6. Skip this if running with a connection to a manager, because rules will be updated automatically. If there is nothing in the rules directory the tool will be useless, so make sure there are some gene rules in there. Some rules are packaged with WHIDS and you will be prompted to choose if you want to install those or not. If you want the last up to date rules, you can get those here (take the compiled ones)
  7. Start the services from appropriate option in manage.bat or just reboot (preferred option otherwise some enrichment fields will be incomplete leading to false alerts)
  8. If you configured a manager do not forget to run it in order to receive alerts and dumps

NB: At installation time the Sysmon service will be made dependent of WHIDS service so that we are sure the EDR runs before Sysmon starts generating some events.

EDR Manager

The EDR manager can be installed on several platforms, pre-built binaries are provided for Windows, Linux and Darwin.

  1. Create TLS certificate if needed for HTTPS connections
  2. Create a configuration file (there is a command line argument to generate a basic config)
  3. Run the binary

Configuration Examples

Please visit doc/configuration.md

Further Documentation

Known Issues

  • Does not work properly when ran from a network share mapped as a network drive (this case prevent whids to identify itself and thus generate some noise). Example: if \\vbox\test is mounted as Z: drive, running Z:\whids.exe won't work while running \\vbox\test\whids.exe actually would.

Roadmap until next release

  • find a new name to the project because we all agree it sucks
  • better sysmon integration (config, deployment, update)
  • endpoint configuration from manager
  • tooling management (update, install), like OSQuery
  • code refactoring and optimization
  • implement a performance monitor
  • get rid of any on-disk configuration
  • implement IOC management capabilities
  • ETW support
  • automatic documentation (OpenAPI) and testing of manager's API
  • provide endpoint system information in manager
  • implement actionable rules
  • provide canary file management
  • builtin commands to be executed by endpoints
  • provide Incident Response reports about endpoints
  • overall manager API improvement
  • provide event streams so that a client can receive events in realtime
  • standardize HTTP headers
  • provide a python library to interact with EDR manager (https://github.com/0xrawsec/pywhids)

Changelog

v1.7

  • New Administrative HTTP API with following features:
    • Manage endpoints (list, create, delete)
    • Get basic statistics about the manager
    • Execute commands on endpoints and get results
      • Can drop files prior to execution, to execute binaries/scripts not present on endpoint. Dropped files are deleted after command was ran.
      • Can retrieve files (post command execution), to retrieve results of the command
    • Collect files from endpoints for forensic purposes
    • Contain / Uncontain endpoints by restricting any network traffic except communication to the manager.
    • Query endpoints logs
    • Query endpoints alerts
    • Pivot on a timestamp and retrieve logs/alerts around that time pivot
    • Access endpoint report
      • Scoring (relative to each environment) allowing to sort endpoints and spot the ones behaving differently from the others.
      • Alerts / TTPs observed on a given time frame
    • Manage rules (list, create, update, save, delete)
  • Integration with Sysmon v12 and v13
    • Integrate ClipboardData events
      • Put the content of the clipboard data inside the event to allow creating rule on the content of the clipboard
    • Integrate ProcessTampering events
      • Enrich event with a diffing score between .text section on disk and in memory
  • Implemented certificate pinning on client to enhance security of the communication channel between endpoints and management server
  • Log filtering capabilities, allowing one to collect contextual events. Log filtering is achieved by creating Gene filtering rules (c.f. Gene Documentation).
  • Configuration files in TOML format for better readability
  • Better protection of the installation directory

Related Work

Sponsors

Tines

Github:https://github.com/tines Website:https://www.tines.com/ Twitter:@tines_io

whids's People

Contributors

qjerome avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

whids's Issues

Dump process tree

The IDS maintains a structure to track processes.
The idea is to recursively go through that structure and generate a full process tree when desired.

Implement actionnable rules

Implement a way to have actionable rules for doing several actions:

  • kill process and prevent it to run again
  • block (network) process
  • ...

NB: this feature needs to implement a new field in the rule definition (0xrawsec/gene#11)

Allow manager to run commands on endpoints

Implement different cases:
- commands where output is stdout
- commands where output is a file
- execute binaries which are not initially present on the machine (provided by manager)

Generate IR ready reports on detections

The idea is to create reports for IR when detection triggers

  • can be managed the same way artifacts are dumps
  • use OSQuery for generating reports as it provides standard JSON output
  • implements an action to do that too

MISP sightings

Leverage MISP API to report IoC sightings by the EDR to a MISP instance.

Reload rules on change

Reload rules when a rule file is changed on disk
Don't forget to handle the case where the added rule causes errors.

ERROR - Upload handler failed to dump file .... Bad guid

Hi,

I have a manager running on Ubuntu, and whids running on Win 10 (running in VBox).
I get the following type of errors popping up in the manager's output:

2019/10/02 11:25:25 ERROR - Upload handler failed to dump file ({19480def-65c1-5d94-0000-001081270200}/c9df481b3d530a4c9d79c6cd526a186f/c9df481b3d530a4c9d79c6cd526a186f_event.json.gz): Bad guid

I think it's a problem of case sensitivity here:

guidRe = regexp.MustCompile(`\{[A-F0-9]{8}-([A-F0-9]{4}-){3}[A-F0-9]{12}\}`)

`(?i)\{[A-F0-9]{8}-([A-F0-9]{4}-){3}[A-F0-9]{12}\}`

should work, but I am unsure you need case sensitivity elsewhere.

cheers,
jlouis

Improve dumping feature for events with CommandLine fields

Thanks to enrichment feature, the CommandLine field is set to several Sysmon events.
However artifact dumping from CommandLine field is only available for some events.
Thus it might be relevant to extend this feature to all events having a CommandLine field, this would be very helpful to dump files appearing in the command line of a process such as scripts, documents and so forth.

Way to EDR

Hi!
Do you want to make open source EDR system from your whids? Are you planning to leave your software free?
If i right understand your project can log in Log channel windows and take logs and sent another place?
It will be cool to add support sent via kafka or another event broker.
I have some idea which can be useful for edr system for more advance monitoring hosts and protect users from hackers.

Do you planning develop gene and evtxdump/evtxmon in the future ?

Thank for your project!

Issue forwarding log from endpoint to manager

Endpoint fails to delete the logs after sending those to the manager.
This makes the logs being continuously re-sent to the manager and thus duplicating logs.

This bug is due to an improper file closing.

Ability to tag an alert based on MITRE ATT&CK Framework

With MITRE ATT&CK becoming a standard way to classify an attacker action (aka technique) across the cyber kill change, it would be good to be able that, when a rule matches, be able to enrich that rule with the actual MITRE ATT&CK Tactic and Technique being used.

This has several benefits opening the door for doing correlation and eventually be able to draw (via some sort of dot notation graphs) the attacker objectives.

For example, this rule:

  "Name": "MaliciousLsassAccess",
  "Tags": ["Mimikatz", "Credentials", "Lsass"],
  "Meta": {
    "EventIDs": [10],
    "Channels": ["Microsoft-Windows-Sysmon/Operational"],
    "Computers": [],
    "Traces": [],
    "Criticality": 10,
    "Author": "0xrawsec"
  },
  "Matches": [
    "$ct: CallTrace ~= 'UNKNOWN'",
    "$lsass: TargetImage ~= '(?i:\\\\lsass\\.exe$)'"
  ],
  "Condition": "$lsass and $ct"
}```


Potentially matches [T1003](https://attack.mitre.org/techniques/T1003/)  Credential Dumping, so it would be nice to have this 

`

{
  "Event": {
    "EventData": {
      "CallTrace": "C:\\Windows\\SYSTEM32\\ntdll.dll+4bf9a|C:\\Windows\\system32\\KERNELBASE.dll+189b7|UNKNOWN(00000000259123BC)",
      "GrantedAccess": "0x1410",
      "SourceImage": "C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\powershell.exe",
      "SourceProcessGUID": "{49F1AF32-DD18-5A72-0000-0010042C0A00}",
      "SourceProcessId": "2248",
      "SourceThreadId": "3308",
      "TargetImage": "C:\\Windows\\system32\\lsass.exe",
      "TargetProcessGUID": "{49F1AF32-DB3B-5A72-0000-001013690000}",
      "TargetProcessId": "492",
      "UtcTime": "2018-02-01 11:24:53.331"
    },
    "GeneInfo": {
      "Criticality": 10,
      "Signature": [
        "MaliciousLsassAccess"
      ]
      "ATT&CK Tactic"   : "Credential Access",
      "ATT&CK Technique : "T1003",
      "ATT&CK Reference": "https://attack.mitre.org/techniques/T1003/"
    },
    "System": {
        "Classical Windows Event System Section": "..."
    }
  }
}`

Intergrate with SIGMA

As SIGMA is already an open source log signature standard adopted by many tools it might be beneficial to integrate it with whids. As an instant benefit the already tested SIGMA detection rules can be used.

REF: https://github.com/Neo23x0/sigma

Enrich candidate sysmon event with CurrentDirectory information

CurrentDirectory information is used by the dump routine to dump files appearing in the CommandLine field of the event. Since CommandLine is propagated to almost any sysmon event, it would make sense to also enrich event with CurrentDirectory to enhance file artifact dumping.

Benchmark and performance

Hi there, great tool. I was wondering if you have run some sort of benchmarks or performance testing on workstations upon running a threat emulation like Caldera or even Cobalt Strike not only to measure the detection rate but to measure performance (aka CPU and Memory).

Thanks.

Configuration to have alerts output in json so its easier to read and parse

For standalone installations (aka those that do not use a manager), it would be good to have a config option to generate the alerts output in JSON, with a log rotation of a configurable value (maybe with a default value of 24 hours), so upon 24 hours the log is rotated and the old alerts are gzipped.

There are good reasons for doing that but a potential one is to be able to read those alerts and ship them via Filebeat (https://www.elastic.co/products/beats/filebeat) and send it to a remote location.

Integrate Security Event 4697

This event is not logged by default and needs to be enabled via gpedit.msc
https://docs.microsoft.com/en-us/windows/security/threat-protection/auditing/event-4697

It is used to audit service creation:

 <EventData>
 <Data Name="SubjectUserSid">S-1-5-18</Data> 
 <Data Name="SubjectUserName">WIN-GG82ULGC9GO$</Data> 
 <Data Name="SubjectDomainName">CONTOSO</Data> 
 <Data Name="SubjectLogonId">0x3e7</Data> 
 <Data Name="ServiceName">AppHostSvc</Data> 
 <Data Name="ServiceFileName">%windir%\\system32\\svchost.exe -k apphost</Data> 
 <Data Name="ServiceType">0x20</Data> 
 <Data Name="ServiceStartType">2</Data> 
 <Data Name="ServiceAccount">localSystem</Data> 
 </EventData>

Enrich with:

  • Hash
  • ServiceType -> String
  • ServiceStartType -> String

Dump:

  • ServiceFileName

Integrate with Sysmon 11

Integrate with new features introduces in Sysmon 11:

  • FileDelete events + enrichment
  • Collect files archived by FileDelete feature

Automatic canary folder management

  • make canary folders configurable
    • configure file extensions
    • configure folder locations
    • actions to handle touched canaries
  • detect when canaries are touched (evaluate two possibilities)
    • generate rule(s) on the fly
    • through event hooks

Find a way of log filtering

Log filtering would allow one not only to log alerts but also to log regular events going through the engine.
The aim of this feature is mainly to bring additional context around alerts being reported by the engine.

Implementation: implement the core of the feature in Gene and reflect changes in IDS

Implement an Admin API

Features to implement:

  • Basic endpoint management features
    • Add an endpoint to the managing server
    • Delete an endpoint from the managing server
    • Get information on all endpoints (IP, hostname, ID ...)
    • Get information on a single endpoint
  • Ask an endpoint to execute a command
    • Command can also drop files (before execution), to execute scripts or binaries not on endpoint
    • Command can retrieve files (after execution), to collect artifacts or output files
  • Retrieve the results of a command executed (stdout, stderr, files collected)
  • Search logs by endpoint
  • Search alerts by endpoint
  • Generate reports on endpoints, with a metric to prioritize endpoints between them

Event log logging

I wouldn't log to the application channel as it's world readable by users by default, can you log to a new channel with only admin read permissions?

Implement event count

Some events might be interesting to have a counter associated with them, such events are:

  • Sysmon FileCreate -> can be used to detect cryptolocker
  • Sysmon FileDelete -> can be used to detect cryptolocker
  • Sysmon FileDeleteDetected -> can be used to detect cryptolocker
  • FileSystem events -> can be used to detect cryptolocker
  • Sysmon ProcessCreate -> can be used to detect RATs, WebShells ...

Enrich events with Hashes

Extract Hashes information from Sysmon CreateProcess event and propagate this to any other relevant events.

This can be used to better whitelist but also to ease alert analysis

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.