I see the example have rules_medium_sysmon_performance_v3.json, so where I can find it

I input 370 evtx files from the system I am using in a real environment.
In this case, a query that creates about 2000~3000 sqlite db columns is made, which causes an error. The sqlite db is limited to creating 2000 columns by default.

Hello Zircolite team,

Can you add a feature for Zircolite, even if this feature will not be understood by everyone, can you add the possibility to delete the file after its analysis.
A bit like "logstash" with its "file_completed_action => "delete" option

A option like "--remove".

Thanks for your software and your jobs

@+ wagga :)

--exportcsv (zircolite_win10_nuitka_embedded.exe) generates no valid CSV format.

What is the delimiter?

Headers are not matching the content of the columns: descriptionrow_id --> space missing???

Running this on Windows 10, I get an error KeyError: 'SystemTime' on Line 229 of zircolite.py. I get the same error message when running it with zircolite_legacy.py and the Windows release version 2.0.0. I fixed my local version with a check for the key SystemTime, as shown below:

This creates a JSON exception if there is no SystemTime in the JSONLine.

if self.timeAfter != "1970-01-01T00:00:00" and self.timeBefore != "9999-12-12T23:59:59":
      timestamp = time.strptime(JSONLine["SystemTime"].split(".")[0].replace("Z",""), '%Y-%m-%dT%H:%M:%S')
      if timestamp > self.timeAfter and timestamp < self.timeBefore:
          JSONOutput.append(JSONLine)

Can you update it to something like this, which checks if that key exists:

if self.timeAfter != "1970-01-01T00:00:00" and self.timeBefore != "9999-12-12T23:59:59" and "SystemTime" in JSONLine:
    timestamp = time.strptime(JSONLine["SystemTime"].split(".")[0].replace("Z",""), '%Y-%m-%dT%H:%M:%S')
    if timestamp > self.timeAfter and timestamp < self.timeBefore:
        JSONOutput.append(JSONLine)

i've got a huge amount of event lots to process with Zircolite and put into elasticsearch. its an amazing tool, but the elasticsearch connector/exporter seems to lock up on me more often than not.

this is the command i'm running:
python3 zircolite.py --debug --evtx //Windows/system32/winevt --ruleset rules/rules_windows_generic.json --remote http://:9200 --index

there are no errors, it just hangs and stops submitting records. any suggestions would be appreciated.

(really do appreciate the tool)

Hi,

I'm trying to produce a use case of processing key assets for an IR in Zircolite and uploading to TS for timeline analysis, I've tried using the template with outputting to CSV and unable to have the correct header fields for datetime, message and timestamp_desc.

I understand the fields have the header required fields appended but shouldn't need to massage the csv to get it to process correctly.

Thoughts?

If you take this rules for example: https://github.com/SigmaHQ/sigma/blob/master/rules/windows/powershell/powershell_classic/posh_pc_susp_zip_compress.yml

And use sigmac as mentioned in the documentation, you will get something similar to (simplified condition):

"SELECT * FROM logs WHERE (Channel = "Windows PowerShell" AND HostApplication LIKE "%hidden%" ESCAPE '\')"

However the ingestion of powershell event do not create a HostApplication column so the query fail (silently)

The information is found within the message column.

Hi,
Thank you for the cool solution.

I am trying to use zircolite_lin_nuitka_embedded v 1.4.1 on linux and getting this error:

Traceback (most recent call last):
  File "/tmp/.mount_zircol5dxM6c/zircolite.py", line 393, in <module>
  File "/tmp/.mount_zircol5dxM6c/zircolite.py", line 251, in quitOnError
NameError: name 'consoleLogger' is not defined

First, thanks for all your work on this!

It would be nice to have an option to disable console output or consoleLogger.info, or simply write the results to a file for automated/remote runs of the tool.

Currently, I'm having an issue with the encoding when trying to run non-interactively via cmd and Powershell on Windows 10, and it seems the only way to get around it is to set the local/region prefs to UTF-8 system-wide, which might not be ideal in certain instances.

I can work on this if needed, as time allows.

Hey, There Is no possibility to fowards the event using --remote to a splunk specific index. Is there any way to do that with the --remote option ?

Thanks in advance

Currently, when I run Zircolite using the 'exportForSplunk' template each event that is generated contains the Sigma rule title and description, along with the information from the relevant evtx entry. Would you please consider adding the Sigma rule id to each event as well? Having that unique id would be helpful with some automation efforts.

I am importing a JSON file from Splunk and trying to analyze it. My detected_events.json shows empty. Do i need the EVTX or is it because of the splunk?

HI,
in MITRE it is "resource_development" not "ressource_development"
Test with my vm get "file_event_mal_vhd_download.yml" match 0 Ressource development.

In the same way get mamy FP.

From documentation:

Since Zircolite 2.2.0, if you have sigmatools >= 0.21, Zircolite is able to convert the rules on-the-fly if you provide a SIGMA config file and the sigmac path. It is very convenient for testing but you should avoid it since this is slower :

But they are no option --sigma or --sigmac valid

How much work would it take to ingest xml data rather than evtx files?

The XML data is generated by: https://github.com/williballenthin/EVTXtract. The tools recovers (partial) xml records from corrupted partitions.

Could you indicate if this is feasible and give some pointers towards ingesting xml data? I'd be happy to submit a PR if I get this to work.

I have a pretty large stack of EVTX file with actual TA artifacts in them. When running them in Zircolite it will process the 338 files (extract), however it will reach 60 ~ 90% in Processing EVTX stage and Kill the whole process.

I was able to process around 337 different EVTX files, however I have 1 4.4GB EVTX Security file that won't be processed.

Is there any way to process large sums of EVTX files with possible large filesize?

Examples:
--evtx C:"ExampleFolder"
--evtx C:\Folder"example".txt

I am trying Zircolite which uses the evtx parser to extract the events and it seems to get stuck while shelling out to the parser.

If I copy paste the commandline into a new terminal and run the subprocess command it returns really quickly .

Everything is running as admin. BTW this particular event file is confirmed to be actually empty.

Not sure if here is a more proper fix. I was looking at the 3cx sigma rules

https://github.com/SigmaHQ/sigma/blob/master/rules/windows/image_load/image_load_malware_3cx_compromise_susp_dll.yml

which look both in the hashes field or for a hash in a field associated with the algo i.e. sha256,md5,etc. The converted query causes Zircolite not to match because of the OR statement. This patch maps the individual hashes and allows these rules to fire.

index da24233..92bc36a 100755
--- a/zircolite.py
+++ b/zircolite.py
@@ -582,6 +582,22 @@ class JSONFlattener:
                             key = "".join(
                                 e for e in name[:-1].split(".")[-1] if e.isalnum()
                             )
+
+                        #Event 15 uses "Hash" instead of "Hashes". Hashing algo gets lower cased in rules
+                        if key == "Hashes" or key == "Hash":
+                            try:
+                                hashes = value.split(',')
+                                for fhash in hashes:
+                                    k,v = fhash.split('=')
+                                    kl = k.lower()
+                                    JSONLine[kl] = v
+                                    if kl not in self.keyDict:
+                                        self.keyDict[kl] = kl
+                                        fieldStmt += f"'{kl}' TEXT COLLATE NOCASE,\n"
+                            except Exception as e:
+                                self.logger.info(f"[+] Couldn't normalize out Hash list into individual hashes {e}")
+                                pass
+
                         JSONLine[key] = value
                         # Creating the CREATE TABLE SQL statement
                         if key.lower() not in self.keyDict:

python3 zircolite.py --evtx /opt/out//sysmon --debug --ruleset
[+] Cleaning unused objects
[+] Loading ruleset from : rules/rules_windows_sysmon_full.json
[+] Executing ruleset - 2083 rules
- Malicious DLL Load By Compromised 3CXDesktopApp [critical] : 9 events
- Potential Compromised 3CXDesktopApp Execution [high] : 9 events
- Potential Configuration And Service Reconnaissance Via Reg.EXE [medium] : 1 events
- Suspicious Msiexec Quiet Install [medium] : 1 events
- CurrentVersion Autorun Keys Modification [medium] : 1 events
- Remote Thread Creation in Suspicious Targets [high] : 1 events

When running Zircolite on a Linux Triage Package it does not parse audit logs with a number at the end. An example of one that does work is audit.log. One that does not work is audit.log.1 however when you add .log to the end for example audit.log.1.log it does work.
Could you please add handling of .log.number files to this thanks.

For .csv files could you please add a command line argument to specify the delimiter. I like to have comma separated rather than semi-colon due to automatic parsing by spreadsheet software. Something like --delimiter comma

Additionally I noticed the documentation refers to a .csv template that no longer exists. I got confused when reading it just for your reference in case people try like me to use the csv template instead of --csv

Hi,
I am a big fan of the tool. I'm lovin 'it. works very very well. Congratulations.

I wanted to inform you that I have created a module for kape so that the blue team community can use your tool. It's been available a few hours ago at:

https://github.com/EricZimmerman/KapeFiles/tree/master/Modules/Apps/GitHub

Also, if you don't mind, I'll create a fork of Zircolite and add some additions that may be interesting and powerful collaborate in the improvement of Zircolite.

Great job, the community needs tools like yours to help investigate more easily. Thank you

For old linux version there is an issue, would be good to have -lin-musl - one.
Issue type: /lib64/libc.so.6: version `GLIBC_2.26' not found (required by ./zircolite_lin)

In both shebangs, the default Python interpreter is hard-coded to /usr/bin/python3, though my own interpreter is installed to /usr/local/bin/python3:

root@5f64728ceaf4:/mnt/host# ls -l zircolite.py
-rwxrwxr-x 1 1000 1000 20227 Apr  6 18:59 zircolite.py
root@5f64728ceaf4:/mnt/host# ./zircolite.py
bash: ./zircolite.py: /usr/bin/python3: bad interpreter: No such file or directory
root@5f64728ceaf4:/mnt/host# which python3
/usr/local/bin/python3

Proposal : switch the shebang from #!/usr/bin/python3 to #!python3 (or even #!python) in order to increase compatibility.

I tried Zircolite v2.1.0 (zircolite_win10_nuitka_embedded.exe). Zircolite vs. an EVTX files works smoothly, but parsing an EVTX folder with the dropped evtx_dump.exe stops latest after 5% (17 EVTX files). I tried different terminals.

I gave also zircolite_win10_embedded.exe a chance...same issue. Please have a look. Thank you!

I'm new of Zircolite, but when I tty to use it, have this issues:

███████╗██╗██████╗  ██████╗ ██████╗ ██╗     ██╗████████╗███████╗
╚══███╔╝██║██╔══██╗██╔════╝██╔═══██╗██║     ██║╚══██╔══╝██╔════╝
  ███╔╝ ██║██████╔╝██║     ██║   ██║██║     ██║   ██║   █████╗
 ███╔╝  ██║██╔══██╗██║     ██║   ██║██║     ██║   ██║   ██╔══╝
███████╗██║██║  ██║╚██████╗╚██████╔╝███████╗██║   ██║   ███████╗
╚══════╝╚═╝╚═╝  ╚═╝ ╚═════╝ ╚═════╝ ╚══════╝╚═╝   ╚═╝   ╚══════╝

-= Standalone SIGMA Detection tool for EVTX/Auditd/Sysmon Linux =-

[+] Checking prerequisites
[+] Extracting events Using 'tmp-YIWJDQ4F' directory
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 161/161 [00:05<00:00, 28.50it/s]
[+] Processing events
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 161/161 [00:14<00:00, 11.20it/s]
[+] Creating model
[-] Unable to create table

Why?

Hello,
get a error in in first try.

My step:

D:\FrackSigma\sigma\tools>python sigmac -t sqlite -c config\generic\sysmon.yml -r ..\rules\windows\ -d --backend-option table=logs --output-fields title,id,description,author,tags,level,falsepositives,filename --output-format json -o rules.json

D:\FrackSigma\Zircolite>python zircolite.py -e d:\temp\sysmon_13_rdp_settings_tampering.evtx -r rules.json

    ███████╗██╗██████╗  ██████╗ ██████╗ ██╗     ██╗████████╗███████╗
    ╚══███╔╝██║██╔══██╗██╔════╝██╔═══██╗██║     ██║╚══██╔══╝██╔════╝
      ███╔╝ ██║██████╔╝██║     ██║   ██║██║     ██║   ██║   █████╗
     ███╔╝  ██║██╔══██╗██║     ██║   ██║██║     ██║   ██║   ██╔══╝
    ███████╗██║██║  ██║╚██████╗╚██████╔╝███████╗██║   ██║   ███████╗
    ╚══════╝╚═╝╚═╝  ╚═╝ ╚═════╝ ╚═════╝ ╚══════╝╚═╝   ╚═╝   ╚══════╝
             -= Standalone SIGMA Detection tool for EVTX =-

[+] Checking prerequisites
[+] Extracting EVTX Using 'tmp-DL9G50FQ' directory
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.18it/s]
[+] Processing EVTX
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 335.22it/s]
[+] Creating model
[+] Inserting data
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 9170.88it/s]
[+] Cleaning unused objects
[+] Loading ruleset from : rules.json
   [-] Load JSON ruleset failed, are you sure it is a valid JSON file ? : 'charmap' codec can't decode byte 0x8d in position 305265: character maps to <undefined>
[+] Executing ruleset - 0 rules
0it [00:00, ?it/s]
[+] Results written in : detected_events.json
[+] Cleaning

Finished in 0 seconds

D:\FrackSigma\Zircolite>python -V
Python 3.9.2

When get an error in Zircolite the color of the terminal is not restored

Hi!

Firstly, thanks so much for putting in your time to make this tool.

Would you be open to changing the dockerhub builds so that a "latest" tag is assigned in addition to the version tag? This would allow users to grab the image without needing to pin a version.

Trouble

Can not match rule when the Microsoft field name have a space in it

Usefull Informations

https://github.com/SigmaHQ/sigma/blob/master/rules/windows/builtin/win_event_log_cleared.yml
https://github.com/sbousseaden/EVTX-ATTACK-SAMPLES/blob/master/Defense%20Evasion/DE_104_system_log_cleared.evtx

 "rule": [
     "SELECT * FROM logs WHERE (EventID = \"1102\" AND Provider_Name = \"Microsoft-Windows-Eventlog\")"

In the DB it is simply Name