When using the default settings of logging JSON to a result.txt file, there are empty lines between each JSON object

The u"\n" should not be appended when serializing each result to JSON
https://github.com/sophos/Sophos-Central-SIEM-Integration/blob/v1.1.0/siem.py#L278

In our organization we have standardized on Python 3 for our codebase. I know Python 3 fairly well and would be willing to make a pull request with a working version of this program in Python 3 (and see if I just can't get it backwards compatable with 2 as well). Is this something that you are able to accept, or is this something I'll have to wait on?

Downloaded the zip.
Extracted to downloads folder
Modified config.ini
Ran python siem.py

CONFIG.INI

# API Access URL + Headers
# API token setup steps: https://community.sophos.com/kb/en-us/125169
token_info = url: https://api1.central.sophos.com/gateway, x-api-key: xxXXxxXXXxxXXxxXXXxxxxXXxxXXXxxXXxxXXXxx, Authorization: Basic xxXXxxXXxXxxXXxxXXxXxxXXxxXXxXxxXXxxXXxXxxXXxxXXxXxxXXxxXXxXxxXXxxXXxXxxXXxxXXxXxxXXxxXXxXxxXXxxXXxXxxXXxxXXxXxxXXxxXXxXxxXXxxXXxXxxXXxxXXxXxxXXxxx=

# format can be json, cef or keyvalue
format = json

# filename can be syslog, stdout, any custom filename
filename = result.txt

# endpoint can be event, alert or all
endpoint = event

# syslog properties
# for remote address use <remoteServerIp>:<port>, for e.g. 192.1.2.3:514
# for linux local systems use /dev/log
# for MAC OSX use /var/run/syslog
address = /var/run/syslog
facility = daemon
socktype = udp

ERROR:

Traceback (most recent call last):
  File "/Users/username/Downloads/Sophos-Central-SIEM-Integration-master/siem.py", line 476, in <module>
    main()
  File "/Users/username/Downloads/Sophos-Central-SIEM-Integration-master/siem.py", line 153, in main
    token = config.Token(cfg.token_info)
  File "/Users/username/Downloads/Sophos-Central-SIEM-Integration-master/config.py", line 30, in __getattr__
    return self.config.get('login', name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ConfigParser.py", line 607, in get
    raise NoSectionError(section)
ConfigParser.NoSectionError: No section: 'login'

PYTHON VERSION:

Python 2.7.10

I am attempting to writing parsing rules for a LogRhythm SIEM, however, am faced with the challenge that the data is not in a consistent order when using either the CEF or KeyValue output formats.

Since the parsing rules for several SIEM solutions are order-dependent, is it possible to force either a manual order, or sort the fields in a particular way, prior to outputting them to CEF or KeyValue format?

I understand that the underlying issue stems from the fact that JSON is non-pedantic about field ordering, and flattening the JSON just outputs it in whatever order the JSON data has it. Since this order changes from time to time, this is resulting in what I am seeing.

I have provided some sample logs below in KeyValue format.

UPDATING logs at different times

`2017-07-28T00:05:59.114Z rt="2017-07-28T00:05:59.114Z"; end="2017-07-28T00:05:59.101Z"; severity="low"; duid="duid"; whitelist_properties="{}"; dhost="host-a"; endpoint_type="computer"; endpoint_id="endpoint_id"; suser="user 1"; group="UPDATING"; customer_id="customer_id"; type="Event::Endpoint::UpdateSuccess"; id="id"; name="Update succeeded";

2017-08-06T10:27:42.481Z rt="2017-08-06T10:27:42.481Z"; group="UPDATING"; name="Update succeeded"; whitelist_properties="{}"; dhost="host-b"; endpoint_type="server"; endpoint_id="endpoint_id"; suser="n/a"; end="2017-08-06T10:27:42.474Z"; customer_id="customer_id"; type="Event::Endpoint::UpdateSuccess"; id="id"; severity="low";
`

PERIPHERAL logs at different times

`2017-08-06T23:15:26.039Z rt="2017-08-06T23:15:26.039Z"; end="2017-08-06T23:15:26.039Z"; name="Peripheral allowed: SAMSUNG Mobile USB Modem #2"; duid="duid"; whitelist_properties="{}"; dhost="host-c"; endpoint_type="computer"; endpoint_id="endpoint_id"; suser="user a"; group="PERIPHERALS"; customer_id="customer_id"; type="Event::Endpoint::Device::AlertedOnly"; id="id"; severity="low";

2017-07-27T22:56:41.855Z rt="2017-07-27T22:56:41.855Z"; end="2017-07-27T22:56:41.855Z"; severity="low"; duid="duid"; whitelist_properties="{}"; dhost="host-d"; endpoint_type="computer"; endpoint_id="endpoint_id"; suser="user b"; group="PERIPHERALS"; customer_id="customer_id"; type="Event::Endpoint::Device::AlertedOnly"; id="id"; name="Peripheral allowed: WD My Passport 0730 USB Device";
`

Is possible add tag for add name of customer in CEF(or another) format in the first of the line?
This is actually message;
<30>Access was blocked to "www.veterinariotraumatologo.com" because of "Mal∕HTMLGen-A".|CEF:0|Sophos|sophos central|1.0|Event::Endpoint::WebFilteringBlocked|Access was blocked to "www.vetesadfdsao.com" because of "Mal∕HTMLGen-A".|1|sour
And my petition is:
CUSTOMER_NAME<30>Access was blocked to "www.veterinariotraumatologo.com" because of "Mal∕HTMLGen-A".|CEF:0|Sophos|sophos central|1.0|Event::Endpoint::WebFilteringBlocked|Access was blocked to "www.vetesadfdsao.com" because of "Mal∕HTMLGen-A".|1|sour
Thanks

Hi,
I tried to use api to get list of my endpoint devices but it just gives me random 100 devices every request but it shows me there are 909 device exists!

{
"total": 909,
"filtered": 909,
"items": [],
"next_key": null
}

Configured output to go to a remote syslog server and returned "syslog is not supported on this platform".
Python 2.7.9 and script is running on a Windows 10 machine.
Output to all other formats work as expected.

Is there an extra step to push to the remote syslog server?

Using a fresh pull on the repo, I see the following on my first run:

  File "siem.py", line 476, in <module>
    main()
  File "siem.py", line 190, in main
    process_endpoint(endpoint, opener, endpoint_config, token)
  File "siem.py", line 223, in process_endpoint
    write_json_format(results, siem_logger)
  File "siem.py", line 257, in write_json_format
    for i in results:
  File "siem.py", line 330, in call_endpoint
    events = json.loads(events_response)
  File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
    s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'

The code in question

        events_response = request_url(opener, events_request)
        if DEBUG:
            log("RESPONSE: %s" % events_response)
        events = json.loads(events_response)

Data always come as "unicode" type in the format_extension method.
When debugging with this code:

print type(data)
print data

We usually get an output like this for every field:

<type 'unicode'>
d98e9a76-878e-42cc-9e93-72f410df7863

But sometimes data comes as different type and causes a Type Error:

<type 'int'>
1484926007817
Traceback (most recent call last):
  File "./siem.py", line 539, in <module>
    main()
  File "./siem.py", line 209, in main
    process_endpoint(endpoint, opener, endpoint_config, creds)
  File "./siem.py", line 267, in process_endpoint
    write_cef_format(results, siem_logger)
  File "./siem.py", line 316, in write_cef_format
    siem_logger.info(format_cef(flatten_json(i)) + u'\n')
  File "./siem.py", line 527, in format_cef
    value = format_extension(value)
  File "./siem.py", line 471, in format_extension
    return EXTENSION_PATTERN.sub(r'\\\1', data)
TypeError: expected string or buffer

I quick fixed with following code, but I'm not sure if the fix should actually be elsewhere in the code:

def format_extension(data):
    # equal sign and backslash in extension value must be escaped
    # escape group with backslash
    if not isinstance(data, unicode):
        return EXTENSION_PATTERN.sub(r'\\\1', str(data).encode('utf8'))
    return EXTENSION_PATTERN.sub(r'\\\1', data)

Hi I am trying to export syslogs to a SIEM using this script. The script downloads the logs fine and stores them in the results.txt but it is seems to not be sending the logs onto the SIEM. Im preety sure my config is right(copied below), but seems like nothing at all is being sent.

[login]
-# API Access URL + Headers
-# API token setup steps: https://community.sophos.com/kb/en-us/125169
token_info = mytokeninfo

-# format can be json, cef or keyvalue
format = json

-# filename can be syslog, stdout, any custom filename
filename = syslog

-# endpoint can be event, alert or all
endpoint = event

-# syslog properties
-# for remote address use :, for e.g. 192.1.2.3:514
-# for linux local systems use /dev/log
-# for MAC OSX use /var/run/syslog
address = xxx.xxx.xxx.xxx:514
facility = daemon
socktype = udp

edit: after checking a tcpdump of what is being sent when i run the script i get this:

-Syslog message id: : PAM unable to dlopen(/usr/lib64/security/pam_passwdqc.so): /usr/lib64/security/pam_passwdqc.so: cannot open shared object file: No such file or directory
-Syslog message id: : PAM adding faulty module: /usr/lib64/security/pam_passwdqc.so

among some other things also. Think this may have something to do with it.

Hey team,

We're trying to leverage this API to build out automated reports for our clients (as well as monitoring their activity).

We've noticed a bit of a lack of detail that would be extremely beneficial. Specifically it would be useful to have the CATEGORY supplied for the object that is blocked or by-passed. In this example the Web Filter.

{"customer_id": "1111111111111111111111111", "severity": "low", "endpoint_id": "111111111111111111", "endpoint_type": "computer", "source_info": {"ip": "10.80.111.111"}, "type": "Event::Endpoint::WebControlViolation", "id": "111111111111111111", "group": "WEB", "name": "User bypassed category block to 'https://config.edge.skype.com'", "datastream": "event", "duid": "1111111111", "rt": "2020-07-09T19:05:59.696Z", "end": "2020-07-09T19:00:57.000Z", "suser": "Joe User", "dhost": "COMPUTER NAME"}

It would be nice to know what the CATEGORY was.

HI,

I am trying to install it on red hat and send it to QRadar. However it is giving me below error.

Sophos state file not found
Traceback (most recent call last):
File "siem.py", line 404, in
main()
File "siem.py", line 401, in main
run(options, config_data, state_data)
File "siem.py", line 393, in run
endpoint, options, config_data, state
File "siem.py", line 366, in get_alerts_or_events
api_client_obj = api_client.ApiClient(endpoint, options, config, state)
File "/root/Sophos-Central-SIEM-Integration/api_client.py", line 68, in init
self.opener = self.create_request_builder()
File "/root/Sophos-Central-SIEM-Integration/api_client.py", line 94, in create_request_builder
handler = urlrequest.HTTPSHandler()
AttributeError: module 'urllib.request' has no attribute 'HTTPSHandler'

Hi Sophos,

Apologies if I have missed how to configure this in the config.ini but it does not appear that it this tool supports multi tenants.

Looking at the config.ini it was not clear how you would handle multi tenant credentials?

Instead of having just the customer id field it would be nice to have the actual customer name come over.
This would help with separating multiple entities when using a SIEM to collect data from multiple customers.

When I run the siem.py, I get the following error. The name_mapping.py file is in the same folder as siem.py. Not sure what I'm doing wrong

Traceback (most recent call last):
File "C:\Users\xxxx\Desktop\Sophos-Central-SIEM-Integration-master\siem.py", line 40, in
import name_mapping
ModuleNotFoundError: No module named 'name_mapping'

There is insufficient output from the siem.py script; No timestamps; No warning levels;

In addition, output by default contains sensitive authentication and jwt tokens - if sent to a syslog - thats a possible penetration..

Have workaround; can share

Traceback (most recent call last):
File "siem.py", line 473, in
main()
File "siem.py", line 190, in main
process_endpoint(endpoint, opener, endpoint_config, token)
File "siem.py", line 244, in process_endpoint
write_json_format(results, siem_logger)
File "siem.py", line 254, in write_json_format
for i in results:
File "siem.py", line 327, in call_endpoint
events = json.loads(events_response)
File "/usr/local/lib/python3.4/json/init.py", line 312, in loads
s.class.name))
TypeError: the JSON object must be str, not 'bytes'

Currently describes requires python2 needs to utilize python3 due to EOL

Hi Guy,
I try to use script for API contact but nothing happen with result.txt . I sure that API token and Client, Secret ID is collect.

Is there a way to script out multiple API URL's and Tokens if one has sub estates in the same config file?

Getting a HTTP Error 500. I had previously tested pulling down logs with no error, but now we are getting this Internal Server Error.

Error during request. Error code: 500, Error message:
Traceback (most recent call last):
File "siem.py", line 476, in
main()
File "siem.py", line 190, in main
process_endpoint(endpoint, opener, endpoint_config, token)
File "siem.py", line 223, in process_endpoint
write_json_format(results, siem_logger)
File "siem.py", line 257, in write_json_format
for i in results:
File "siem.py", line 327, in call_endpoint
events_response = request_url(opener, events_request)
File "siem.py", line 385, in request_url
response = opener.open(request)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 437, in open
response = meth(req, response)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 550, in http_response
'http', request, response, code, msg, hdrs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 475, in error
return self._call_chain(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 558, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 500: Internal Server Error

Errors when we run the "python siem.py" command:

Traceback (most recent call last):
File "siem.py", line 476, in
main()
File "siem.py", line 153, in main
token = config.Token(cfg.token_info)
File "/run/Sophos/config.py", line 38, in init
self.url = m.group("url")
AttributeError: 'NoneType' object has no attribute 'group'

WE have imported the "keys" or tokens from the Sophos Central Console into the config.ini file.

we set the SOPHOS_SIEM_HOME environment variable to point to the folder where config.ini, siem_cef_mapping.txt, siem.py etc are located.

In the ~500 line script there is no method called NoneType and the only time the word group is uses, is in a comment.

How do we fix this issue or run the 'python siem.py" command to get the desired results w/o errors?

Thank You, David

Hi team,

I went to install the latest version on a machine today, and realized that when I configured the data input in Splunk, the events were not coming in proper JSON format - they were broken up into lines, so instead of a single event with the following:
{ customer_id: xxx, datastream: xxx }

I was actually getting multiple events in the logs:
{
customer_id: xxx

you get the idea :)

Once I reverted to the latest 1.x version things worked just fine and the events were generated and indexed properly. Guessing this is a bug but thought I'd raise the issue so you could take a look. Hoping it's not a feature anyway :)

Thanks in advance!

Hello
i try to get the logs from sophos to send them to my siem.
i set the remote ip in the syslog properties
port 514 udp
the facility is daemon
i set the filename were i want him to save the file.

i make a cron job run any 1 min.
the file is created and pull the logs.
but my problem is the script not forwarding the logs to my siem using syslog.
the linux server itself send to my siem system logs correctly but the sophos logs not showing.
i use ubuntu 22.04.

Thank you

I just clone the project, put API Access URL + Headers in config.ini and try to run siem.py, but I got this error:

Traceback (most recent call last):
  File "siem.py", line 24, in <module>
    import api_client
  File "/home/centos/Sophos-Central-SIEM-Integration/api_client.py", line 511
    whoami_url = f"https://{self.config.api_host}/whoami/v1"
                                                           ^
SyntaxError: invalid syntax

I'm using python 3.5 in a Centos 7 box:

Python 3.5.1

config.ini:

# API Access URL + Headers
# API token setup steps: https://community.sophos.com/kb/en-us/125169
token_info = url: https://api5.central.sophos.com/gateway, x-api-key: xxxxxxxxxx, Authorization: Basic xxxxxxxx=

# Client ID and Client Secret for partner
# <Copy Client ID and Client Secret from Sophos Central here>
client_id = 
client_secret = 
# Customer tenant Id
tenant_id = 

# Host URL for Oauth token
auth_url = 

# whoami API host url
api_host = 

# format can be json, cef or keyvalue
format = json

# filename can be syslog, stdout, any custom filename
filename = result.txt

# endpoint can be event, alert or all
endpoint = event

# syslog properties
# for remote address use <remoteServerIp>:<port>, for e.g. 192.1.2.3:514
# for linux local systems use /dev/log
# for MAC OSX use /var/run/syslog
#address = /var/run/syslog
address = xxx.yyy.zzz.www:514
facility = daemon
socktype = udp

# cache file full or relative path (with a ".json" extension)
state_file_path = state/siem_sophos.json

Where is my error?

I'm using -c option for using different file .ini for different sophos account.
For each .ini I specify different API access and log file name.

I have a problem: the name of the generated status file is always the same ("siem_lastrun_events.obj")

Is it possible add a variable in the .ini file to configure different "state_file_name"?

Is there a way to modify the siem.py script to only show High and very high events in the SEVERITY_MAP?

SEVERITY_MAP = {"none": 0, "low": 1, "medium": 5, "high": 8, "very_high": 10}

Different log types are grouped together but the explanation is not 100%. Can this be better explained in the Read.me?

'--light', - It is not apparent what will not be logged using this option
'--quiet', - It is not apparent what will not be logged using this option

will it be possible to better explain the two? What is the difference between light and quiet?

If I only want to log problems like malware detected, devices not compliant, or bad websites visited will light be enough?

Hi,

I'm getting this error when trying to run 'python siem.py' after editing the config.ini file with relevant token info:

root@Rg-siemlog:/opt/Sophos-Central-SIEM-Integration# python siem.py
Config loaded, retrieving results for 'XXXXX'
Config retrieving results for 'Basic XXXXX'
Config endpoint=/siem/v1/events, filename='result.txt' and format='json'
Config state_file='/opt/Sophos-Central-SIEM-Integration/state/siem_lastrun_events.obj' and cwd='/opt/Sophos-Central-SIEM-Integration'
No datetime found, defaulting to last 12 hours for results
Retrieving results since: 1565467974
URL: https://api5.central.sophos.com/gateway/siem/v1/events?from_date=1565467974&limit=1000
Traceback (most recent call last):
File "siem.py", line 494, in
main()
File "siem.py", line 212, in main
process_endpoint(endpoint, opener, endpoint_config, token)
File "siem.py", line 244, in process_endpoint
write_json_format(results)
File "siem.py", line 274, in write_json_format
for i in results:
File "siem.py", line 347, in call_endpoint
events = json.loads(events_response)
File "/usr/lib/python2.7/json/init.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Please advise.
Thanks!

Hi all,

I am trying to use this siem.py with setting up my configuration as follows:

[login]
client_id = $CLIENTID
client_secret = $CLIENTSECRET
tenant_id = 
auth_url = https://id.sophos.com/api/v2/oauth2/token
api_host = api.central.sophos.com
format = json
filename = syslog
endpoint = all
address = address = "${LOGSTASHADDRESSSOPHOS}"
facility = daemon
socktype = udp
state_file_path = sophos/state/siem_sophos.json

As we may need to set this up for a few of our clients, and we are running this on K8s, the best way we can do this would be by using env vars that are passed to each of the pods so we can run multiple instances with each clients own IDs and secrets.

Reading up on the configParser, this might need to be written into the python scripts. This issue is to check if something like this is not already implemented.

Ref: https://stackoverflow.com/questions/26586801/configparser-and-string-interpolation-with-env-variable

We are having issues parsing events been sent via TCP because the scripts adds facility <30> to every event. How do we remove these?
Can't find a configuration item.

I noticed a previous issue reported that the KeyValue formats are not consistant. The comments related said the issue was fixed however I am still observing issues with less than 8% of our logs (in that population was the MALWARE group of logs)

Here is the siem.py file that we changed to fix the consistency issues. Appears that if the value of a key is null the key is dropped and the order is not consistent. With trying to fix the issue I ran into some encoding issues which I fixed by changing the characters that could not be encoded as ascii characters to xml.

This is now working with our SIEM and thought we would pass on the code.

Enjoy,

siem.txt

Customer has an RSA Netwitness log feed setup successfully with version 1.1.0 of SIEM.py script (using Python 2.7).

They have just setup a new feed to QRadar with version 2.0.1 (Sophos script) and Python 3.10 on the same server (Windows Server 2016).

The script is successfully pulling logs from Central and Forwarding to QRadar.

The issue is the script does not appear to be respecting the state file (state/siem_sophos.json) and continues to pull the last 12 hours of logs.

Any ideas on how we can remediate this issue?

I am setting up a syslog export for rapid 7 and following the guide here: https://docs.rapid7.com/insightidr/sophos-central/
I was able to get a connection done and got a result.txt that showed recent data back but I am not able to get the syslog connection working. Do I have the correct config file?

Edit: The script is running on a windows server with python 3

---snip--
[login]

# API Access URL + Headers
# API token setup steps: https://community.sophos.com/kb/en-us/125169
token_info = url: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# format can be json, cef or keyvalue
format = json
# filename can be syslog, stdout, any custom filename
filename = syslog
# endpoint can be event, alert or all
endpoint = event
# syslog properties
# for remote address use <remoteServerIp>:<port>, for e.g. 192.1.2.3:514
# for linux local systems use /dev/log
# for MAC OSX use /var/run/syslog
address = xx.xx.xx.xx:514
facility = daemon
socktype = udp

--snip--

Hi,

Lately, we have been noticing Sophos central events lost (not being pulled) by the v1 of the integration script.
I guess it has something to do with the state file not properly being read by the script.
We have deployed v2 of the script too and have seen similar reports. Is anyone else facing similar issues.

Cheers,
Gautham

Hi Team,

I am trying to use the script to get data from multiple Sophos tenant. Can this be accomodated?

Hi everyone,

Has anyone thought about adapting this script for the Sophos Partner API?
https://developer.sophos.com/docs/partner-account-enum/1/overview

Because we would definitely be interested :)
Or I'll build something myself.

Regards,

Linus

It doesn't appear that name field in DLP events is currently being parsed.

I'm having trouble running this Python script from within a batch file that is set to run through a Windows scheduled task. If I run the script from the command line [python siem.py] it runs without issue. I am trying to set this up to run once every hour via Windows Task Scheduler. I created a batch file with the following syntax:

"C:\Users\redacted\AppData\Local\Programs\Python\Python37-32\python.exe" "c:\SophosSIEM\siem.py"

Here is the output that I receive when I run the batch file:

C:\Scripts>"C:\Users\redacted\AppData\Local\Programs\Python\Python37-32\python.ex
e" "c:\SophosSIEM\siem.py"
Traceback (most recent call last):
File "c:\SophosSIEM\siem.py", line 494, in
main()
File "c:\SophosSIEM\siem.py", line 173, in main
token = config.Token(cfg.token_info)
File "c:\SophosSIEM\config.py", line 34, in getattr
return self.config.get('login', name)
File "C:\Users\redacted\AppData\Local\Programs\Python\Python37-32\lib\configpar
ser.py", line 780, in get
d = self._unify_values(section, vars)
File "C:\Users\redacted\AppData\Local\Programs\Python\Python37-32\lib\configpar
ser.py", line 1146, in _unify_values
raise NoSectionError(section) from None
configparser.NoSectionError: No section: 'login'

I saw another issue someone reported that looked similar to this where there was a problem with the "login" section. Again, if I run the siem.py script manually, it works without issue. It is only when I try to run it in a batch file that my problems arise. Has anyone gotten this script to run successfully in a Windows environment using Task Scheduler? I'm not married to Batch scripting if someone has another scripting recommendation.

Hello,

Whenever I run the script and pull log files from the sophos central API the date and time of the log files are 4 hours ahead!

For example

(sophos central cloud)
Event register new pc - recorded at 13:29pm on sophos central website
Event update succeeded - recorded at 14:01pm on sophos central website

(alien vault SIEM)
Event register new pc - recorded at 17:29pm in the log pulled down to alienvault
Event update succeeded - recorded at 18:01pm in the log pulled down to alienvault

Any ideas why the log file saves to alienvault 4 hours ahead?

thank you

SyslogHandler Module in python has introduced new variable append_nul which controls whether or not null byte should be appended to syslog message.
Since, not all siem vendor support null byte at end of syslog ( especially those that conform to RFC5424), so, an option to control this flag in config would be good feature.

For now, I have made mine work by directly adding
logging_handler.append_nul = False
after creating SyslogHandler in api_client.py.

See Python12168.

Left a comment on another issue, but not sure if it is the same.

When the GET command "/migration-tool/v1/endpoints" is used , if the number of endpoint is >100, the JSON is only 100 objects.
Also, the JSON is corrupted, even though the1st element of the JSON provides the correct value for the total number of endpoints.

Does the command use pagination – if so, can you explain how it is implemented ?
There does not seem to be any indication of such in the swagger documentation.

Hello, is it possible to include also audit information? For example the login (succeded or failed) of users on Sophos Central?

When I run this script it continually pulls events from the last 12 hours which creates thousands of duplicate events in Splunk. It appears that it's not checking the state file since it shows "No datetime found, defaulting to last 12 hours for results" every time I run it.

I'm running Python 3.8, but I've tried Python 3.6 with the same results.

I'm using the instructions provided here to get this data into Splunk: https://splunkbase.splunk.com/app/4647/#/details

Would be advantageous if we could run this script across multiple clients that exist in our Sophos Central Partner Portal and tag the entry with a particular customer / customer ID.

the 2.1.0 release of siem.py still reports as version 2.0.1 as the version string wasn't updated in the script.

Hi there,
I am trying to use this tool to convert my Sophos Central logs into the Common Event Format, but am having some issues when importing them into another tool.

The problem I am having is that the dhost in my Sophos Central logs is not a properly formatted hostname. This appears to only be an issue when dealing with macOS machines. Here area few examples:

"dhost": "Philippe???s MacBook Pro"
"dhost": "James???s MacBook Pro"
"dhost": "Adam???s MacBook Air"
"dhost": "Phil???s MacBook"

This is causing problems downstream when I am trying to import the CEF into other tools as the hostnames are not valid.
From most of the documentation from other vendors:

The format should be a fully qualified domain name associated with the device node, when a node is available. Examples: “host.domain.com” or “host”.

I understand this may not be an issue with this tool, but I am unsure where to ask this for Sophos Central. Can anyone help me out?

For filtering and reporting it would make life a lot easier if "TYPE" was broken out into "TYPE" and "SUBTYPE".

Currently:
{"customer_id": "1111111111111111111111111", "severity": "low", "endpoint_id": "111111111111111111", "endpoint_type": "computer", "source_info": {"ip": "10.80.111.111"}, "type": "Event::Endpoint::WebControlViolation", "id": "111111111111111111", "group": "WEB", "name": "User bypassed category block to 'https://config.edge.skype.com'", "datastream": "event", "duid": "1111111111", "rt": "2020-07-09T19:05:59.696Z", "end": "2020-07-09T19:00:57.000Z", "suser": "Joe User", "dhost": "COMPUTER NAME"}

Suggestion:
{"customer_id": "1111111111111111111111111", "severity": "low", "endpoint_id": "111111111111111111", "endpoint_type": "computer", "source_info": {"ip": "10.80.111.111"}, "type": "Event", "class": "Endpoint", "subtype": "WebControlViolation", "id": "111111111111111111", "group": "WEB", "name": "User bypassed category block to 'https://config.edge.skype.com'", "datastream": "event", "duid": "1111111111", "rt": "2020-07-09T19:05:59.696Z", "end": "2020-07-09T19:00:57.000Z", "suser": "Joe User", "dhost": "COMPUTER NAME"}

I transfered the files from github to our Qradar Collector server, I confgured config.ini with required parameters and attempted to use it with Syslog.
I verifyed that the server is listening on port 514 and started Tcpdump to watch if there is any traffic generated,
Once i run siem.py it pulled the logs from Sophos Central and saved it to result.txt in the /log directory but I did not managed to view the traffic comming thourgth port 514..

Config.ini File:

[login]
$ API Access URL + Headers
$ API token setup steps: https://community.sophos.com/kb/en-us/125169
token_info = url: https://api5.central.sophos.com/gateway, x-api-key: AAAAAAAAAAAAAAAAAAAAAAAA, Authorization: Basic MGE5YTY1Y2ItZWAAAAAAAAAAAAAAAAAAAAAAAAAAA0N1BJVzVTRkZIWAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAARvVGZhMU1zWVFxUEU=

$ format can be json, cef or keyvalue
format = json

$ filename can be syslog, stdout, any custom filename
filename = syslog

$ endpoint can be event, alert or all
endpoint = event

$ syslog properties
$ for remote address use :, for e.g. 192.1.2.3:514
$ for linux local systems use /dev/log
$ for MAC OSX use /var/run/syslog
address = x.x.x.x:514
facility = daemon
socktype = udp

** Additional Issue i faced with is when attempting to create a cron job it seems to run but no new logs are saved to logs/result.txt file for some reason.

Please help.

$ = #