mfesiem / msiempy Goto Github PK

View Code? Open in Web Editor NEW

23.0 4.0 7.0 9.51 MB

McAfee SIEM API Python wrapper

Home Page: https://mfesiem.github.io/docs/msiempy/msiempy.html

License: MIT License

Python 92.94% Shell 3.62% HTML 3.44%

mcafee siem api pyhton esm nsm msiem library datasource api-wrapper

msiempy's Introduction

McAfee SIEM API Python wrapper

This module aims to provide a simple API wrapper around the McAfee SIEM API principal components. Code design is accessible and pythonic via list-like and dict-like objects interfaces.

Main features

ESM operations: monitor, show statuses
DataSource operations, via DevTree: add, edit, delete - including client datasources, retreive from ID
Alarm operations and querying, via AlarmManager: filter, load pages, acknowledge, unacknowledge, delete, get triggering event, retreive from ID
Event operations and querying, via EventManager and GroupedEventManager: group queries, filter, add fields, set event's note, retreive from ID
Watchlist operations, via WatchlistManager: list, add/remove watchlists, add/remove values, get values, retreive from ID
Make direct API calls, via NitroSession

(Links are directing to the latest documentation version)

Known module implementations

msiem CLI : CLI tools for ESM
esm_healthmon : Monitors ESM operations
track-host: Rapidly request event logs to track a workstation
See samples folder or the tests. for other implementation examples and scripts !

Installation

pip install -U msiempy

Documentation

Read the latest documentation.

Or navigate the Index.

Authentication and configuration setup

The module offers a single point of authentication against your SIEM, so you don't have to worry about authentication when writting your scripts. This means that you need to preconfigure the authentication using the configuration file.

The configuration file is located (by default) securely in your user directory since it contains credentials.

For Windows: %APPDATA%\.msiem\conf.ini
For Mac : $HOME/.msiem/conf.ini
For Linux : $XDG_CONFIG_HOME/.msiem/conf.ini or $HOME/.msiem/conf.ini

Exemple:

[esm]
# Your ESM credentials
host = HOST
user = USER
passwd = PASSWORD's BASE64
[general]
# Verbosity
verbose = no
quiet = no
# Path to a logfile, the logfile output will always be verbose
logfile = 
# Misc 
timeout = 60
ssl_verify = no

To set the password, you can use the msiempy_setup.py script. You can also directly paste the password's base64 in the config file by doing:

>>> import base64 
>>> passwd = 'P@assW0rd'
>>> print(base64.b64encode(passwd.encode('utf-8')).decode()) 
UEBhc3NXMHJk

Changelog

Please refer to the releases github page.

Contribute

Pull requests are welcome!

Please read the contributing file.

Disclaimer

This is an UNOFFICIAL project and is NOT sponsored or supported by McAfee, Inc. If you accidentally delete all of your datasources, don't call support (or us). Product access will always be in respect to McAfee's intellectual property.

msiempy's People

Contributors

Stargazers

Watchers

Forkers

siniysv deoxykev andywalden uldeepak benhe119 nitish-awasthi saleny

msiempy's Issues

Datasource delete() raise Error but the Datasource is getting deleted anyway

Describe
Datasource fails to delete.

SIEM and msiempy versions:

msiempy: 0.3.3
ESM version: 11.3.0

Additional context

./dstools.py --remove 144116290775154688                   
DEBUG - Calling nitro request : get_devtree kwargs={}
DEBUG - Calling nitro request : login kwargs=***
DEBUG - Requesting HTTP post login ***
DEBUG - Returning raw requests Response object : <Response [201]>
DEBUG - Unpacking SIEM response: {
  "privileges" : {
    "master" : true,
    "admin" : false,
    "power" : false,
    "audit" : false,
    "crypto" : false,
    "systemSettings" : {
      "read" : true,
      "write" : true
    },
DEBUG - Calling nitro request : build_stamp kwargs={}
DEBUG - Requesting HTTP post essmgtGetBuildStamp with data None
DEBUG - Unpacking SIEM response:  { "return":
{
  "buildStamp" : "11.3.0 20191109004423"
} }
DEBUG - <Response [200]> -> Result (<class 'dict'>): {'buildStamp': '11.3.0 20191109004423'}
INFO - Logged into ESM URL with username NGCP. Last login 09/25/2020 17:11:31
DEBUG - Requesting HTTP post GRP_GETVIRTUALGROUPIPSLISTDATA with data {'ITEMS': '#{DC1 + DC2}', 'DID': '1', 'HD': 'F', 'NS': '0'}
DEBUG - Private API call : GRP_GETVIRTUALGROUPIPSLISTDATA Formatted params : Request=API%13GRP_GETVIRTUALGROUPIPSLISTDATA%13%14ITEMS%13#{DC1 + DC2}%13%14DID%131%13%14HD%13F%13%14NS%130%13%14
DEBUG - Unpacking SIEM response: Response=EC%130%13%14AC%130%13%14AF%13F%13%14FC%130%13%14FF%13F%13%14LC%130%13%14LF%13F%13%14DF%130%13%14DNAME%13Physical%20Display%13%14NS%131%13%14ITEMS%1314%11Local%20ESM%11144115188075855872%110%1
DEBUG - <Response [200]> -> Result (<class 'dict'>): {'EC': '0', 'AC': '0', 'AF': 'F', 'FC': '0', 'FF': 'F', 'LC': '0', 'LF': 'F', 'DF': '0', 'DNAME': 'Physical Display', 'NS': '1', 'ITEMS': '14%11Local%20ESM%11144115188075855872%110%11T%11T%11T%11T%11T
DEBUG - Calling nitro request : get_zones_devtree kwargs={}
DEBUG - Requesting HTTP post GRP_GETVIRTUALGROUPIPSLISTDATA with data {'ITEMS': '#{DC1 + DC2}', 'DID': '3', 'HD': 'F', 'NS': '0'}
DEBUG - Private API call : GRP_GETVIRTUALGROUPIPSLISTDATA Formatted params : Request=API%13GRP_GETVIRTUALGROUPIPSLISTDATA%13%14ITEMS%13#{DC1 + DC2}%13%14DID%133%13%14HD%13F%13%14NS%130%13%14
DEBUG - Unpacking SIEM response: Response=EC%130%13%14AC%130%13%14AF%13F%13%14FC%130%13%14FF%13F%13%14LC%130%13%14LF%13F%13%14DF%130%13%14DNAME%13Zone%20Display%13%14NS%131%13%14ITEMS%131%11Undefined%114294967295%11T%1117%111%11%123%
DEBUG - <Response [200]> -> Result (<class 'dict'>): {'EC': '0', 'AC': '0', 'AF': 'F', 'FC': '0', 'FF': 'F', 'LC': '0', 'LF': 'F', 'DF': '0', 'DNAME': 'Zone Display', 'NS': '1', 'ITEMS': '1%11Undefined%114294967295%11T%1117%111%11%123%11CS%20CEF%1114411
DEBUG - Calling nitro request : zonetree kwargs={}
DEBUG - Requesting HTTP post zoneGetZoneTree with data None
DEBUG - Unpacking SIEM response:  { "return":
[ ] }
DEBUG - <Response [200]> -> Result (<class 'list'>): []
DEBUG - Calling nitro request : ds_last_times kwargs={}
DEBUG - Requesting HTTP post QRY_GETDEVICELASTALERTTIME with data {}
DEBUG - Private API call : QRY_GETDEVICELASTALERTTIME Formatted params : Request=API%13QRY_GETDEVICELASTALERTTIME%13%14
DEBUG - Unpacking SIEM response: Response=EC%130%13%14NS%1316%13%14ITEMS%13CS%20CEF%11144116288594116608%11Common%20Event%20Format%1103%2F11%2F2020%2011%3A54%3A02%11%12DC01_DNS%11144116290775154688%11Linux%11%11%12Local%20Receiver-EL
DEBUG - <Response [200]> -> Result (<class 'dict'>): {'EC': '0', 'NS': '16', 'ITEMS': 'CS%20CEF%11144116288594116608%11Common%20Event%20Format%1103%2F11%2F2020%2011%3A54%3A02%11%12DC01_DNS%11144116290775154688%11Linux%11%11%12Local%20Receiver-ELM%111441
Delete the datasource and all the data? 
12, 3, DC01_DNS, 144116290775154688, True, 10.10.1.34, , 65, , Linux, , , , , 0, , 0, False, Local Receiver-ELM, 144116287587483648, never
[y/n]y
DEBUG - Calling nitro request : del_ds2 kwargs={'parent_id': '144116287587483648', 'ds_id': '144116290775154688'}
DEBUG - Requesting HTTP post dsDeleteDataSources with data {'receiverId': {'value': '144116287587483648'}, 'datasourceIds': [{'value': '144116290775154688'}]}
WARNING - An HTTP error occured (400 Client Error: 400 for url: https://URL/rs/esm/dsDeleteDataSources ErrMsg=NotOk Argument "linux" isn't numeric in numeric ne (!=) at /usr/local/bin/SetThirdPartyConfig line 796.

), retrying api_request()
DEBUG - Requesting HTTP post dsDeleteDataSources with data {'receiverId': {'value': '144116287587483648'}, 'datasourceIds': [{'value': '144116290775154688'}]}
ERROR - Error with method (dsDeleteDataSources) and data : {'receiverId': {'value': '144116287587483648'}, 'datasourceIds': [{'value': '144116290775154688'}]}. From requests.HTTPError 400 Client Error: 400 for url: https://URL/rs/esm/dsDeleteDataSources ErrMsg=NotOk Argument "linux" isn't numeric in numeric ne (!=) at /usr/local/bin/SetThirdPartyConfig line 796.


Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/core/session.py", line 744, in api_request
  File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: 400 for url: https://URL/rs/esm/dsDeleteDataSources

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/core/session.py", line 744, in api_request
  File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: 400 for url: https://URL/rs/esm/dsDeleteDataSources

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./dstools.py", line 451, in <module>
    main()
  File "./dstools.py", line 429, in main
    ds.delete()
  File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/device.py", line 1160, in delete
  File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/core/session.py", line 957, in request
  File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/core/session.py", line 781, in api_request
  File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/core/session.py", line 792, in api_request
msiempy.core.session.NitroError: Error with method (dsDeleteDataSources) and data : {'receiverId': {'value': '144116287587483648'}, 'datasourceIds': [{'value': '144116290775154688'}]}. From requests.HTTPError 400 Client Error: 400 for url: https://URL/rs/esm/dsDeleteDataSources ErrMsg=NotOk Argument "linux" isn't numeric in numeric ne (!=) at /usr/local/bin/SetThirdPartyConfig line 796.


tristan@Tristans-MBP dstools % ./dstools.py --list     
|                                   name                                  |    ds_ip     |       ds_id        |     parent_id      | client | type_id |      last_time      |
|                                Local ESM                                |  55.69.24.6  | 144115188075855872 |         0          | False  |   306   |         None        |
|                           ePolicy Orchestrator                          | 55.69.22.235 | 144117387099111424 | 144115188075855872 | False  |    0    |        never        |
| ePolicy Orchestrator_Endpoint Security Adaptive Threat Protection (ePO) | 55.69.22.235 | 144117387216551936 | 144117387099111424 | False  |   583   |        never        |
|          ePolicy Orchestrator_Endpoint Security Firewall (ePO)          | 55.69.22.235 | 144117387149443072 | 144117387099111424 | False  |   575   |        never        |
|          ePolicy Orchestrator_Endpoint Security Platform (ePO)          | 55.69.22.235 | 144117387115888640 | 144117387099111424 | False  |   574   |        never        |
|      ePolicy Orchestrator_Endpoint Security Threat Prevention (ePO)     | 55.69.22.235 | 144117387182997504 | 144117387099111424 | False  |   576   |        never        |
|         ePolicy Orchestrator_Endpoint Security Web Control (ePO)        | 55.69.22.235 | 144117387233329152 | 144117387099111424 | False  |   577   |        never        |
|                 ePolicy Orchestrator_ePO Audit Log (ePO)                | 55.69.22.235 | 144117387166220288 | 144117387099111424 | False  |   466   | 01/14/2020 01:49:09 |
|          ePolicy Orchestrator_ePolicy Orchestrator Agent (ePO)          | 55.69.22.235 | 144117387132665856 | 144117387099111424 | False  |   360   |        never        |
|                         ePolicy Orchestrator_TIE                        | 55.69.22.235 | 144117387199774720 | 144117387099111424 | False  |   524   |        never        |
|                            Local Receiver-ELM                           |  127.0.0.1   | 144116287587483648 | 144115188075855872 | False  |    0    | 03/11/2020 11:29:23 |
|                                  CS CEF                                 |  10.0.99.2   | 144116288594116608 | 144116287587483648 | False  |   143   | 03/11/2020 11:54:02 |
|                             Test Datasource                             |  55.69.24.4  | 144116287604260864 | 144116287587483648 | False  |    65   | 09/25/2020 17:11:03 |
|                            Test DataSource 11                           |  10.10.1.41  | 144116290859040768 | 144116287587483648 | False  |    65   |        never        |
|                            Test DataSource 12                           |  10.10.1.42  | 144116290875817984 | 144116287587483648 | False  |    65   |        never        |
|                            Test DataSource 13                           |  10.10.1.43  | 144116290825486336 | 144116287587483648 | False  |    65   |        never        |

Typo

msiempy/msiempy/__init__.py

Line 469 in 735863c

"get_flows_now" : ("""IPS_GETALERTSNOW""", """{"IPSID": "%(ds_id)s"}"""),

Shouldn't be IPS_GETFLOWSSNOW ? @andywalden @rhart211

DataSources fails to instantiated from ID

Describe
When trying to create a datasource object from id, the KeyError pops up.

Workaround: use DevTree:

devtree = DevTree()
ds = list(devtree.search_ds_group(field='ds_id', term='144116290808709120'))
if len(ds): ds=ds[0]
else: print("Datasource not found")

SIEM and msiempy versions:

msiempy: 0.3.3
ESM version: 11.3.0

Additional context

>>> from msiempy import DataSource
>>> d=DataSource(id=str(144116290808709120))
INFO - Logged into ESM url with username NGCP. Last login 09/25/2020 03:12:30
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/core/types.py", line 101, in __init__
    """Call the __prepare__ method of the appropriate metaclass.
  File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/device.py", line 1134, in data_from_id
  File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/device.py", line 1198, in _map_parameters
KeyError: 'desc_id'

Edit Datasources

Describe
DataSource objects should allow value modification with ease.

Maybe with a DataSource.edit(field, value)

Additional context
The readme says the library can edit datasources.. but it's untrue.

Test ESM is broken

Describe
Test ESM is broken

Rare "Invalid username or password for the ESM" errors

Describe
Random "Invalid username or password for the ESM" errors.

SIEM and msiempy versions:
msiempy verison: 0.2.2
ESM version: 11.2.1

Additional context

Traceback (most recent call last):
  File "/home/script-server/isrm-scripts/McAfeeSIEM/Alarms/ack_irrelevants.py", line 59, in <module>
    alarms.load_data(pages=args.pages, use_query=True, extra_fields= ['Description', 'Device_URL', 'Alert.DstIP', 'Rule.msg'])
  File "/usr/local/lib/python3.6/site-packages/msiempy/alarm.py", line 182, in load_data
    items, completed = self.qry_load_data(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/msiempy/alarm.py", line 242, in qry_load_data
    page_number=page_number
  File "/usr/local/lib/python3.6/site-packages/msiempy/__init__.py", line 919, in request
    self.login()
  File "/usr/local/lib/python3.6/site-packages/msiempy/__init__.py", line 650, in login
    raise NitroError('Invalid username or password for the ESM')
msiempy.NitroError: Invalid username or password for the ESM

Watchlist attributes stored as object properties

Hi Andy,

Just wondering why you declared all those Watchlist attributes as properties and not simply as key/value of the underlying dict object ? Is there a special reason for this ?

You can alway access underlying dict obejct of any NitroDict class by calling the data property. example : self.data['name']='the watchlist name'

Thanks

msiempy/msiempy/watchlist.py

Line 123 in 372ed4e

def __init__(self, *args, **kwargs):

Periodic ESM Maintenance Tasks with API

Describe
The library should cover simple maintenance tasks like

Removing the old triggered alarms:
OK: This can be done with AlarmManager
Removing older reports
I don't think so.
Creating a ESM Settings backup
I don't think so.
Creating a Full Data backup.
Andy's code should be imported to the ESM object to create a start_full_backup() method maybe.
Removing the older Settings back up files:
I don't think so.
Removing the older full back up files locally stored in the ESM
I don't think so.
Performing a dummy write and policy rollout
Not yet, See #96
Running Cron jobs to capture hardware faults, device health status, available disk space in the partitions like checking the index_hd and data_hd available space, swap memory, load average CPU utilization, RAM utilization etc.
Ok: This can be done with ESM object
Performing manual rules update if automatic rules update is not enabled.
I don't think so.

SIEM and msiempy versions:

msiempy: 0.3.5

Additional context
Ticket open after review of this thread: https://community.mcafee.com/t5/Security-Information-and-Event/Periodic-ESM-Maintenance-Tasks/m-p/672378/highlight/false#

Call miscKeepAlive every 5mn

Describe
NitroSession object should call the "miscKeepAlive" API call every 5mn to minimise the need of (automatically) re-login all the time

msiempy version:

msiempy: 0.3.5

Additional context
This issue was open after the review of the following thread: https://community.mcafee.com/t5/Security-Information-and-Event/ESM-Rest-API-client-connection-session-best-practicies/m-p/671599

ESM Login Password not base64 encoded

Describe
The password is not base64 encoded in msiempy\core\session.py

Details:
Password is not base64 encoded in msiempy\core\session.py. Because of this, I was not able to login
to ESM. I don't know if it is supposed to be base64 encoded always, or if there are version that accept it not base-64 encoded.

    def login(self, retry=1):
    """
    Authentication is done lazily upon the first call to `msiempy.core.session.NitroSession.request` method, but you can still do 
    it manually by calling this method.
    
    Raises:
        `msiempy.core.session.NitroError` if login fails
    """
    userb64 = tob64(self.config.user)
    passb64 = self.config.passwd

Implement grouped queries

Describe
We should be able to handle grouped queries as offed by the API

Additional context
We could access a new object: GroupedEventManager(FilteredQueryList)

GroupedEventManager(field, filters)

Reminder of the API documentation

qryExecuteGrouped
Description
Execute a grouped query giving the count and sum

Parameters
queryType
Type: EsmGroupedQueryType
Description: Query type
Accepted Values:
EVENT
FLOW
config
Type: EsmGroupedQueryConfig
Description: The parameters to apply to the query including filters, time range, field, etc
Return Value ("return" JSON root element IS returned)
Type: EsmRunningQuery
Description: The active query information, created as a result of executing the query successfully.
Example REST Call (with JSON if applicable)
https://URL:4443/rs/esm/qryExecuteGrouped?queryType=EVENT

Example JSON Content:

{"config": {
    "filters": [{
        "type": "EsmFieldFilter",
        "field": {"name": "(name)"},
        "operator": "IN",
        "values": [{
            "type": "EsmWatchlistValue",
            "watchlist": {"value": 0}
        }]
    }],
    "field": {"name": "(name)"},
    "timeRange": "CUSTOM",
    "customStart": "2020-03-11T11:30:12.595Z",
    "customEnd": "2020-03-11T11:30:12.596Z"
}}

Typo

Typo causes "Interpolation failed probably because of the private API calls formatting... Unexpected behaviours can happend." Error.

Solution
"ds_last_times": ("QRY%5FGETDEVICELASTALERTTIME","""{}"""),
Should be:
"ds_last_times": ("QRY_GETDEVICELASTALERTTIME","""{}"""),

notifyGetTriggeredNotification being removed from external API

More of a heads up than a current bug but looking at the 10.4 release notes, the notifyGetTriggeredNotification external API endpoint is being "removed to improve security", among some 77 others.

Looking for another way to get alarm details out of the API surfaces; I had hoped to use qryExecuteDetail with the TRIGGERED_ALARMS_QUERY type supplied but the API spits Unsupported query type: TRIGGERED_ALARMS_QUERY back at me, so go figure.

msiempy/msiempy/params.py

Line 201 in 85d6ed8

 'get_alarm_details': ("""notifyGetTriggeredNotification""", """{"id":%(id)s}"""), 

Add datasource method should ensure it's well added

Describe
Like in the code snippet, the add() method should call dsAddDataSourcesStatus , make sure there is no error or raise an Exception if the Datasource were not added.

Only if special argument check_added=True to keep compatibility.

** msiempy versions:**

msiempy: 0.3.3

Additional context

                if ds.get('client', None):
                    print("Adding Client Datasource: {}".format(ds))
                    resp = devtree.add_client(ds)
                else: 
                    print("Adding Datasource: {}".format(ds))
                    resp = devtree.add(ds)
                
                if not resp:
                    print('Something went wrong, Datasource {} not added.')
                    continue
                else:
                    # Wait for the add DS query to execuite ...
                    time.sleep(1)
                    ds_status = NitroSession().api_request('dsAddDataSourcesStatus', {"jobId": resp}, retry=0)
                    if not isinstance(ds_status, dict):
                        print('Something went wrong, Datasource {} not added.\n{}'.format(ds['name'], ds_status))
                        continue
                    while not ds_status['jobStatus'] == 'COMPLETE':
                        time.sleep(1)
                        ds_status = NitroSession().api_request('dsAddDataSourcesStatus', {"jobId": resp}, retry=0)
                    if len(ds_status['unsuccessfulDatasources'])>0:
                        print('Something went wrong, Datasource {} not added. {}'.format(ds['name'], ds_status['unsuccessfulDatasources'][0]))
                        continue
                    else:
                        ds_to_verify.append(ds['name'])
                        devtree.refresh()

Documentation generation problems

I'm unable to generate the documentation using Sphinx, and the docs on readthedocs appears to be missing.

Expected result: issuing "make " from ./msiempy/docs/ should generate docs.

Actual result: multiple errors in autodoc:
WARNING: autodoc: failed to import module 'alarm' from module 'msiempy'; the following exception was raised: No module named 'msiempy'

Attempts to resolve the issue include modifying the second parameter in conf.py file's "sys.path.insert" function call to "..", "../msiempy/".

Define Requirements for Test ESM

In the interest of creating a more stable live test environment, what are the requirements for a test ESM? It should have alarms, watchlists, events, etc - going how far back? Ideally testing would be limited to shorter timeframes but this requires some amount of intent to ensure those conditions always exist on the ESM for the test. Can we define what needs to exist for tests not to fail?

AlarmManager._event_match() only consider the first triggering events of the alarm

Describe
When using AlarmManager, only the first event is compared against events filters.

msiempy/msiempy/alarm.py

Line 351 in ad5c754

value = str(alarm["events"][0][event_filter[0]])

Expected behaviour would consider the filter matching if any triggering event match all passed filters.

msiempy version:
msiempy : 0.3.3

Refactoring Datasource and DevTree code

Describe
The Datasource and DevTree codebase have to be rewritten to:

Fix #78 and #79
Unify the search() method across list-like objects of the library as described in the 0.3.4 release notes

SIEM and msiempy versions:

msiempy: 0.3.4

Additional context
Add any other context about the problem here.

SIEM Field nicknames are not being mapped properly

Describe
When trying to call the GroupEventManager API with SIEM nicknames such as with below code an error message throws saying "EVENT Field not supported". Checking the code for event.py on line 566, I see that "self.get_field_nickname(field)" method is being called, shouldn't it be rather different and be trying to get the original internal name for Mcafee instead of the nickname?

Code link:

msiempy/msiempy/event.py

Line 566 in e5a4f34

if field:

Full error message
Error with method (v2/qryExecuteGrouped?queryType=EVENT) and data : {'config': {'filters': [{'type': 'EsmFieldFilter', 'field': {'name': 'IPSID'}, 'operator': 'IN', 'values': [{'type': 'EsmBasicValue', 'value': '144125089401536512'}, {'type': 'EsmBasicValue', 'value': '144125084385148928'}, {'type': 'EsmBasicValue', 'value': '144125089418313728'}, {'type': 'EsmBasicValue', 'value': '144125089435090944'}]}], 'field': {'name': 'New_Value'}, 'timeRange': 'CURRENT_DAY'}}. From requests.HTTPError 400 Client Error: 400 for url: https://SIEM_IP/rs/esm/v2/qryExecuteGrouped?queryType=EVENT Field not supported: New_Value"

Code
query = GroupedEventManager(
time_range='CURRENT_DAY',
field="Alert.4259885",
filters=[
FieldFilter("IPSID", ["144125089401536512","144125084385148928","144125089418313728","144125089435090944"]),#'SrcIP', 'AlertID',
#FieldFilter("Alert.Action", ["11","12"]),
])
query.load_data()
results = list(reversed(sorted(query, key=lambda k: int(k['SUM(Alert.EventCount)']))))
top10=results[:10]
pprint.pprint(top10)

SIEM and msiempy versions:
SIEM and msiempy versions:

msiempy: 0.3.5
ESM version: 11.4.7

Better polymorphism : cast every sub-NitroList class's item dynamically

Currently all NitroList sub-classes have to cast their item in the __init__ method.

For exemple EventManager :

msiempy/msiempy/event.py

Line 126 in 9937ae3

 collections.UserList.__init__(self, [Event(adict=item) for item in self.data if isinstance(item, (dict, NitroDict))]) 

. AlarmManager :

msiempy/msiempy/alarm.py

Line 55 in 9937ae3

 collections.UserList.__init__(self, [Alarm(adict=item) for item in self.data if isinstance(item, (dict, NitroDict))]) 

NitroList class should handle dynamically the casting of all of its item :

msiempy/msiempy/__init__.py

Line 926 in 9937ae3

#TODO better polymorphism to cast every sub-NitroList class's item dynamcally !

Splitting query bug on windows : TypeError: unhashable type: 'Event'

This issue is from someone sending us emails, please directly create new issue in the future :)
Trying to filter data from a specific normalize ID.

here is the code:

import msiempy.event

events = msiempy.event.EventManager(
        time_range='CUSTOM',
 start_time='2019-10-16',
 end_time='2019-10-31',
        fields=['Name','EventCount','Category','UserIDSrc','IPSID'],
        filters=[('NormID', ['537919488']) ],
        limit=500,
 order=('DESCENDING','LastTime'),
        max_query_depth=2)
events.load_data(delta='2h', slots='10', workers=1)
print (events.json)

here is the output

c:\msiempy>c:\SIEM_Reporter\venv\Scripts\python.exe test.py
INFO - New ESM session instance is created with : 10.28.0.50
Traceback (most recent call last):
  File "test.py", line 12, in <module>
    events.load_data(delta='2h', slots='10', workers=1)
  File "c:\msiempy\msiempy\event.py", line 351, in load_data
    sub_query = copy.copy(self)
  File "C:\Users\eaguilar.DOMINIO\AppData\Local\Programs\Python\Python37\lib\copy.py", line 88, in copy
    return copier(x)
  File "C:\Users\eaguilar.DOMINIO\AppData\Local\Programs\Python\Python37\lib\collections\__init__.py", line 1131, in __copy__
    inst.__dict__["data"] = self.__dict__["data"][:]
  File "C:\Users\eaguilar.DOMINIO\AppData\Local\Programs\Python\Python37\lib\collections\__init__.py", line 1096, in __getitem__
    return self.__class__(self.data[i])
  File "c:\msiempy\msiempy\event.py", line 111, in __init__
    self.fields=list(set(self.fields+fields))
TypeError: unhashable type: 'Event'

Can you help me to fix the issue?

Event query sample in readme doesn't work.

First of all, thank you for your work on this project. There's not a lot of resources out there for this kind of stuff.

I'm running the example event query script on the README.md page, but I run into this msiempy.NitroError: Sorry the filters must be either a tuple(fiels, [values]) or a QueryFilter sub class. error.

Script being used

import msiempy.event
import msiempy.query

filter = msiempy.query.FieldFilter(name='DstIP',values=['10.1.13.0/24'],operator='IN')
customfilter = filter.config_dict

events = msiempy.event.EventManager(
        time_range='LAST_MINUTE',
        filters=[
        msiempy.query.FieldFilter('DstIP', ['8.8.0.0/8',]),
        msiempy.query.FieldFilter('HostID', ['mydomain.local'], operator='CONTAINS') ],
        limit=1,
        max_query_depth=1)
events.load_data(delta='2h', slots='4', workers=5)
print(events.get_text(fields=['Alert.SrcPort', 'Rule.NormID']))

Output

INFO - New ESM session instance is created with : [REDACTED]
Traceback (most recent call last):
  File "event.py", line 13, in <module>
    max_query_depth=1)
  File "/usr/local/lib/python3.6/dist-packages/msiempy/event.py", line 141, in __init__
    super(self.__class__, self.__class__).filters.__set__(self, filters)
  File "/usr/local/lib/python3.6/dist-packages/msiempy/__init__.py", line 1460, in filters
    self.add_filter(f)
  File "/usr/local/lib/python3.6/dist-packages/msiempy/event.py", line 195, in add_filter
    raise NitroError("Sorry the filters must be either a tuple(fiels, [values]) or a QueryFilter sub class.")
msiempy.NitroError: Sorry the filters must be either a tuple(fiels, [values]) or a QueryFilter sub class.

Removing the filters keys seems to work, however.

DataSource description documentation

Describe
The documentation regarding Datasources is not 100% clear regarding the field desc_id :

The field “desc_id” represent the device type, I understood it can be 13, 3 or '254' for a client but the docs is unclear on the meaning of this. Here is the mapping:

{"1": "Zone",
"2": "ERC",
 "3": "Datasource",
"4": "Database Event Monitor (DBM)",
"5": "DBM Database",
 "7": "Policy Auditor",
"10": "Application Data Monitor (ADM)",
 "12": "ELM",
"14": "Local ESM",
"15": "Advanced Correlation Engine (ACE)",
"16": "Asset datasource",
 "17": "Score-based Correlation",
"19": "McAfee ePolicy Orchestrator (ePO)",
"20": "EPO",
"21": "McAfee Network Security Manager (NSM)",
"22": "McAfee Network Security Platform (NSP)",
 "23": "NSP Port",
 "24": "McAfee Vulnerability Manager (MVM)",
"25": "Enterprise Log Search (ELS)",
"254": "Client group",
 "256": "Client"}

We should rehabilitate the method _insert_desc_names (which is commented) in DevTree to insert the datasource type as String in a new DataSource field : desc

DevTree search fails after adding a DataSource

Describe
After adding anew DataSource withDevTree.add(), if we call DevTree.search() if raises a KeyError "hostname".
We need to call DevTree.refresh() before being able to search the tree again.

The expected behaviour is that the add() method would verify if the Data source was indeed added, then refresh only the Datasource infos.

Better error handling

NitroSession should handle correctly SIEM errors :

Currently, calls to NitroSession.esm_request that raises requests.HTTPError errors can raise NitroError if the error is unhandled.
See docs : https://mfesiem.github.io/docs/msiempy/index.html#msiempy.NitroSession.esm_request

Unhandled 400 errors :
- ERROR_InvalidFilter (228)
- No such command
- ERROR_Internal (-1)
- ERROR_JEC_ResponseNotAvailable (3503)
- Cannot deserialize value of type com.mcafee.siem.api.data.query.EsmTimeRange
- Cannot deserialize instance of java.lang.String out of START_OBJECT token
- Both customStart and customEnd properties must be set when running a query with a custom time range.
- Cannot construct instance of com.mcafee.siem.api.data.alert.EsmAlertId
- Parameter "xxx" cannot be missing
- All other errors
Unhandled 500 errors :
- All
Connexion errors -> Raises !
- Read timed out
- Max retries exceeded

Can't perform actions on alarms

Hey there, I'm running into this bug where the library will crash if I try to perform an action on an alarm.

Any insight?

Load alarm data:

import msiempy.alarm
alarms = msiempy.alarm.AlarmManager()
alarms.load_data()

Test to see if we got valid data back

testalarm = alarms[0]
print(testalarm.json)

{
    "id": {
        "value": 134
    },
    "severity": 50,
    "summary": "Event rate exceeded 1500 by 3983",
    "alarmName": "EPS Rate Exceeded",
    "conditionType": 14,
    "assignee": "[REDACTED]",
    "triggeredDate": "10/31/2019 17:55:15",
    "acknowledgedDate": "",
    "acknowledgedUsername": "",
    "filters": null,
    "queryId": "0",
    "alretRateMin": "5",
    "alertRateCount": "7500",
    "percentAbove": "0",
    "percentBelow": "0",
    "offsetMinutes": "0",
    "maximumConditionTriggerFrequency": null,
    "useWatchlist": null,
    "matchField": null,
    "matchValue": null,
    "assigneeId": "14",
    "escalatedDate": null,
    "caseId": "135",
    "caseName": "EPS rate exceeded (5k)",
    "iocName": null,
    "iocId": "0",
    "description": "Triggers when the system receives more then 10000 events within one minute",
    "actions": "5|EPS rate exceeded (5k)|135|[REDACTED]| 1|| ",
    "events": [
        {}
    ]
}

Then try to perform an action on our selected alarm

testalarm.delete()

WARNING - An HTTP error occured (Cannot deserialize instance of `java.util.ArrayList` out of START_OBJECT token
 at [Source: (StringReader); line: 1, column: 1]), retrying request
ERROR -
400 Client Error: 400 for url: https://[REDACTED]/rs/esm/alarmDeleteTriggeredAlarm A JSONObject text must begin with '{' at character 1 of "{\"triggeredIds\": {\"alarmIdList\": [39]}}"
See debug log for more infos.

It seems like other actions don't work either

testalarm.acknowlege()

WARNING - An HTTP error occured (Cannot deserialize instance of `java.util.ArrayList` out of START_OBJECT token
 at [Source: (StringReader); line: 1, column: 1]), retrying request
ERROR -
400 Client Error: 400 for url: https://[REDACTED]/rs/esm/alarmAcknowledgeTriggeredAlarm A JSONObject text must begin with '{' at character 1 of "{\"triggeredIds\": {\"alarmIdList\": [134]}}"

Here is the debug log

2019-10-31 11:06:22,429 - DEBUG - Calling nitro request : delete_alarms kwargs={'ids': [135]}
2019-10-31 11:06:22,430 - DEBUG - Requesting HTTP post alarmDeleteTriggeredAlarm with data {'triggeredIds': {'alarmIdList': [135]}}
2019-10-31 11:06:23,448 - WARNING - An HTTP error occured (Cannot deserialize instance of `java.util.ArrayList` out of START_OBJECT token
 at [Source: (StringReader); line: 1, column: 1]), retrying request
2019-10-31 11:06:23,649 - DEBUG - Requesting HTTP post alarmDeleteTriggeredAlarm with data {"triggeredIds": {"alarmIdList": [135]}}
2019-10-31 11:06:24,673 - ERROR -
400 Client Error: 400 for url: https://[REDACTED]/rs/esm/alarmDeleteTriggeredAlarm A JSONObject text must begin with '{' at character 1 of "{\"triggeredIds\": {\"alarmIdList\": [135]}}"
See debug log for more infos.
2019-10-31 11:06:24,673 - DEBUG - 400 Client Error: 400 for url: https://10.60.7.51/rs/esm/alarmDeleteTriggeredAlarm A JSONObject text must begin with '{' at character 1 of "{\"triggeredIds\": {\"alarmIdList\": [135]}}" {'_content': b'A JSONObject text must begin with \'{\' at character 1 of "{\\"triggeredIds\\": {\\"alarmIdList\\": [135]}}"', '_content_consumed': True, '_next': None, 'status_code': 400, 'headers': {'Date': 'Thu, 31 Oct 2019 18:06:24 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'X-XSS-Protection': '1; mode=block', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=63072000; includeSubdomains; preload', 'Cache-Control': 'no-cache, no-store, must-revalidate', 'Pragma': 'no-cache', 'Expires': 'Thu, 01 Jan 1970 00:00:00 GMT', 'Content-Type': 'text/plain; charset=utf-8', 'Content-Length': '102', 'Content-Security-Policy': "default-src 'none'; script-src 'self' 'unsafe-inline'; object-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self'; form-action 'self'; media-src 'self'; font-src 'self'; connect-src 'self'; plugin-types application/pdf application/x-shockwave-flash; reflected-xss block ; frame-src 'self';frame-ancestors 'self'", 'Connection': 'close'}, 'raw': <urllib3.response.HTTPResponse object at 0x7f240bf801d0>, 'url': 'https://[REDACTED]/rs/esm/alarmDeleteTriggeredAlarm', 'encoding': 'utf-8', 'history': [], 'reason': '400', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(0, 1, 21153), 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x7f2410e202b0>}

Transition to ESM API V2

Describe
The whole library uses API V1 and the Private ESM API. Which is not a major issue, in any case the library will depend on private ESM methods.

Calls to ESM API should be edited inside PARAMS property ensuring the parameters stays the same (if they can)

See complete McAfee note about this: https://kc.mcafee.com/corporate/index?page=content&id=KB90289&locale=en_US

Also, not to forget, the session already handles "ESM API V1" changes across different SIEM versions with Session api_v property (can be 1 or 2) Not be confused with the ESM API v1 and v2 which are different. (It's confusing, should be changed). See #11 and #27

ERROR - 404 Client Error: 404 for url: ipsAddAlertNote No such command.

Is there an alternative to ipsAddAlertNote in the new v11.2.1 API ?

Write Data Sources and Roll Policy

Describe
The library should handle a easy way to Write Data Sources and Roll Policy.

May be create a DevTree.write_policy() method?
May be also create a DevTree.add(write_policy=True) argument that will call the method? (See #82 for another DevTree.add() improvement)

SIEM and msiempy versions:

msiempy: 0.3.5

Additional context
The APIs that need to be called are dsWriteThirdpartyConfig, plcyRollPolicy, and miscJobStatus

Rule_NDSNormSigID.msg breaks Queries in 11.3.2

Describe
When trying to get Rule_NDSNormSigID.msg, all results break, for example:

Result without asking for Rule_NDSNormSigID.msg field

Result asking for Rule_NDSNormSigID.msg

Results are mixed, Rule.msg gents wrong info, Rule_NDSNormSigID.msg contains Rule.msg, Rule_NDSNormSigID.msg shoud contain correct normalization name, however it puts that "unknown_0".

SIEM and msiempy versions:

msiempy: 0.3.5
ESM version: breaks in 11.3.2, works in 10.4

Additional context
Every other query works with every field except Rule_NDSNormSigID.msg

Code Used:

from msiempy import EventManager, FieldFilter
print('Simple event query sorted by AlertID')
events = EventManager(
time_range='PREVIOUS_DAY',
fields=['LastTime','Rule.msg','Rule_NDSNormSigID.msg'],
filters=[
FieldFilter('SrcIP', ['X.X.X.X'], operator='IN'),
],
order=(('ASCENDING', 'AlertID')),
limit=100) # Will only load 100 events (per query)
events.load_data()
print(events)
print(events.get_text(fields=['LastTime','Rule.msg','Rule_NDSNormSigID.msg']))

Rare ERROR_JobEngine_CannotConnectToJobEngine caused by custom query in Event.data_from_id(use_query=True)

Describe

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/msiempy/__init__.py", line 758, in esm_request
    result.raise_for_status()
  File "/home/script-server/.local/lib/python3.6/site-packages/requests-2.22.0-py3.6.egg/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: 400 for url: https://nitro.mtl.cbc.ca/rs/esm/qryExecuteDetail?type=EVENT&reverse=false

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/msiempy/__init__.py", line 758, in esm_request
    result.raise_for_status()
  File "/home/script-server/.local/lib/python3.6/site-packages/requests-2.22.0-py3.6.egg/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: 400 for url: https://nitro/rs/esm/qryExecuteDetail?type=EVENT&reverse=false

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/script-server/isrm-scripts/McAfeeSIEM/Alarms/ack_irrelevants.py", line 59, in <module>
    alarms.load_data(pages=args.pages, use_query=True, extra_fields= ['Description', 'Device_URL', 'Alert.DstIP', 'Rule.msg'])
  File "/usr/local/lib/python3.6/site-packages/msiempy/alarm.py", line 182, in load_data
    items, completed = self.qry_load_data(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/msiempy/alarm.py", line 267, in qry_load_data
    workers=workers)
  File "/usr/local/lib/python3.6/site-packages/msiempy/__init__.py", line 1394, in perform
    func, elements))
  File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 586, in result_iterator
    yield fs.pop().result()
  File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/site-packages/msiempy/alarm.py", line 442, in load_events
    the_first_event.data = Event().data_from_id(id=the_id, use_query=use_query, extra_fields=extra_fields)
  File "/usr/local/lib/python3.6/site-packages/msiempy/event.py", line 1133, in data_from_id
    e.load_data()
  File "/usr/local/lib/python3.6/site-packages/msiempy/event.py", line 241, in load_data
    items, completed = self.qry_load_data()
  File "/usr/local/lib/python3.6/site-packages/msiempy/event.py", line 180, in qry_load_data
    includeTotal=False
  File "/usr/local/lib/python3.6/site-packages/msiempy/__init__.py", line 931, in request
    return self.esm_request(method=method, data=data, **params)
  File "/usr/local/lib/python3.6/site-packages/msiempy/__init__.py", line 777, in esm_request
    return self.esm_request(method, data, http, callback, raw, secure, retry=retry-1)
  File "/usr/local/lib/python3.6/site-packages/msiempy/__init__.py", line 798, in esm_request
    raise error
msiempy.NitroError: Error with method (qryExecuteDetail?type=EVENT&reverse=false) and data : {'config': {'timeRange': 'CUSTOM', 'customStart': '2019-07-26T00:00:53.000Z', 'customEnd': '2020-07-26T00:00:53.000Z', 'fields': [{'name': 'LastTime'}, {'name': 'Rule.msg'}, {'name': 'DstIP'}, {'name': 'Description'}, {'name': 'IPSIDAlertID'}, {'name': 'Device_URL'}], 'filters': [{'type': 'EsmFieldFilter', 'field': {'name': 'IPSIDAlertID'}, 'operator': 'IN', 'values': [{'type': 'EsmBasicValue', 'value': '144126183242467328|1333439283'}]}], 'limit': 2, 'offset': 0, 'order': [{'field': {'name': 'LastTime'}, 'direction': 'DESCENDING'}]}}. From requests.HTTPError 400 Client Error: 400 for url: https://url/rs/esm/qryExecuteDetail?type=EVENT&reverse=false ERROR_JobEngine_CannotConnectToJobEngine (3014)

SIEM and msiempy versions:
msiempy verison: 0.2.3
ESM version: 11.2.1 20190725050014

Additional context
Need to add ERROR_JobEngine_CannotConnectToJobEngine to the qry_load_data() retry conditions.
Here :

msiempy/msiempy/event.py

Line 206 in 59b5da7

'JobEngine_GetQueryResults_QueryNotFound_Unrecoverable'])

mfesiem / msiempy Goto Github PK

msiempy's Introduction

McAfee SIEM API Python wrapper

Main features

Known module implementations

Installation

Documentation

Authentication and configuration setup

Changelog

Contribute

Disclaimer

msiempy's People

Contributors

Stargazers

Watchers

Forkers

msiempy's Issues

Script being used

Output

Load alarm data:

Test to see if we got valid data back

Then try to perform an action on our selected alarm

It seems like other actions don't work either

Here is the debug log

Recommend Projects

Recommend Topics

Recommend Org