alienvault-otx / otx-python-sdk Goto Github PK
View Code? Open in Web Editor NEWThe Python SDK for AlienVault OTX
License: Other
The Python SDK for AlienVault OTX
License: Other
Only few plugins are being parsed in the result that's being returned in
result = otx.get_indicator_details_full(hash_type, hash)
Fields like msdefender, exiftool and some others aren't being taken into consideration in the def file(otx, hash) function. This is leading to some hashes not being recognised as malicious even though they have been listed as malicious on OTX site
We are facing the following error when making a call to the getAll() method using the otxv2 sdk.
pulses = otx.getall() File "<user_dir>/env/lib/python3.6/site-packages/OTXv2.py", line 341, in getall return self.walkapi(self.create_url(SUBSCRIBED, **args), iter=iter, max_page=max_page, max_items=max_items) File "<user_dir>/env/lib/python3.6/site-packages/OTXv2.py", line 322, in walkapi return list(self.walkapi_iter(url, max_page=max_page, max_items=max_items, method=method, body=body)) File "<user_dir>/env/lib/python3.6/site-packages/OTXv2.py", line 303, in walkapi_iter data = self.get(next_page_url) File "<user_dir>/env/lib/python3.6/site-packages/OTXv2.py", line 148, in get return self.handle_response_errors(response).json() File "<user_dir>/env/lib/python3.6/site-packages/OTXv2.py", line 131, in handle_response_errors raise Exception("Unexpected http code: %r, response=%r", response.status_code, _response_json()) {{Exception: ('Unexpected http code: %r, response=%r', 504, {'internal_error': 'Unable to decode response json: Expecting value: line 1 column 1 (char 0)'} )}}
Version of OTXV2 we are using - 1.5.2
Please let me know if any other information is needed, thanks.
Hi:
Triying to use indicators and recieving error 504, I switch to basic testing with curl to debug the problem and still get 504
`curl "https://otx.alienvault.com/api/v1/indicators/export?modified_since=2022-02-25T12:35:00+00:00&limit=50&page=1" -H "X-OTX-API-KEY: XXX"
<title>504 Gateway Time-out</title>Not able to get results, don't know the proble, but it looks like something not working as expected.
Can anybody help here?
Hello Everyone,
I've noticed that the default sorting of the API responses are ascending.
This of course makes searching for recent Pulses a pain, I would have to pull huge amounts of data and sort locally, which is an unnecessary strain on the API
Would anyone be interested in adding a parameter to sort descending?
Regards,
Linus
API_V1_ROOT is "api/v1"
when it looks like it should be "/api/v1"
or requests to the API fail because otx.alienvault.comapi
is not a valid hostname for HTTP requests. I'm on Python 2.7.11 just fyi.
Simply prepending a backslash to the API_V1_ROOT variable fixed this for me when I reinstalled the latest master branch locally. Added the backslash to the line above, ran pip install . --upgrade
, and it worked as expected.
API call & Traceback:
>>> mirai_iocs = otx.get_pulse_indicators('58e39804b789b25b15083fc6', limit=219)
Traceback (most recent call last):
File "<input>", line 1, in <module>
mirai_iocs = otx.get_pulse_indicators('58e39804b789b25b15083fc6', limit=219)
File "/usr/local/lib/python2.7/site-packages/OTXv2.py", line 404, in get_pulse_indicators
return self.walkapi(self.create_url(PULSE_DETAILS + str(pulse_id) + "/indicators", limit=limit))
File "/usr/local/lib/python2.7/site-packages/OTXv2.py", line 274, in walkapi
return list(self.walkapi_iter(url, max_page=max_page))
File "/usr/local/lib/python2.7/site-packages/OTXv2.py", line 265, in walkapi_iter
data = self.get(next_page_url)
File "/usr/local/lib/python2.7/site-packages/OTXv2.py", line 112, in get
proxies=self.proxies,
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 501, in get
return self.request('GET', url, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 487, in send
raise ConnectionError(e, request=request)
ConnectionError: HTTPSConnectionPool(host='otx.alienvault.comapi', port=443): Max retries exceeded with url: /v1/pulses/58e39804b789b25b15083fc6/indicators?limit=219 (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x10c0d5110>: Failed to establish a new connection: [Errno 8] nodename nor servname provided
, or not known',))
As shown above the ConnectionError is failing due to HTTPSConnectionPool(host='otx.alienvault.comapi', port=443)
where the host variable is not a valid hostname.
After Patch:
>>> from OTXv2 import OTXv2
>>> otx = OTXv2(key)
>>> mirai_iocs = otx.get_pulse_indicators('58e39804b789b25b15083fc6', limit=100)
>>> len(mirai_iocs)
100
Hello guys ๐
I need to find when an indicator has been created.
Unfortunately, to achieve this, we have to
/api/v1/indicators/{type}/{value}/{section}
And the call to get all the indicators related to one pulse is freaking slow.
I suggest to return the created
, modified
and expiration
fields (all useful date fields) when fetching the indicator.
I am trying export all the event from MISP to .json and import into OTX Pulse. In the example folder only accept one .json file. Is there any method to bulk import all .json file into OTX?
MaxRetryError Traceback (most recent call last)
C:\ProgramData\Anaconda3\lib\site-packages\requests\adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
448 retries=self.max_retries,
--> 449 timeout=timeout
450 )
C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
730 release_conn=release_conn,
--> 731 body_pos=body_pos, **response_kw)
732
C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
730 release_conn=release_conn,
--> 731 body_pos=body_pos, **response_kw)
732
C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
730 release_conn=release_conn,
--> 731 body_pos=body_pos, **response_kw)
732
C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
730 release_conn=release_conn,
--> 731 body_pos=body_pos, **response_kw)
732
C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
730 release_conn=release_conn,
--> 731 body_pos=body_pos, **response_kw)
732
C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
710 try:
--> 711 retries = retries.increment(method, url, response=response, _pool=self)
712 except MaxRetryError:
C:\ProgramData\Anaconda3\lib\site-packages\urllib3\util\retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
398 if new_retry.is_exhausted():
--> 399 raise MaxRetryError(_pool, url, error or ResponseError(cause))
400
MaxRetryError: HTTPSConnectionPool(host='otx.alienvault.com', port=443): Max retries exceeded with url: /api/v1/pulses/subscribed?limit=20 (Caused by ResponseError('too many 504 error responses'))
During handling of the above exception, another exception occurred:
RetryError Traceback (most recent call last)
C:\ProgramData\Anaconda3\lib\site-packages\OTXv2.py in get(self, url, **kwargs)
175 headers=self.headers,
--> 176 proxies=self.proxies,
177 )
C:\ProgramData\Anaconda3\lib\site-packages\requests\sessions.py in get(self, url, **kwargs)
545 kwargs.setdefault('allow_redirects', True)
--> 546 return self.request('GET', url, **kwargs)
547
C:\ProgramData\Anaconda3\lib\site-packages\requests\sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
532 send_kwargs.update(settings)
--> 533 resp = self.send(prep, **send_kwargs)
534
C:\ProgramData\Anaconda3\lib\site-packages\requests\sessions.py in send(self, request, **kwargs)
645 # Send the request
--> 646 r = adapter.send(request, **kwargs)
647
C:\ProgramData\Anaconda3\lib\site-packages\requests\adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
506 if isinstance(e.reason, ResponseError):
--> 507 raise RetryError(e, request=request)
508
RetryError: HTTPSConnectionPool(host='otx.alienvault.com', port=443): Max retries exceeded with url: /api/v1/pulses/subscribed?limit=20 (Caused by ResponseError('too many 504 error responses'))
During handling of the above exception, another exception occurred:
RetryError Traceback (most recent call last)
in
----> 1 pulses = otx.getall()
C:\ProgramData\Anaconda3\lib\site-packages\OTXv2.py in getall(self, modified_since, author_name, limit, max_page, max_items, iter)
393 return self.walkapi(
394 self.create_url(SUBSCRIBED, **args), iter=iter,
--> 395 max_page=max_page, max_items=max_items
396 )
397
C:\ProgramData\Anaconda3\lib\site-packages\OTXv2.py in walkapi(self, url, iter, max_page, max_items, method, body)
372 return self.walkapi_iter(url, max_page=max_page, max_items=max_items, method=method, body=body)
373 else:
--> 374 return list(self.walkapi_iter(url, max_page=max_page, max_items=max_items, method=method, body=body))
375
376 def getall(self, modified_since=None, author_name=None, limit=20, max_page=None, max_items=None, iter=False):
C:\ProgramData\Anaconda3\lib\site-packages\OTXv2.py in walkapi_iter(self, url, max_page, max_items, method, body)
353
354 if method == 'GET':
--> 355 data = self.get(next_page_url)
356 elif method == 'POST':
357 data = self.post(next_page_url, body=body)
C:\ProgramData\Anaconda3\lib\site-packages\OTXv2.py in get(self, url, **kwargs)
178 return self.handle_response_errors(response).json()
179 except requests.exceptions.RetryError:
--> 180 raise RetryError()
181
182 def patch(self, url, body, **kwargs):
RetryError: 'Exceeded maximum number of retries'
When i am trying to fetch the pulses. It is throwing an "OSError: Tunnel connection failed: 407 Proxy Authentication Required" error. Is it because i am using office network. If yes, than how to resolve the issue.
I used the below code to get indicator details for this IP address (8.8.8.8)
result = otx_client.get_indicator_details_by_section(IndicatorTypes.IPv4, '8.8.8.8', 'general')
But it doesn't return the whitelisted validation as in the example
If I use the other API, it returns the whitelisted validation successfully
validation = otx_client.validate_indicator(IndicatorTypes.IPv4, '8.8.8.8')
Thoughts?
Hello,
I am using pthon2.7, when I use
pulses = otx.getall()
It gives below error,
Traceback (most recent call last):
File "otx.py", line 7, in
pulses = otx.getall()
File "/usr/local/lib/python2.7/dist-packages/OTXv2.py", line 236, in getall
json_data = self.get(next_page_url)
File "/usr/local/lib/python2.7/dist-packages/OTXv2.py", line 81, in get
raise e
urllib2.URLError: <urlopen error [Errno 8] _ssl.c:510: EOF occurred in violation of protocol>
Anyone is aware of this issue, I tried the same with python3.4 as well still getting same error
Running ubuntu 14.04
Thanks,
Npormambi
The endpoint GET /api/v1/pulses/{pulse_id}
does not have the ability to limit the returned fields, or limit the number of indicators returned.
Example of not supporting limit
to limit the indicators:
>>> pulse_id = '57eee565e65dd9042acfc830'
>>> pulse_url = otx.create_url(PULSE_DETAILS + str(pulse_id), limit=10)
>>> pulse_url
'https://otx.alienvault.com/api/v1/pulses/57eee565e65dd9042acfc830?limit=10&'
>>> pulse = otx.get(pulse_url)
>>> len(pulse['indicators'])
643
In the example, the pulse queried has 643 indicators, and even if I pass the parameter limit=10
, I still get all 643.
I'd like to be able to limit the size of the response either by limiting the number of indicators returned, or by requesting specific fields to be only included in the response, for example:
otx.get('https://otx.alienvault.com/api/v1/pulses/57eee565e65dd9042acfc830?fields=name,author_name')
The reason I want to do this is because I want to be able to only get details of a pulse (for example, its name and author) by its pulse_id. I tried doing this using a search, but pulse IDs are not a supported query:
>>> otx.search_pulses('57ef3a9be65dd9042acfd1b0')
{u'count': 0, u'groups_count': 0, u'exact_match': u'', 'results': [], u'users_count': 0}
Is there any other way I can get the details of a pulse ID without having to pull all of its indicators?
New feature request: please extend the API with a method to get the server's version. Also please consider invoking this method in the OTXv2 class init method and log the result at level debug.
In addition to yielding version details, this is a cheap way of testing if the API key is valid before trying to fetch data.
Thanks in advance for considering it.
When looping through IP's using the otx.get_indicator_details_full function, I get an error - AttributeError: 'NoneType' object has no attribute 'read'.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 375, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 632, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/src/app/app/tasks.py", line 32, in process_ips
document = IPHandler().process(ip)
File "/usr/src/app/app/engine/ip_handler.py", line 39, in process
OTX_response = self.get_OTX_data(ip)
File "/usr/src/app/app/engine/ip_handler.py", line 15, in get_OTX_data
otx_data = self.otx.get_indicator_details_full(IndicatorTypes.IPv4, ip)
File "/usr/local/lib/python3.6/site-packages/OTXv2.py", line 492, in get_indicator_details_full
indicator_dict[section] = self.get(indicator_url)
File "/usr/local/lib/python3.6/site-packages/OTXv2.py", line 83, in get
data = response.read().decode('utf-8')
AttributeError: 'NoneType' object has no attribute 'read'
Using Python Version - 3.6.4
If I try to run a search against /api/v1/search/pulses
with spaces in the query, I get a Bad Request:
>>> otx.search_pulses('phpMyAdmin honeypot logs for 2016-10-01')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "OTXv2.py", line 293, in search_pulses
return self._get_paginated_resource(search_pulses_url, max_results=max_results)
File "OTXv2.py", line 317, in _get_paginated_resource
json_data = self.get(next_page_url)
File "OTXv2.py", line 79, in get
raise BadRequest("Bad Request")
OTXv2.BadRequest: 'Bad Request'
This is because the query in the search_pulses
method is not URL encoded, so it tries to GET something like this: https://otx.alienvault.com/api/v1/search/pulses?q=phpMyAdmin honeypot logs for 2016-10-01&
.
To fix this we can use urllib.quote_plus
for Python 2 (or urllib.parse.quote_plus()
for Python 3) to properly encode our query.
>>> import urllib
>>> urllib.quote_plus('phpMyAdmin honeypot logs for 2016-10-01')
'phpMyAdmin+honeypot+logs+for+2016-10-01'
So a quick fix would be to update the q=query
in the search_pulses
method to q=urllib.quote_plus(q)
, like so:
def search_pulses(self, query, max_results=25):
search_pulses_url = self.create_url(SEARCH_PULSES, q=urllib.quote_plus(q), page=1, limit=20)
return self._get_paginated_resource(search_pulses_url, max_results=max_results)
I'm not sure if this is the norm in APIs, but wouldn't it make sense to update your python API module to do the URL encoding? If you're expecting the user to do it, then maybe mention that under the API module documentation?
The python notebook says to replace YOUR_KEY with your actual API key. It doesnt say anywhere where to specify the key.
As a feature request it would be beneficial to submit threat indictors via an API rather than only requesting.
Line 411, the documentation for the body parameter for edit_pulse(), should be changed to indicate it's actual the delta set, or set of adds and removes, rather than a "complete set".
Suspect the currently wording was not changed as a result of copying from what is now line 426, the new_indicators parameter documentation for replace_pulse_indicators.
>>> mtime = (datetime.now() - timedelta(days=1)).isoformat()
>>> mtime
'2015-12-09T14:33:11.722751'
>>> events = otx.getevents_since(mtime)
>>> json_normalize(events)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3/dist-packages/pandas/io/json.py", line 684, in json_normalize
if any([isinstance(x, dict) for x in compat.itervalues(data[0])]):
IndexError: list index out of range
This snippet of code from README.md fails in the google.com portion:
from OTXv2 import OTXv2
otx = OTXv2("API_KEY")
# Get all the indicators associated with a pulse
indicators = otx.get_pulse_indicators("pulse_id")
for indicator in indicators:
print indicator["indicator"] + indicator["type"]
# Get everything OTX knows about google.com
otx.get_indicator_details_full(IndicatorTypes.DOMAIN, "google.com")
Needs the following additional line:
from OTXv2 import IndicatorTypes
In trying to use some simple python like this:
from OTXv2 import OTXv2
try:
with open('config.json', 'r') as file:
config = json.load(file)
except:
quit('config.json not found, or unreadable.')
otx = OTXv2(config['otx_api_key'])
pulses = otx.get_my_pulses(max_items=200)
print(pulses)
The result would always be an empty list: []
No matter what arguments I passed to the get_my_pulses
function.
I started digging into the source OTXv2 library. And hacked together the equivalent necessary functions using the requests library
from requests import Session
from requests.packages.urllib3.util import Retry
from requests.adapters import HTTPAdapter
import json
try:
with open('config.json', 'r') as file:
config = json.load(file)
except:
quit('config.json not found, or unreadable.')
api = "https://otx.alienvault.com"
headers = {
'X-OTX-API-KEY': config['otx_api_key'],
'Content-Type': 'application/json'
}
session = Session()
session.mount('https://', HTTPAdapter(
max_retries=Retry(
total=5,
status_forcelist=[429, 500, 502, 503, 504],
backoff_factor=1,
)
))
response = session.get(api+'/api/v1/pulses/my', headers=headers).json()
print(response['count'])
Which yielded a more expected response 214
which is accurate.
Upon digging more into the OTXv2 library functions, I hacked this together to give me just the necessary bits so that I could place some print statements and find what URLs were being formed.
import requests
try:
from urllib.parse import urlencode
except ImportError:
from urllib import urlencode
import json
API_V1_ROOT = "/api/v1"
MY_PULSES = "{}/pulses/my".format(API_V1_ROOT)
server="https://otx.alienvault.com"
headers = {
'X-OTX-API-KEY': '<redacted>',
'Content-Type': 'application/json'
}
def post(url, body=None, headers=None, files=None, **kwargs):
"""
Internal API for POST request on a OTX URL
:param url: URL to retrieve
:param body: HTTP Body to send in request
:param headers: (optional) dict of headers to use, instead of default headers
:param files: (optional) list of file tuples, if posting multipart form data
:return: response as dict
"""
response = requests.session().post(
create_url(url, **kwargs),
data=json.dumps(body) if body else None,
files=files,
headers=headers
)
print(json.dumps(response.json(), indent=2))
return response.json()
def get(url, **kwargs):
"""
Internal API for GET request on a OTX URL
:param url: URL to retrieve
:return: response in JSON object form
"""
response = requests.session().get(create_url(url, **kwargs), headers=headers)
print(json.dumps(response.json(), indent=2))
return response.json()
def walkapi_iter(url, max_page=None, max_items=None, method='GET', body=None):
next_page_url = url
print('next_page_url', next_page_url)
count = 0
item_count = 0
while next_page_url:
count += 1
if max_page and count > max_page:
break
if method == 'GET':
data = get(next_page_url)
elif method == 'POST':
data = post(next_page_url, body=body)
else:
raise Exception("Unsupported method type: {}".format(method))
for el in data['results']:
item_count += 1
if max_items and item_count > max_items:
break
yield el
next_page_url = data["next"]
def create_url(url_path, **kwargs):
""" Turn a path into a valid fully formatted URL. Supports query parameter formatting as well.
:param url_path: Request path (i.e. "/search/pulses")
:param kwargs: key value pairs to be added as query parameters (i.e. limit=10, page=5)
:return: a formatted url (i.e. "/search/pulses")
"""
print('----start create url----')
uri = url_path.format(server)
print(uri)
uri = uri if uri.startswith("http") else server.rstrip('/') + uri
print(uri)
if kwargs:
uri += "?" + urlencode(kwargs)
print(uri)
print('----stop create url---')
return uri
def walkapi(url, iter=False, max_page=None, max_items=None, method='GET', body=None):
if iter:
print(walkapi_iter(url, max_page=max_page, max_items=max_items, method=method, body=body))
return walkapi_iter(url, max_page=max_page, max_items=max_items, method=method, body=body)
else:
print(list(walkapi_iter(url, max_page=max_page, max_items=max_items, method=method, body=body)))
return list(walkapi_iter(url, max_page=max_page, max_items=max_items, method=method, body=body))
def get_my_pulses(query=None, max_items=200):
print('pulse?', walkapi(create_url(MY_PULSES, limit=50, q=query), max_items=max_items))
#print(walkapi(create_url(MY_PULSES, limit=50)))
return walkapi(create_url(MY_PULSES, limit=50, q=query), max_items=max_items)
This yielded URLs consistently like this, uring the create_url() function:
----start create url----
https://otx.alienvault.com/api/v1/pulses/my?limit=50&q=None
https://otx.alienvault.com/api/v1/pulses/my?limit=50&q=None
https://otx.alienvault.com/api/v1/pulses/my?limit=50&q=None
----stop create url---
q=
is not a valid API query parameter according to the OTX API documentation: https://otx.alienvault.com/api
only limit
, page
, 'since`.
I'm haven't completely learned the OTXv2 code base, but I don't see anywhere were q=
is supposed to be formatted into something the API accepts out of the above list of valid queries.
If you provide any sort of data by passing it in with get_my_pulses(query=<anything>)
it always formats the URL as
https://otx.alienvault.com/api/v1/pulses/my?limit=50&q=<anything>
which the API responds to with 0 results.
Am I missing something?
Note: I saw similar issues with get_user_pulses()
but it appears to be virtually identical just with a different API endpoint URL.
When passing in anything other than URLs like hashes into otx.get_indicator_details_full for type IndicatorTypes.URL it hangs and there's no way to control timeout
Receiving the following error upon first run:
Traceback (most recent call last):
File "test_rules.py", line 41, in testRuleGenerate
self.suricata_client.generate_rules(True, True)
File "/root/OTX-Suricata/otx-suricata/suricata.py", line 37, in generate_rules
for pulse in self.client.getall_iter():
File "/usr/local/lib/python2.7/dist-packages/OTXv2.py", line 265, in walkapi_iter
data = self.get(next_page_url)
File "/usr/local/lib/python2.7/dist-packages/OTXv2.py", line 115, in get
except requests.exceptions.RetryError:
AttributeError: 'module' object has no attribute 'RetryError'
Ran 1 test in 5.509s
FAILED (errors=1)`
Can the OTX python pull private pulses, or must they be public? The API key is tied to my account which created the private pulses.
No license for this project has been defined, thus it is copyrighted by default. Therefore the code cannot be used by other project without violating copyright laws!
The API: https://otx.alienvault.com/api/v1/indicators/export
was earlier being used to fetch all the indicators. As per latest docs, the API isn't listed under https://otx.alienvault.com/assets/static/external_api.html#Home
Is the API removed? If so, what is the alternate API to use to fetch all indicators?
If this is not the correct repository to post in, please direct me to appropriate repository to create this issue in.
Hello All,
As per Dashboard there are more that 4000 Pulses, but when I try to fetch all the pulses with below code
pulses = otx.getall(), I am able to fetch around 250 pulses only.
Does anyone knows how to fetch all the pulse in one go.
Thanks and Regards.
If I do res = otx.get_indicator_details_full(IndicatorTypes.EMAIL, args.EMAIL)
I get the following error :
File "/home/user/env/analysis3/lib/python3.5/site-packages/OTXv2.py", line 490, in get_indicator_details_full
for section in indicator_type.sections:
TypeError: 'NoneType' object is not iterable
Which makes sense because IndicatorTypes.EMAIL is the only one not having sections at all in IndicatorTypes.py
while get_indicator_details_full
use this to request information.
I have tried to fix this by adding a section but I end up having other errors :
~/env/analysis3/lib/python3.5/site-packages/OTXv2.py in get(self, url)
81 else:
82 raise e
---> 83 data = response.read().decode('utf-8')
84 json_data = json.loads(data)
85 return json_data
AttributeError: 'NoneType' object has no attribute 'read'
I have this bug with python 3.5.2 and the pip package 1.2 (but I am pretty sure this bug is the master code)
Hello,
Could you provide a documentation for OTXv2.py
file ?
I know it is not a complex file and reading the source provide the information, but it would be more convenient to provide documentation and to be able to refer to, for instance a readthedocs.io
page.
Cheers!
For improved proxy support, could we add the verify and cert parameters to the requests query? I have a working copy and can do a pull request to add this functionality.
Hello,
I am currently creating a pool of processes that simultaneously hit the API with multiple IOC's to retrieve indicator details.
The API stops pulling after about 670 MB of IOC data.
I made sure my machine is hardwired to the internet and the network has not gone down during my runs.
I have also made sure my machine doesn't go to sleep or that anything is stopping the script from running.
I am handling failing IOC's using exceptions and they get logged with no issues.
After pulling details for about 3-4k IOC's the API hangs.
Any idea what could be causing this issue?
Thank you,
Baalgar
Currently it is necessary to copy the sdk file to software using this lib. It would be great if it would be a package, so we can just install it with pip and set it as requirement.
EDIT: I can help creating packages if needed, just did it with intelmq.
Code bellow returned empty for an AN_EMAIL_ADDRESS I have already found on
https://otx.alienvault.com/indicator/email/{{AN_EMAIL_ADDRESS}}
from OTXv2 import OTXv2
from OTXv2 import IndicatorTypes
API_KEY = "mykey_is_was_tested_an_was_ok"
AN_EMAIL_ADDRESS = "[email protected]"
otx = OTXv2(API_KEY)
found_entries = otx.get_indicator_details_full(IndicatorTypes.EMAIL, AN_EMAIL_ADDRESS)
print found_entries
Some more details:
$pip list
Package Version
--------------- ---------
certifi 2020.12.5
chardet 4.0.0
idna 2.10
OTXv2 1.5.12
pip 20.3.4
pkg-resources 0.0.0
python-dateutil 2.8.1
pytz 2021.1
requests 2.25.1
setuptools 44.1.1
six 1.15.0
urllib3 1.26.4
wheel 0.36.2
Code branch master on 2021-04-15:
$ git status
On branch master
Your branch is up to date with 'origin/master'.
[...]
$ git config -l | grep remote.origin.url
remote.origin.url=https://github.com/AlienVault-OTX/OTX-Python-SDK
If necessary, please contact me to get the AN_EMAIL_ADDRESS address I have used do identify the problem.
in your OTXv2.py module, within the get function, you have the ProxyHandler set to http:
if self.proxy:
proxy = ProxyHandler({'http': self.proxy})
If the user sets the proxy value, this will cause an error because the url being called out to is https:
server="https://otx.alienvault.com"
Updating the OTXv2.py module to 'https' rather than 'http' fixes the issue:
if self.proxy:
proxy = ProxyHandler({'https': self.proxy})
Configured per instructions but receiving the following in startup log.
`[1378] 27/6/2017 -- 13:10:58 - (conf-yaml-loader.c:241) (ConfYamlParse) -- Including configuration file /etc/suricata/otx-suricata.yaml.
[1378] 27/6/2017 -- 13:10:58 - (conf-yaml-loader.c:265) (ConfYamlParse) -- Configuration node 'rule-files' redefined.
[1378] 27/6/2017 -- 13:10:58 - (conf-yaml-loader.c:180) (ConfYamlParse) -- [ERRCODE: SC_ERR_CONF_YAML_ERROR(242)] - Failed to parse configuration file at line 9: did not find expected key
[1378] 27/6/2017 -- 13:10:58 - (conf-yaml-loader.c:148) (ConfYamlHandleInclude) -- [ERRCODE: SC_ERR_CONF_YAML_ERROR(242)] - Failed to include configuration file /etc/suricata/otx-suricata.yaml`
Hello there,
I would like to report a Python warning:
$ python3 -Wall is_malicious.py -file /etc/passwd
/home/test/is_malicious.py:63: ResourceWarning: unclosed file <_io.BufferedReader name='/etc/passwd'>
hash = hashlib.md5(open(args['file'], 'rb').read()).hexdigest()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Unknown or not identified as malicious
Cheers!
A question regarding OTXv2.session().
Please correct me if I'm wrong, but should 429 be included in status_forcelist? If the server says "too many requests", asking again in a second or half probably won't help.
self.request_session.mount('https://', HTTPAdapter(
max_retries=Retry(
total=5,
status_forcelist=[429, 500, 502, 503],
backoff_factor=1,
)
))
Wrong repo, ignore.
When requesting an non-existent pulse, with pip version 1.2:
indicators = otx.get_pulse_indicators("aaaaaaaaaa")
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-6-78f47ae1d67a> in <module>()
----> 1 indicators = otx.get_pulse_indicators("aaaaaaaaaa")
/home/user/envs/analysis3/lib/python3.5/site-packages/OTXv2.py in get_pulse_indicators(self, pulse_id, limit)
412 next_page_url = self.create_url(PULSE_DETAILS + str(pulse_id) + "/indicators", limit=limit)
413 while next_page_url:
--> 414 json_data = self.get(next_page_url)
415 for r in json_data["results"]:
416 indicators.append(r)
/home/user/envs/analysis3/lib/python3.5/site-packages/OTXv2.py in get(self, url)
81 else:
82 raise e
---> 83 data = response.read().decode('utf-8')
84 json_data = json.loads(data)
85 return json_data
AttributeError: 'NoneType' object has no attribute 'read'
With masterbranch (commit da9cd34 ):
indicators = otx.get_pulse_indicators("aaaaaaaaaa")
---------------------------------------------------------------------------
Exception Traceback (most recent call last)
<ipython-input-3-78f47ae1d67a> in <module>()
----> 1 indicators = otx.get_pulse_indicators("aaaaaaaaaa")
/home/user/envs/analysis3/lib/python3.5/site-packages/OTXv2.py in get_pulse_indicators(self, pulse_id, limit)
402 :return: Indicator list
403 """
--> 404 return self.walkapi(self.create_url(PULSE_DETAILS + str(pulse_id) + "/indicators", limit=limit))
405
406
/home/user/envs/analysis3/lib/python3.5/site-packages/OTXv2.py in walkapi(self, url, iter, max_page)
272 return self.walkapi_iter(url, max_page=max_page)
273 else:
--> 274 return list(self.walkapi_iter(url, max_page=max_page))
275
276 def getall(self, modified_since=None, author_name=None, limit=20, max_page=None, iter=False):
/home/user/envs/analysis3/lib/python3.5/site-packages/OTXv2.py in walkapi_iter(self, url, max_page)
263 break
264
--> 265 data = self.get(next_page_url)
266 for el in data['results']:
267 yield el
/home/user/envs/analysis3/lib/python3.5/site-packages/OTXv2.py in get(self, url, **kwargs)
112 proxies=self.proxies,
113 )
--> 114 return self.handle_response_errors(response).json()
115 except requests.exceptions.RetryError:
116 raise RetryError()
/home/user/envs/analysis3/lib/python3.5/site-packages/OTXv2.py in handle_response_errors(cls, response)
95 raise BadRequest(_response_json())
96 elif str(response.status_code)[0] != "2":
---> 97 raise Exception("Unexpected http code: %r, response=%r", response.status_code, _response_json())
98
99 return response
Exception: ('Unexpected http code: %r, response=%r', 404, {'internal_error': 'Unable to decode response json: Expecting value: line 1 column 1 (char 0)'})
It would be way better to raise a custom exception here.
The search_pulses function only returns 5 results. Please add functionality to remove this limit when pulling large datasets.
Here is my snippet:
from OTXv2 import OTXv2
from OTXv2 import IndicatorTypes
otx = OTXv2("xxxx")
pulses = otx.getall()
print(len(pulses))
the output:
Traceback (most recent call last):
File "C:\Python39\lib\site-packages\requests\adapters.py", line 486, in send
resp = conn.urlopen(
File "C:\Python39\lib\site-packages\urllib3\connectionpool.py", line 878, in urlopen
return self.urlopen(
File "C:\Python39\lib\site-packages\urllib3\connectionpool.py", line 878, in urlopen
return self.urlopen(
File "C:\Python39\lib\site-packages\urllib3\connectionpool.py", line 878, in urlopen
return self.urlopen(
[Previous line repeated 2 more times]
File "C:\Python39\lib\site-packages\urllib3\connectionpool.py", line 868, in urlopen
retries = retries.increment(method, url, response=response, _pool=self)
File "C:\Python39\lib\site-packages\urllib3\util\retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='otx.alienvault.com', port=443): Max retries exceeded with url: /api/v1/pulses/subscribed?limit=50 (Caused by ResponseError('too many 504 error responses'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python39\lib\site-packages\OTXv2.py", line 178, in get
response = self.session().get(
File "C:\Python39\lib\site-packages\requests\sessions.py", line 600, in get
return self.request("GET", url, **kwargs)
File "C:\Python39\lib\site-packages\requests\sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python39\lib\site-packages\requests\sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "C:\Python39\lib\site-packages\requests\adapters.py", line 510, in send
raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPSConnectionPool(host='otx.alienvault.com', port=443): Max retries exceeded with url: /api/v1/pulses/subscribed?limit=50 (Caused by ResponseError('too many 504 error responses'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Chipuku\Desktop\OTX automation\auto_otx.py", line 5, in <module>
pulses = otx.getall()
File "C:\Python39\lib\site-packages\OTXv2.py", line 404, in getall
return self.walkapi(
File "C:\Python39\lib\site-packages\OTXv2.py", line 385, in walkapi
return list(self.walkapi_iter(url, max_page=max_page, max_items=max_items, method=method, body=body))
File "C:\Python39\lib\site-packages\OTXv2.py", line 366, in walkapi_iter
data = self.get(next_page_url)
File "C:\Python39\lib\site-packages\OTXv2.py", line 187, in get
raise RetryError()
OTXv2.RetryError: 'Exceeded maximum number of retries'
Why is that happend ?
Smooth gameplay playing, getting MVP
from OTXv2 import OTXv2
from pandas.io.json import json_normalize
from datetime import datetime, timedelta
otx = OTXv2(my key)
otx.get("http://otx.alienvault.com/pulse/56e96f904637f2285c708f55")
ValueError Traceback (most recent call last)
in ()
----> 1 otx.get("/pulse/56e71ff467db8c4088b184fe")
\OTX-Python-SDK\OTXv2.pyc in get(self, url)
66 response = None
67 try:
---> 68 response = request.open(url)
69 except URLError as e:
70 if e.code == 403:
C:\Anaconda\lib\urllib2.pyc in open(self, fullurl, data, timeout)
421
422 req.timeout = timeout
--> 423 protocol = req.get_type()
424
425 # pre-process request
C:\Anaconda\lib\urllib2.pyc in get_type(self)
283 self.type, self.__r_type = splittype(self.__original)
284 if self.type is None:
--> 285 raise ValueError, "unknown url type: %s" % self.__original
286 return self.type
287
I;m trying to download a specific pulse, how can I do it if not this way?
Thank you
events = otx.getevents_since(mtime)
it doesnt return pulses id.
Example:
{
"action": "unsubscribe",
"created": "2015-12-08T19:40:05.337000",
"id": "566732154637f27ede869c34",
"object_id": "AlienVault",
"object_type": "user"
}
https://otx.alienvault.com/pulse/566732154637f27ede869c34/ - Pulse does not exist
With the new version of OTX-Python-SDK I get an error when calling getall(), which has not been there with the older version of the SDK.
It can be reproduced with my get-otx-iocs.py script and the right subscriptions.
https://github.com/Neo23x0/Loki/blob/master/threatintel/get-otx-iocs.py
As it happens in line 53 of OTXv2.py when decoding the whole response, it is diffucult to determin which pulse has the UNICODE characters that cannot be decoded.
prometheus:threatintel neo$ python get-otx-iocs_flo.py
Starting OTX feed download ...
Traceback (most recent call last):
File "get-otx-iocs_flo.py", line 142, in <module>
otx_receiver.get_iocs_last()
File "get-otx-iocs_flo.py", line 50, in get_iocs_last
self.events = self.otx.getall()
File "/Library/Python/2.7/site-packages/OTXv2.py", line 62, in getall
json_data = self.get(next)
File "/Library/Python/2.7/site-packages/OTXv2.py", line 53, in get
data = response.read().decode()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 128233: ordinal not in range(128)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.