Comments (6)
Sure, that can be done, however are you sure that's really what you want?
Right now the timeout is on 2 seconds to connect (tcp), and 60 seconds for receiving the data.
The 2 seconds allow a user to know quite quickly when a server is not reachable (2 seconds).
The 60 seconds are how long MISP-maltego will wait for the response from a server. Waiting for a whole minute to get results is already kinda long, no?
Which timeout are you hitting? (the connect or response?)
from misp-maltego.
Yes - it does seem a bit extreme. But we have a massive instance of MISP behind the curtain, and regularly see multi-minute waits. We've gotten used to it - and this enhancement would help us work around it.
It's the response timeout that we are getting hit with, the connect seems ok (at least in our circumstance).
from misp-maltego.
Hi @cvandeplas -- I work with @netmg. I'm new to Maltego -- your plugin is very cool! And thanks for being so responsive here!
Being that these recent postings peeked my curiosity re' the timeout, I wrote a simple script that calls pymisp.search
the same as your AttributeToEvent line 176 to measure time and data size simulating a To MISP Events
Maltego click on an attribute that resides on an event with 1M attributes (yes, large events are common here). Here's the interesting results based on various scenarios we used when performance becomes an issue:
- Your
search
call as is withcontroller='events'
retrieved:
3GB in 266 secs
- Adding
metadata=True
withcontroller='events'
retrieved:
300KB in 10 secs
- Changing the
search
tocontroller='attributes'
retrieved:
300KB in 10 secs
Being new with your code, what's not clear, is whether options 2 and 3 above will yield the data content you need further downstream in the Maltego client. I'm eager to hear your thoughts. Thanks again.
from misp-maltego.
I've also found that setting a "limit" value, even if very high (i.e., 1,000,000) can significantly improve pymisp response times (at least in JupyterLab):
r = misp.search(
controller='attributes',
type_attribute=['ip-src', 'ip-dst', 'url', 'domain'],
#org='IBM-ATP',
tags=['nist-cyber-framework:function="protect"','nist-cyber-framework:function="detect"'],
metadata=True,
pythonify=True,
publish_timestamp='3d',
limit=1,000,000)
from misp-maltego.
Being new with your code, what's not clear, is whether options 2 and 3 above will yield the data content you need further downstream in the Maltego client. I'm eager to hear your thoughts. Thanks again.
There is indeed optimisation possible, especially with events that are so large.
The currently logic is:
- search for the event that contains that attribute, return the full event including attributes & objects
- search in the result if the attribute is part of the event , or if it is part of the object
- if it's a direct attribute: return the event
- if it's within an object: return a MISPObject instead
The reason I did so was that without the check there would be a direct link between the attribute and the event, breaking a bit the visual look. However I did not anticipate you would be using such large datasets.
Changing the code with metadata=True
will break things for sure. The event_to_entity for example expects a full event, as it uses that to return the attribute & object count.
So technically the code can be changed to accommodate this situation, but honestly, I think we need to look at it with a big step back to not break the other functionality & usability. Maybe some things can be improved by requesting small changes on MISP side too. (like returning attribute & object counts)
Please feel free to play around with the code, I'm happy to review pull-requests and merge them in the project code.
from misp-maltego.
@cvandeplas Thanks for the feedback. Don't have cycles to attack this now, but it's in the queue.
from misp-maltego.
Related Issues (20)
- Local MISP sends an error message to Maltego client HOT 11
- MISP ask for URL and Server on Maltego HOT 1
- How to add local Transforms to an Windows XXL Maltego Client HOT 3
- MISP_maltego is no connecting. HOT 5
- Meltago to MISP server not connecting HOT 5
- MISP-Maltego locally installed HOT 7
- Transform Execution Failed HOT 2
- Cannot Connect to MISP server HOT 3
- affiliation facebook HOT 2
- Who to contact for security issues HOT 1
- port supported in URL in MISP_maltego.conf? HOT 10
- running MISP-maltego inside (centralized deployment) container HOT 5
- ERROR
- error: Setup script exited with error in safedexml setup command: use_2to3 is invalid. HOT 3
- ImportError: cannot import name 'Iterable' from 'collections' (/usr/lib/python3.10/collections/__init__.py)
- Maltego transform fails to run HOT 1
- cannot import misp-maltego package
- AttributeError: type object 'meta' has no attribute 'namespace' in safedexml packages HOT 2
- ToTags returns error: TypeError: 'NoneType' object is not subscriptable (from entity )
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from misp-maltego.