misp / misp-maltego Goto Github PK

Set of Maltego transforms to inferface with a MISP Threat Sharing instance, and also to explore the whole MITRE ATT&CK dataset.

License: GNU Affero General Public License v3.0

Python 97.60% Shell 0.13% Dockerfile 2.26%

misp maltego misp-maltego attack mitre-attack transform pivoting visualisation graph analysis threat-intelligence threat-intel

misp-maltego's Introduction

This is a Maltego MISP integration tool allowing you to view (read-only) data from a MISP instance. It also allows browsing through the MITRE ATT&CK entities. (no MISP connection needed)

This user guide should help you through the installation of MISP-Maltego, and should guide you how to use it through a few use-cases. As this is a collaborative project, do not hesitate to propose changes, write other use-cases or raise feature requests for missing features.

Quick start

Currently supported MISP elements are : Event, Attribute, Object (incl relations), Tag, Taxonomy, Galaxy (incl relations).

Once installed you can start by creating a MISPEvent entity, then load the Machine EventToAll or the transform EventToAttributes.

Alternatively initiate a transform on an existing Maltego entity. The currently supported entities are: AS, DNSName, Domain, EmailAddress, File, Hash, IPv4Address, NSRecord, Person, PhoneNumber, URL, Website

For MITRE ATT&CK pivoting, feel free to start with an Attack Technique, Software, Threat Actor, or MISPGalaxy. Create your entity, enter a keyword such as %gama% and use the Search in MISP transform to get started.

Installation

Transform Hub

Open the Transform Hub, locate ATT&CK - MISP and press the Install button.

Your transforms will go through Paterva's servers and ours. See the Transform Hub Disclaimer for more information.

ATT&CK transforms do not require a MISP server or API key to be configured.
MISP transforms requires your MISP server to be reachable from the internet! To enter your MISP server URL and key click Details on the Transform Hub item and then Settings at the bottom right.

Local Transform Installation

If you trust nobody, or just want to connect to your local MISP server you can install everything as local transforms.

These instructions have been tested on Ubuntu 18.04 LTS, but should be similar on other systems.

Download and install Maltego
Install using pip: sudo pip3 install MISP-maltego
Generate the Maltego bundle: canari create-profile MISP_maltego
Import this bundle in Maltego.
1. Open Maltego
2. Click on the home button (Maltego icon, top-left corner).
3. Click on 'Import'
4. Click on 'Import Configuration'.
5. Load the MISP_maltego.mtz file and follow the prompts.
Edit $HOME/.canari/MISP_maltego.conf and enter your misp_url and misp_key

Custom Entities

MISP-Maltego tries to use as much as possible the default Paterva entities, or the most popular from the community. It however comes with a few custom entities:

MISPEvent: A representation of an Event on MISP, containing Attributes (MISP) / Entities (Maltego)
MISPObject: A way to group associated attributes in a structured way.
MISPGalaxy: A Tag containing much more metadata. Please refer to the MISP Galaxy for more information. MITRE ATT&CK is for example completely available through MISPGalaxy entities (see use-cases for an example)
Attack Technique: Attack patterns or techniques, see MITRE ATT&CK for more information.
Threat Actor: Threat actor or intrusion sets.
Software: Software is a generic term for custom or commercial code, operating system utilities, open-source software, or other tools used to conduct behavior modeled in ATT&CK.

Use Cases

Transform on existing data

In this use case we will be using already existing entities and will initiate a transform using MISP. The currently supported entities are: AS, DNSName, Domain, EmailAddress, File, Hash, IPv4Address, NSRecord, Person, PhoneNumber, URL, Website.

Example:

create an entity domain with the value 1dnscontrol.com.
right click and choose Local Transforms > MISP_maltego > Domain To Event
continue loading transforms on the MISP Event

Transform from MISP Event ID

While MISP already has a graphing capability we would like to use the power of Maltego to look at the data and expand the work.

Create a MISP Event and give it an event id, or UUID
One manual way is to right click and choose Local Transforms > MISP_maltego > Event To Attributes
- Notice the event is transformed to Attributes, Objects, Tags, Galaxies and related MISP Events
- You can now further transform on an Object > Object To Attributes and see the content of the object
Alternatively you can also use the Maltego Machine to speed up things.
- Click on the MISP Event and in the left menu choose Event to All in the Machines section.
- Notice that the whole event, objects and such will get expanded with data from your MISP instance.
You can now further transform on any data.

Which data is already in MISP?

If you use MISP as central database it can be quite convenient to know which data is present in MISP, and which data is not; especially after using a number of other transforms. To permit this MISP-Maltego will always add a green bookmark to all the data that is present in MISP.

Searching in MISP using keywords

As with the MISP attribute search through the MISP Web UI you can use % wildcards at the front and end to specify the substring. You might be tempted to always use %keyword%, but bare in mind how databases indexes work; a search for keyword% will always be much faster than %keyword.

Transform from Galaxy

Galaxies are actually tags with much more contextual data. Examples are threat actors, malware families, but also the whole MITRE ATT&CK data is available as Galaxy. All this data comes from the MISP Galaxy repository. Today the integration is not done using a MISP server because of limitations in MISP. You might encounter Galaxies when transforming from MISP Events or Attributes. An alternative use-case is by starting immediately from a Galaxy. There are 3 ways to manually create a good Galaxy Entity.

Using a find capability (see below)
Create the Galaxy and set the UUID. You can find the UUIDs in the MISP Galaxy repository.
Create the Galaxy with the right tag name; for example: misp-galaxy:

To use the magical search feature:

Create a MISP Galaxy and type the keyword as value.
Run the Galaxy To Relation transform, notice the search results will appear as connected entities
Remove the non-relevant entities, including the your search-keyword

Visualize MITRE ATT&CK

Apply the same steps for MITRE ATT&CK browsing:

You might end up with such a graph:

Visualise common ATT&CK patterns

Having access to a large amount of Threat information through MISP Threat Sharing communities gives you outstanding opportunities to aggregate this information and take the process of trying to understand how all this data fits together telling a broader story to the next level. We are transforming technical data or indicators of compromise (IOCs) into cyber threat intelligence. This is where the analytical challenge begins. [read more]

Massively large MISP event? Think before you transform.

In some communities such as the COVID-19 MISP some events contain tens of thousands attributes. Loading all the attributes from these events might not be a good idea if you do not have Maltego XL. You can see the amount of attributes and objects in the Event properties, so you can think before you click:

License

This software is licensed under GNU Affero General Public License version 3

Note: Before being rewritten from scratch this project was maintained by Emmanuel Bouillon. The code is available in the v1 branch.

The logo is CC-BY-SA and was designed by Françoise Penninckx

The icons in the intelligence-icons folder are from intelligence-icons licensed CC-BY-SA - Françoise Penninckx, Brett Jordan

misp-maltego's People

Contributors

Stargazers

Watchers

misp-maltego's Issues

How to add local Transforms to an Windows XXL Maltego Client

Hi, i am running a Windows 10 Client with Maltego XXL and need a connection to the local MISP instance. With the embedded Hub i have an Error Message. So our internal instance of MISP has no Internet connection. We have also an external MISP but without Company Knowledge TLP Red and amber.

Need a little help ;o)

Thx

UnicodeEncodeError: 'ascii' codec can't encode character

every time I try to run the transform on a MISP event I end up getting this error, any idea what may be causing it?

stealth mode prevents custom galaxy icons to be loaded

Recently Maltego announced "stealth mode" , this currently prevents custom galaxy icons to be loaded.

TODO: find a workaround

incorrect or unreachable misp_url results in HTTP 503 error in remote transform

TODO; test and fix this

MISP ask for URL and Server on Maltego

ATT&CK transforms do not require a MISP server or API key to be configured." according to https://github.com/MISP/MISP-maltego.

But I'm trying to do an example on Maltego CE and a pop up window asks me for URL and Server.

Steps:

Add a Mail entity.
Use the "To MISP events" transform.

Result:

Any idea?

Error when compiling the MTZ file locally on Windows

 C:\Users\Administrator\projects\2086.MISP_maltego>canari create-profile MISP_maltego
Loading Canari configuration file 'C:\\Users\\Administrator\\AppData\\Local\\canari\\canari.conf'
Looking for transforms in MISP_maltego...
Package loaded.
Creating profile C:\Users\Administrator\projects\2086.MISP_maltego\MISP_maltego.mtz...
Installing transform MISP_maltego.SearchInMISP from MISP_maltego.transforms.attributetoevent.SearchInMISP...
Traceback (most recent call last):
  File "C:\Users\Administrator\projects\2086.MISP_maltego\venv-p\Scripts\canari-script.py", line 11, in <module>
    load_entry_point('canari==3.3.10', 'console_scripts', 'canari')()
  File "C:\Users\Administrator\projects\2086.MISP_maltego\venv-p\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\Administrator\projects\2086.MISP_maltego\venv-p\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "C:\Users\Administrator\projects\2086.MISP_maltego\venv-p\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\Administrator\projects\2086.MISP_maltego\venv-p\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\Administrator\projects\2086.MISP_maltego\venv-p\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\Administrator\projects\2086.MISP_maltego\venv-p\lib\site-packages\click\decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "C:\Users\Administrator\projects\2086.MISP_maltego\venv-p\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\Administrator\projects\2086.MISP_maltego\venv-p\lib\site-packages\canari\entrypoints.py", line 86, in create_profile
    create_profile(ctx.config_dir, ctx.project, package)
  File "C:\Users\Administrator\projects\2086.MISP_maltego\venv-p\lib\site-packages\canari\commands\create_profile.py", line 23, in create_profile
    transform_package.create_profile(config_dir, mtz_dir, in_project=in_project)
  File "C:\Users\Administrator\projects\2086.MISP_maltego\venv-p\lib\site-packages\canari\pkgutils\transform.py", line 393, in create_profile
    self.install(install_prefix, mtz, configure, in_project=in_project)
  File "C:\Users\Administrator\projects\2086.MISP_maltego\venv-p\lib\site-packages\canari\pkgutils\transform.py", line 316, in install
    self._install_transforms(install_prefix, distribution, in_project)
  File "C:\Users\Administrator\projects\2086.MISP_maltego\venv-p\lib\site-packages\canari\pkgutils\transform.py", line 327, in _install_transforms
    distribution.add_transform(prefix, 'Local', transform, server='Local')
  File "C:\Users\Administrator\projects\2086.MISP_maltego\venv-p\lib\site-packages\canari\pkgutils\maltego.py", line 343, in add_transform
    self.write_file(
  File "C:\Users\Administrator\projects\2086.MISP_maltego\venv-p\lib\site-packages\canari\pkgutils\maltego.py", line 705, in write_file
    contents = contents.render(encoding='utf-8', fragment=True, pretty=True)
  File "C:\Users\Administrator\projects\2086.MISP_maltego\venv-p\lib\site-packages\safedexml\__init__.py", line 417, in render
    data.extend(self._render(nsmap))
  File "C:\Users\Administrator\projects\2086.MISP_maltego\venv-p\lib\site-packages\safedexml\__init__.py", line 494, in _render
    for child in children:
  File "C:\Users\Administrator\projects\2086.MISP_maltego\venv-p\lib\site-packages\safedexml\__init__.py", line 530, in _render_children
    for data in datas:
  File "C:\Users\Administrator\projects\2086.MISP_maltego\venv-p\lib\site-packages\safedexml\fields.py", line 620, in render_children
    for data in chunks:
  File "C:\Users\Administrator\projects\2086.MISP_maltego\venv-p\lib\site-packages\safedexml\fields.py", line 603, in child_chunks
    for data in self.field.render_children(obj, item, nsmap):
  File "C:\Users\Administrator\projects\2086.MISP_maltego\venv-p\lib\site-packages\safedexml\fields.py", line 485, in render_children
    for data in val._render(nsmap):
  File "C:\Users\Administrator\projects\2086.MISP_maltego\venv-p\lib\site-packages\safedexml\__init__.py", line 500, in _render
    raise RenderError("Field '%s' is missing" % (f.field_name,))
safedexml.RenderError: Field 'value' is missing
Writing transform set Misp_Maltego to C:\Users\Administrator\projects\2086.MISP_maltego\MISP_maltego.mtz...
Writing server Local to C:\Users\Administrator\projects\2086.MISP_maltego\MISP_maltego.mtz

Python: 3.8
canarI: 3.3.10
MISP-maltego: 1.4.5

Search in MISP should return the same type of entity

Search in MISP returns all types of entities now.
So when you search on an Email Address it will also return domain names.

Is this a feature or a bug?

event level correlations

Event level correlations should be handled differently than they are handled now.
MISP correlates events when attributes correlate.
However using this in Maltego directly is wrong:

it creates a link between two entities without the contextual information. (a correlation is on any type of attribute)
it weakens the links attributes/objects make when correlating with other events.

Ideas to resolve this:

remove the event correlation. On the other hand it's good to know it's there so you will be tempted to do an AttributeToEvent transform.
weaken the strength on the event level correlation (but how? )

performance issue with Search in MISP

For reference, I’m accessing the https://covid-19.iglocska.eu/ community. I tried running a few simple wildcard searches to pull in relevant events, which seem to always result in 504 errors. For instance, I tried running both “Search in MISP” and “To MISP Events” from a Phrase entity with the value “n95%“.
This yields search results pretty quickly on the web version of the community).

In this case, this code is the one doing the search, so using the PyMISP search() function to do a remote search on the (covid-19) MISP instance.

It is possible that some quirks on PyMISP level or MISP level cause a slow response when searching through the API.
A quick check confirms, in pymisp this is terribly slow:

misp = PyMISP('https://covid-19.iglocska.eu/', 'mykey')
r = misp.search(controller='events', value='n95%', with_attachments=False)

This needs to be researched further:

what is sent to MISP
is PyMISP or MISP at fault?

caching system when running as local transform

When running as local transform it would be good to have a caching system to allow MISP_maltego to not query everything from the MISP instance.

The challenge in implementing this lies in the question "how long should a cache be valid?"

Usually data in MISP will likely not change within minutes, on the other hand feature request #18 From Maltego to MISP will require immediate interaction.

Ideas:

only work if it's running as local transform. (as remote transform it would breach data confidentiality)
implement caching by default for X minutes
if upload to MISP is done, purge cache (completely or for the edited event)

show the number of attributes in a MISP event

Before pivoting on an event we might want to know how many attributes/objects are present in an event. An analyst might choose not to expand an event if there are too many attributes.

This feature request is to find a way to show this to the user

overlay on icon with number?
color marker to warn for high attribute count
...?

Other ideas are welcome

running MISP-maltego inside (centralized deployment) container

I'm deploying MISP-maltego in a centralized fashion (because we have a large user base and we don't want to support individual installations on each user desktop). The container runs alongside our internal CTAS, iTDS and MISP servers. Inside the container, configurations have been modified by adding our misp URL and key to:

/var/plume/canari.conf
/var/plume/MISP_maltego.conf
/usr/local/lib/python3.8/site-packages/MISP_maltego/resources/etc/MISP_maltego.conf

However - these comments in the container have me confused (in canari.conf) - do I need to change anything here?:


[canari.remote]
# This section deals exclusively with remote transform execution and has no effect on local transform execution. Modify
# these options if you are running your transforms under plume.

# Additional config files that should be read to merge with the current config in remote exec mode.
configs =

# Specify any transforms that your WSGI container will host. This is for Plume ONLY.
packages =

The running container looks healthy to me (from inside container):

root@415ca0be8d10:/# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Mar09 ?        00:00:00 /bin/sh /etc/init.d/plume start-docker
nobody       7     1  0 Mar09 ?        00:00:11 /usr/local/bin/python /usr/local/bin/twistd --rundir=/var/plume --uid=65534 --gid=1000 -n web --wsgi=canari.tas.plume.application --port=tcp
root        14     0  0 Mar09 pts/0    00:00:00 /bin/bash

And if I try to curl to it from outside the container, I get an answer:

-bash-4.2$ curl -vv -k http://<dockerhost>:<misp-maltego port>/MISP_maltego.SearchInMISP
* About to connect() to <dockerhost> port misp-maltego-port> (#0)
*   Trying 1.2.3.4...
* Connected to <dockerhost> (1.2.3.4) port <misp-maltego-port> (#0)
> GET /MISP_maltego.SearchInMISP HTTP/1.1
> User-Agent: curl/7.29.0
> Host: <dockerhost>:<misp-maltego-port>
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: TwistedWeb/22.1.0
< Date: Thu, 10 Mar 2022 19:45:01 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 4
<
* Connection #0 to host <dockerhost> left intact
Yes?

-bash-4.2$

This all looks great up to this point.

However.... I'm not sure where to go from here? On the maltego client I'm assuming I need to add an internal transform hub item pointing to this container with a seed URL? If correct - where does value needed for the seed come from?

Once the seed has been configured I'm hoping all available MISP-maltego transformations will be present in the client transform drop-down menus?

Thanks as-always for any help provided.

Support multiple MISP Servers

A potentially easier way would be when generating the mtz file, to also have a parameter in the configuration file to have the name of the MISP instance (ex.: superSecretServer), and also use that name in the transform folder within Maltego from MISP_maltego to MISP_maltego (superSecretServert).

This would allow users to create access for as many MISP instances as they want, and they could also easily select from which MISP instances that they can query.

identify which entities are already known in MISP

While working on a dataset created or expanded with non-MISP data sources it is practical to easily identify which entities are already stored in MISP, and which ones aren't.

One way to do this would be to add a distinct color-bookmark to each entity

Unable to run event2ip attribute

/usr/local/lib/python2.7/dist-packages/pymisp/api.py:20: UserWarning: You're using python 2, it is strongly recommended to use python >=3.4
warnings.warn("You're using python 2, it is strongly recommended to use python >=3.4")
/usr/local/lib/python2.7/dist-packages/pymisp/mispevent.py:45: UserWarning: You're using python 2, it is strongly recommended to use python >=3.4
warnings.warn("You're using python 2, it is strongly recommended to use python >=3.4")
/usr/local/lib/python2.7/dist-packages/pymisp/api.py:39: UserWarning: You're using python 2, it is strongly recommended to use python >=3.4
warnings.warn("You're using python 2, it is strongly recommended to use python >=3.4")
/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
Traceback (most recent call last):
File "misp_event2ip.py", line 37, in
mt.addUIMessage("[ERROR] " + str(e))
NameError: name 'mt' is not defined

python --version
Python 2.7.12

python3 --version
Python 3.5.2

If i put in an IP or set of IPs i can search for the associated event. However, after the event(s) have been returned and I want to see all of the IPs in that event I get the above error. I've tried different file associations in the transform configuration and whatnot. Nothing seems to resolve this.

Jimbo

Use instance specific Galaxies

Today the MISP Galaxies used are those from the MISP-galaxies public project.
A copy of the github repository is downloaded and the local data is used.
This was done because of limited support of the Galaxies by MISP webui. (for example: relations are not supported)

Once MISP supports the galaxies fully (see MISP/MISP#3801), MISP-maltego should use the REST API to query the galaxies, as a MISP instance might have private non-public galaxies.

maltego.hashtag and maltego.Hashtag

The default Maltego entity is called maltego.hashtag.
However when MISP-maltego creates such tags it's called maltego.Hashtag. Some transforms do not work because of this.

This also partially relates to malleum-inc/canari3#66 where the entity name on canari3 level is twitter.hashtag.

MISP-Maltego locally installed

I have installed MISP-Maltego locally on Windows and Linux

Windows : ERROR: Cannot connect to MISP server. Please verify your MISP_Maltego.conf settings

and

Linux - Error running local transform: java.io.IOException: Cannot run program "/usr/local/bin/dispatcher" (in directory "\home\MISP-maltego\src"): CreateProcess error=2, The system cannot find the file specified (from entity

I have my Maltego installed on Windows 10.

From Maltego to MISP

While the current version only allows visualizing data from MISP there are use-cases for uploading data to MISP.

This issue is to discuss potential features and track the specific use-cases:

add attribute to existing event
create new event
add object to existing event
add attribute to existing object
add tag to attribute
add tag to event
add galaxy to attribute
add galaxy to event
create relationship between objects/attributes (object > object/attribute)

Please add and describe use-cases you can think of.

deadlock when galaxies github download fails

When the galaxies github download fails misp-maltego server ends in a deadlock.
This is caused by the /tmp/MISP-maltego/*.lock file that's not removed.

galaxies download fails since master -> main rename of galaxies github

the url and file paths are wrong

Allow wildcard prefix/suffix search for attribute types

Allow wildcard searches, as suffix or as prefix, for some attribute types.
Examples:

%.domain.com for domain
%.domain.com% for domain
subodmain.domain.com% for URL
etc...

This could also allow cross-attribute type pivoting (evil.com stored as domain and pivoted to %@%evil.com emails); which could be very interesting in a Maltego machine.

Proxy issue: Traceback with merge_environment_settings

I'm stuck with a strange python requests/session issue using MISP-maltego on a clean environment.

This installation doesn't need a proxy to reach my misp instance and curl is just running fine. No "proxy" environment variables set.
Did someone see that before?

Traceback (most recent call last):
  File "/opt/maltego-venv/lib/python3.7/site-packages/pymisp/api.py", line 119, in __init__
    response = self.recommended_pymisp_version
  File "/opt/maltego-venv/lib/python3.7/site-packages/pymisp/api.py", line 175, in recommended_pymisp_version
    response = self._prepare_request('GET', 'servers/getPyMISPVersion.json')
  File "/opt/maltego-venv/lib/python3.7/site-packages/pymisp/api.py", line 2362, in _prepare_request
    settings = s.merge_environment_settings(req.url, proxies=self.proxies or {}, stream=None, verify=self.ssl, cert=self.cert)
  File "/opt/maltego-venv/lib/python3.7/site-packages/requests/sessions.py", line 698, in merge_environment_settings
    no_proxy = proxies.get('no_proxy') if proxies is not None else None
AttributeError: 'bool' object has no attribute 'get'

affiliation facebook

affiliation facebook Search tools are different than they should be

Other options should appear

It is supposed to appear

galaxies break when used on Maltego XL

It works a first time, but not a second time. The stacktrace points to util.py and line 551

        cluster_uuids = json.load(f)

The cause is likely simultaneous creation of the local_path_uuid_mapping by different simultaneous threads.
If so, locks should prevent this from happening.

[Error] 503 Server Error: Service Temporarily Unavailable for url:

Getting this 503 error using maltego, any pointers where one can look for troubleshooting info?

Stack trace when Event To Attributes - template has no template_uuid

It seems that in some use-cases the MISP Object does not have a template_uuid .

Is this a bug on MISP-maltego or on MISP?

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/canari/maltego/runner.py", line 130, in local_transform_runner
    config
  File "/usr/local/lib/python3.6/dist-packages/MISP_maltego/transforms/eventtoattributes.py", line 109, in do_transform
    self.gen_response_objects()
  File "/usr/local/lib/python3.6/dist-packages/MISP_maltego/transforms/eventtoattributes.py", line 79, in gen_response_objects
    self.response += self.conn.object_to_entity(o)
  File "/usr/local/lib/python3.6/dist-packages/MISP_maltego/transforms/common/util.py", line 106, in object_to_entity
    o_template = self.misp.get_object_template(o['template_uuid'])
  File "/usr/local/lib/python3.6/dist-packages/pymisp/api.py", line 396, in get_object_template
    object_template_id = get_uuid_or_id_from_abstract_misp(object_template)
  File "/usr/local/lib/python3.6/dist-packages/pymisp/api.py", line 54, in get_uuid_or_id_from_abstract_misp
    if 'uuid' in obj:
TypeError: argument of type 'NoneType' is not iterable (from entity "12786")
Transform Event To Attributes returned with an error: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/canari/maltego/runner.py", line 130, in local_transform_runner
    config
  File "/usr/local/lib/python3.6/dist-packages/MISP_maltego/transforms/eventtoattributes.py", line 109, in do_transform
    self.gen_response_objects()
  File "/usr/local/lib/python3.6/dist-packages/MISP_maltego/transforms/eventtoattributes.py", line 79, in gen_response_objects
    self.response += self.conn.object_to_entity(o)
  File "/usr/local/lib/python3.6/dist-packages/MISP_maltego/transforms/common/util.py", line 106, in object_to_entity
    o_template = self.misp.get_object_template(o['template_uuid'])
  File "/usr/local/lib/python3.6/dist-packages/pymisp/api.py", line 396, in get_object_template
    object_template_id = get_uuid_or_id_from_abstract_misp(object_template)
  File "/usr/local/lib/python3.6/dist-packages/pymisp/api.py", line 54, in get_uuid_or_id_from_abstract_misp
    if 'uuid' in obj:
TypeError: argument of type 'NoneType' is not iterable (from entity "12786")

Cannot Connect to MISP server

Hi all, great tool! Looking forward to getting this up and running:

Getting an error "cannot connect to the MISP server. Please verify your settings and ensure the MISP server is reachable from the internet."

Our MISP is currently internal, and we can access it via Browser. I have the URI and the API Key generated from my account in here. Are there settings within MISP that potentially need to be changed in order to allow the API call? Or potentially a firewall port?

Thanks again!

Potential memory leak somewhere

There is maybe a memory leak in the transform server code somewhere:

Meltago to MISP server not connecting

Getting error ERROR: Cannot connect to MISP server. Please verify your settings (MISP URL and API key), and ensure the MISP server is reachable from the internet (from entity

But I have got the URL as the IP address and the api key

What am I doing wrong.

nginx returns a 504 gateway timeout before pymisp timeouts when the MISP server is not accessible remotely

This is probably because pymisp timeouts are longer than nginx timeouts (when connecting to the MISP instance).
This results in an error that does not help the user.

TODO: investigate and fix (add timeout on initial pymisp connection or change timeouts in nginx reverse proxy)

different entities for actor, tool/malware, technique, ...

This makes the graphs more readable

custom icons per object

The MISP objects all have the same icon.
When looking at different type of entities this is confusing as there's no easy way to distinguish visually the different objects.

We should therefore show a different icon depending on the object: Network, Person, email, file, ...

Support NetBlock

Support NetBlock entities for lookups (they are in the format 1.1.1.1-1.1.1.255) for lookups in MISP.

Search in MISP for MISPEvent not working on string

A MISP event entity cannot have text as string.
The search in MISP is therefore confusing.

TODO: implement a human readable name for the MISP Event entity and fix this.

Transform error

Running into this error when executing MISP Transform:

Transform to MISP Event returned with an error: Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/canari/maltego/runner.py", line 130, in local_transform_runner
config
File "/usr/local/lib/python3.7/site-packages/MISP_maltego/transforms/attributetoevent.py", line 109, in do_transform
for e in events_json['response']:
KeyError: 'response' (from entity "1dnscontrol.com")
Transform to MISP Event done (from entity "1dnscontrol.com")
Transform in MISP? returned with an error: Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/canari/maltego/runner.py", line 130, in local_transform_runner
config
File "/usr/local/lib/python3.7/site-packages/MISP_maltego/transforms/attributetoevent.py", line 36, in do_transform
for e in events_json['response']:
KeyError: 'response' (from entity "1dnscontrol.com")

Cannot run program "/usr/local/bin/dispatcher" (in directory "\root\MISP-maltego\src")

When running from Maltego Classic on Windows 10 I get the following error after installation when running a transform:

Error running local transform: java.io.IOException: Cannot run program "/usr/local/bin/dispatcher" (in directory "\root\MISP-maltego\src"): CreateProcess error=2, The system cannot find the file specified (from entity "0")

Installed Python3.5.

externalize timeouts?

Hi Christophe -

I'm just scraping the surface on your code, but after experiencing some timeouts with our MISP server I experimented with the same line that was changed for #52 .

88 self.misp = PyMISP(url=misp_url, key=misp_key, ssl=misp_verify, debug=misp_debug, tool='misp_maltego', timeout=(2, 60))

I found I could modify the 2nd parameter and avoid the timeouts.

Would it be possible to externalize both of these timeout values, making them accessible in MISP_maltego.conf ?

Publish MISP-maltego as PIP package

This would allow users to simply pip3 install misp-maltego and have all the dependencies resolved automagically.

Limiting transforms depending of PAP or TLP classification of a MISP event

Limiting transforms depending of PAP or TLP classification of a MISP event as discussed during the session.

Error message: No module named MISP_maltego

I am trying to install the package in my MISP instance regarding to the tutorial available here.
After installing MISP-maltego, i tried to generate the Maltego bundle with canari command.
After running the command i recieve following error message:

Loading Canari configuration file '/home/misp/.canari/canari.conf'
Usage: canari create-profile [OPTIONS] [PACKAGE]

Error: Invalid value for "[PACKAGE]": Does not appear to be a valid canari package. Couldn't import the 'MISP_maltego' module in 'MISP_maltego'. Error message: No module named MISP_maltego

I can not find any solution for this error, can you please advice me how i can do this step?

Thanks

Who to contact for security issues

Hey there!

I belong to an open source security research community, and a member (@DiG2) has found an issue, but doesn’t know the best way to disclose it.

If not a hassle, might you kindly add a SECURITY.md file with an email, or another contact method? GitHub recommends this best practice to ensure security issues are responsibly disclosed, and it would serve as a simple instruction for security researchers in the future.

Thank you for your consideration, and I look forward to hearing from you!

(cc @huntr-helper)

Local MISP sends an error message to Maltego client

Hi,

Can anyone give guidance to why Maltego client receives an error message connected to local MISP server?
Transform Search in MISP returned with an error: Error running local transform: java.io.IOException: Cannot run program "/usr/local/bin/dispatcher" (in directory "\home\misp.canari"): CreateProcess error=2, The system cannot find the file specified (from entity "91.200.102.245")
Maltego was installed as an docker image and Local MISP instance with GitHub guidelines.
Connection to Maltego client via API (new user with user role).

Error while running local transform

Hey,

I got the following error when running a local transform :

Transform Event To All returned with an error: Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/canari/maltego/runner.py", line 130, in local_transform_runner
config
File "/usr/local/lib/python3.7/dist-packages/MISP_maltego/transforms/eventtoattributes.py", line 94, in do_transform
if super().do_transform(request, response, config):
File "/usr/local/lib/python3.7/dist-packages/MISP_maltego/transforms/eventtoattributes.py", line 42, in do_transform
search_result = self.misp.search(controller='events', eventid=event_id, with_attachments=False)
AttributeError: 'NoneType' object has no attribute 'search' (from entity "57588")

implement different entities per galaxy type

Right now different Galaxies (actor vs technique) are grouped together by the collections, as these are the same entity type in Maltego.

Using different entity types would resolve this issue. (using inheritance).
Generate the entities dynamically at package creation.

MISP_maltego is no connecting.

Hi Guys,

I'm currently facing issues connecting Maltego with MISP-maltego. Im working using: Ubuntu 20.04, Maltego 4.2.14 and MISP_maltego utility.py v1.4.6'

This is the error message that I received while running the transform:

Transform outpu:
ERROR: Cannot connect to MISP server. Please verify your MISP_Maltego.conf settings (from entity "1113")
Transform Search in MISP returned with an error: ERROR: Cannot connect to MISP server. Please verify your MISP_Maltego.conf settings (from entity "1113")
Transform Search in MISP done (from entity "1113")

Debug:
Loading Canari configuration file '/home/amdocsmisp/.canari/canari.conf'
ERROR: Cannot connect to MISP server. Please verify your MISP_Maltego.conf settings.
INFO [api.py:124 - init() ] To configure logging in your script, leave it to None and use the following: import logging; logging.getLogger('pymisp').setLevel(logging.DEBUG)
DEBUG [api.py:3001 - _prepare_request() ] GET - https://x.x.x.x/servers/getPyMISPVersion.json
DEBUG [api.py:3003 - _prepare_request() ] {}
DEBUG [api.py:3021 - _prepare_request() ] {'User-Agent': 'PyMISP 2.4.134 - Python 3.8 - misp_maltego', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Authorization': '9KDdXF...', 'content-type': 'application/json'}

My MISP_maltego.conf has the following config:

[MISP_maltego.local]
misp_url = http://x.x.x.x/
misp_key = 9KDdXFr....

misp_verify = True
misp_debug = True

check_updates = True

[MISP_maltego.remote]

I test this local transfor using Windows 10 and still not able to connect Maltego with my local MISP Server. Any idea why Maltego is not able to connect to my local MISP server?

inheritance problem with ThreatActor, Software, ...

Some entities such as ThreatActor, Software do not have the proper inheritance configured.

port supported in URL in MISP_maltego.conf?

Hi -

We are trying to run MISP-maltego via a local install, installed on each user's desktop alongside a Maltego client.

We are also running internal MISP, CTAS and ITDS instances (in case it matters - but I don't believe it does).

From the same desktop I've got MISP-maltego installed on, I can curl to the MISP server with the desired url/port and key and get a valid response.

curl -v --header "Authorization: <misp_key_here>" --header "Accept: application/json" --header "Content-Type: application/json" https://<misp_fqdn_here>:<misp_port_here>/

However - when I configure ~/.canari/MISP_maltego.conf with the same values:

[MISP_maltego.local]

misp_url = https://<misp_fqdn_here>:<misp_port_here>/
misp_key = <misp_key_here>

I get the error:

_ERROR: Cannot connect to MISP server. Please verify your MISP_Maltego.conf settings_

It makes me wonder if the port addition in misp_url (required in our environment) is supported by MISP-maltego?

Thanks in advance for any help/comments.

ERROR

When I run create-profile MISP_maltego I get this message on kali Linux Gnome 2022.1
Traceback (most recent call last):
File "/home/kali/.local/bin/canari", line 5, in
from canari.entrypoints import main
File "/home/kali/.local/lib/python3.10/site-packages/canari/entrypoints.py", line 11, in
from canari.commands.framework import (pass_context, CanariPackage, is_new_transform, CanariGroup,
File "/home/kali/.local/lib/python3.10/site-packages/canari/commands/framework.py", line 9, in
from canari.pkgutils.transform import TransformDistribution
File "/home/kali/.local/lib/python3.10/site-packages/canari/pkgutils/transform.py", line 16, in
from canari.maltego.message import EntityTypeFactory
File "/home/kali/.local/lib/python3.10/site-packages/canari/maltego/message.py", line 8, in
from collections import Iterable, OrderedDict
ImportError: cannot import name 'Iterable' from 'collections' (/usr/lib/python3.10/collections/init.py)

Transform Execution Failed

Hello, I am starting to use Maltego, and ATT & CK has caught my attention. I understand that I should not use any API or server for execution. (Leave blank space) However when I run against an email I get the following error.

It is worth mentioning that I have updated Maltego.

Thank you for your answer,