misp / misp-dashboard Goto Github PK

View Code? Open in Web Editor NEW

193.0 37.0 66.0 5.21 MB

A live dashboard for a real-time overview of threat intelligence from MISP instances

License: GNU Affero General Public License v3.0

Shell 1.29% Python 14.39% JavaScript 75.77% HTML 7.61% CSS 0.95%

misp cybersecurity cyber-security threatintel threat-intelligence dashboard

misp-dashboard's Issues

idea: a timeline based visualisation for all attributes referenced by time.

redis.exceptions.BusyLoadingError: Redis is loading the dataset in memory

Redis errors once you execute "start_all.sh".

Can you add some mechanism (idle time) to handle to this exception until the data loading has completed?

Traceback (most recent call last):
  File "/usr/local/src/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/client.py", line 668, in execute_command
    return self.parse_response(connection, command_name, **options)
  File "/usr/local/src/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/client.py", line 680, in parse_response
    response = connection.read_response()
  File "/usr/local/src/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/connection.py", line 624, in read_response
    response = self._parser.read_response()
  File "/usr/local/src/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/connection.py", line 301, in read_response
    raise error
redis.exceptions.BusyLoadingError: Redis is loading the dataset in memory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./zmq_dispatcher.py", line 299, in <module>
    main(args.sleeptime)
  File "./zmq_dispatcher.py", line 264, in main
    content = serv_list.rpop(LISTNAME)
  File "/usr/local/src/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/client.py", line 1377, in rpop
    return self.execute_command('RPOP', name)
  File "/usr/local/src/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/client.py", line 674, in execute_command
    return self.parse_response(connection, command_name, **options)
  File "/usr/local/src/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/client.py", line 680, in parse_response
    response = connection.read_response()
  File "/usr/local/src/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/connection.py", line 624, in read_response
    response = self._parser.read_response()
  File "/usr/local/src/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/connection.py", line 301, in read_response
    raise error
redis.exceptions.BusyLoadingError: Redis is loading the dataset in memory

Bug dispatcher

Starting zmq-dispatcher
Traceback (most recent call last):
File "./zmq_dispatcher.py", line 282, in
main(args.sleeptime)
File "./zmq_dispatcher.py", line 258, in main
process_log(zmqName, content)
File "./zmq_dispatcher.py", line 240, in process_log
dico_action[topic](zmq_name, jsonevent)
File "./zmq_dispatcher.py", line 214, in handler_attribute
geo_helper.getCoordFromIpAndPublish(jsonattr['value'], jsonattr['category'])
File "/home/ail/git/misp-dashboard/helpers/geo_helper.py", line 106, in getCoordFromIpAndPublish
self.push_to_redis_geo(self.keyCategRad, coord['lon'], coord['lat'], json.dumps(ordDic))
File "/home/ail/git/misp-dashboard/helpers/geo_helper.py", line 166, in push_to_redis_geo
self.serv_redis_db.geoadd(keyname, lon, lat, content)
File "/home/ail/git/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/client.py", line 2121, in geoadd
return self.execute_command('GEOADD', name, *values)
File "/home/ail/git/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/client.py", line 668, in execute_command
return self.parse_response(connection, command_name, **options)
File "/home/ail/git/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/client.py", line 680, in parse_response
response = connection.read_response()
File "/home/ail/git/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/connection.py", line 629, in read_response
raise response
redis.exceptions.ResponseError: invalid longitude,latitude pair 0.000000,-90.000000

Geolocalise all the things

Geolocalise all the things:

Phone numbers (based on phone prefix)
AS number (via whois)
domain or hostname (via IP)
Bank account (based on bank registration location)

Trendings: Showing only specified taxonomies

Live dashboard can't be opened on multiple screen

It seems that when multiple windows of the live dashboard are open, the eventStream is sending data one to another instead of one per session.

Show event tags in live dashboard

It will be nice to have the event tags appearing in live dashboard, same that what happens for attribute tag.
Basically it would be nice to have a page with live events sentiment using misp fields present in the event page, like Published | Org | Owner Org | Id | Clusters | Tags | #Attr. | Email | Date | Info | Distribution
And have an events trending and not an attributes trending.
And also an attribute type trending and stats it will be appreciated, in order to view the trend of url/domain/ip/md5/sha1 etc.

Filtering rules on live-logs

Provides the possibility to users to add filtering rules like: I only want logs concerning event 42 or I only want logs about attributes/objects/events that contain the tag TLP:RED

Change the name of the gamification project

quality rating
Contribution quality
...

Trigger onchange when typing

Add support of MISP-Object in live logs

MISP-Object are not supported yet.

No disk space due to excessive correlations and large tmp misp-dashboard files

Encounter MISP blackhole requests due to insufficient disk space and the only way to remedy it is to
truncate correlation in mysql to free up disk space:

Details below:

 tail -f /var/www/MISP/app/tmp/logs/error.log
#7 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke(Object(UsersController), Object(CakeRequest))
#8 /var/www/MISP/app/webroot/index.php(92): Dispatcher->dispatch(Object(CakeRequest), Object(CakeResponse))
#9 {main}
Warning: Warning (2): session_write_close(): write failed: No space left on device (28) in [Unknown, line 0]
Trace:
ErrorHandler::handleError() - APP/Lib/cakephp/lib/Cake/Error/ErrorHandler.php, line 230
session_write_close - [internal], line ??
[main] - [internal], line ??

 Warning: Warning (2): session_write_close() [<a href='http://php.net/function.session-write-close'>function.session-write-close</a>]: Failed to write session data (files). Please veri

Further details:

/usr/local/src/misp-dashboard/data/temp-5530.rdb: 2.2G
/usr/local/src/misp-dashboard/data/temp-1404.rdb: 455M
/usr/local/src/misp-dashboard/data/temp-1286.rdb: 326M
/usr/local/src/misp-dashboard/data/temp-1612.rdb: 549M
/usr/local/src/misp-dashboard/data/temp-1582.rdb: 439M
/usr/local/src/misp-dashboard/data/temp-9849.rdb: 2.0G
/usr/local/src/misp-dashboard/data/temp-2020.rdb: 329M
/usr/local/src/misp-dashboard/data/temp-1405.rdb: 407M
/usr/local/src/misp-dashboard/data/dump.rdb: 2.1G
/usr/local/src/misp-dashboard/data/temp-1415.rdb: 436M
/usr/local/src/misp-dashboard/data/temp-1412.rdb: 350M
/usr/local/src/misp-dashboard/data/temp-1435.rdb: 420M
/var/log/lastlog: 264M
/var/log/syslog.1: 107M
/var/lib/redis/dump.rdb: 144M
/var/lib/mysql/misp/logs.ibd: 4.3G
/var/lib/mysql/misp/attributes.ibd: 1.2G
**/var/lib/mysql/misp/correlations.ibd: 72G**
/var/lib/mysql/ibdata1: 268M
/var/www/misp-dashboard/data/dump.rdb: 576M
/var/www/MISP/app/tmp/logs/resque-worker-error.log: 126M

Logging into MYSQL to delete all correlations resolves until the correlations build up again

Resque error log:

main.ERROR: {"queue":"default","id":"a7780ba1ac4f3faeca7604712f6e3511","class":"ServerShell","args":[{"0":"enqueuePull","1":1544719200,"2":"2","3":"2","s_time":1544697602}]} failed: SQLSTATE[HY000]: General error: 1114 The table 'correlations' is full {"type":"fail","log":"SQLSTATE[HY000]: General error: 1114 The table 'correlations' is full","job_id":"a7780ba1ac4f3faeca7604712f6e3511","time":2132390,"worker":"server:6778"} []

Make even more basic badges to influence participation positively

Currently the current most basic badge is one person that does a sighting.

You could have even "lower" badges that would be super positive to encourage people who are typically shy or frightened or mystified by these Threat Intelligence system, but who would contribute something critical or something at critical times.

So here are a few badges suggestions (from most Basic to most Advanced, but remember it's very basic level below the "sighter" ensign badge):

Visitor (logs once a year)
User (longs once a month)
Regular (logs once a week)
Regular Pro (logs 3 times a week)
Addict (logs once a day)
...

and importantly, send them a mail if they're going to loose their Visitor or User badge (not other ones, that would be improductive flooding).

the goal is to have stickiness with users that you WILL depend on when in the red zone (i.e. emergency, critical event, ...) or who can be contributing rare but key information / files / hashes / events / attributes.

No MISP Standard ZMQ Green Icon on MISP Dashboard

Still having issues running ZMQ on MISP dashboard to publish events from MISP to the MISP-Dashboard.
Each update has either ran ZMQ temporarily or not at all i.e. the Standard ZMQ Icon is not present on the default MISP-dashboard landing page.

git describe --tags
v1.1-2-gce26610

ZMQ is active and running on MISP

ZeroMQ Server Status

Start time: 2018/10/12 11:33:30
Settings read at: 2018/10/12 10:49:24
Events processed: 13

ZMQ Listening:

0.0.0.0:50000 2473/python3

Redis-server listening on port 6250 TCP

0.0.0.0:6250 1405/redis-server
127.0.0.1:6250 1405/redis-server
127.0.0.1:6250 1405/redis-server
127.0.0.1:6250 1405/redis-server
127.0.0.1:6250 1405/redis-server
127.0.0.1:6250 1405/redis-server
127.0.0.1:6250 1405/redis-server
127.0.0.1:6250 1405/redis-server
127.0.0.1:6250 1405/redis-server
127.0.0.1:6250 1405/redis-server
127.0.0.1:6250 1405/redis-server
127.0.0.1:6250 1405/redis-server
127.0.0.1:6250 1405/redis-server
127.0.0.1:6250 1405/redis-server
127.0.0.1:6250 1405/redis-server
127.0.0.1:6250 1405/redis-server
127.0.0.1:6250 1405/redis-server
127.0.0.1:6250 1405/redis-server
127.0.0.1:6250 1405/redis-server
127.0.0.1:6250 1405/redis-server
127.0.0.1:6250 1405/redis-server
:::6250 1405/redis-server

How can I check to ensure the ZMQ is running correctly for the MISP-Dashboard?

Log absolute path is required if misp-dashboard path is different from default

Hello,

please add to the README file a notice that informs an absolute path is required if MISP dashboard's path is different from default /var/www/misp-dashboard. Indeed, copying MISP dashboard into /var/www/MISP/misp-dashboard, if the config variable is set to directory=logs, then apache warns it has not the right permissions to write (likely it does not find the correct log file). Changing the variable like this directory=/var/www/MISP/misp-dashboard/logs, then it works.

Furthermore, please add a notice about start_all.sh since I found it pretty useful.

Thanks,
Francesco

Export PDF of dashboard

Add an API for MISP-dashboard

Be able to query the dashboard for the data

Threat level interface

Page showing the threat level similar to a forecast page:

Today everything is OK
Today rise of threat X

WSGI Apache Reload Error, Need Correct Format for Apache Conf file

Reinstall the dashboard according to the new MISP install guide:

https://misp.github.io/MISP/INSTALL.ubuntu1804/#misp-dashboard

Reloading the apache service fails:

 Reloading LSB: Apache2 web server.
apache2[5786]:  * Reloading Apache httpd web server apache2
apache2[5786]:  *
apache2[5786]:  * The apache2 configtest failed. Not doing anything.
apache2[5786]: Output of config test was:
apache2[5786]: AH00543: apache2: bad user name misp
apache2[5786]: Action 'configtest' failed.

Output of misp-dashboard.conf for Apache2:

<VirtualHost *:8001>
    ServerAdmin admin@admin
    ServerName misp.local
    DocumentRoot /var/www/misp-dashboard

    WSGIDaemonProcess misp-dashboard
        user=misp group=misp
        python-home=/var/www/misp-dashboard/DASHENV
        processes=1
        threads=15
        maximum-requests=5000
        listen-backlog=100
        queue-timeout=45
        socket-timeout=60
        connect-timeout=15
        request-timeout=60
        inactivity-timeout=0
        deadlock-timeout=60
        graceful-timeout=15
        eviction-timeout=0
        shutdown-timeout=5
        send-buffer-size=0
        receive-buffer-size=0
        header-buffer-size=0
        response-buffer-size=0
        server-metrics=Off
    WSGIScriptAlias / /var/www/misp-dashboard/misp-dashboard.wsgi
    <Directory /var/www/misp-dashboard>
        WSGIProcessGroup misp-dashboard
        WSGIApplicationGroup %{GLOBAL}
        Require all granted
    </Directory>
    LogLevel info
    ErrorLog /var/log/apache2/misp-dashboard.local_error.log
    CustomLog /var/log/apache2/misp-dashboard.local_access.log combined
    ServerSignature Off
</VirtualHost>

Permissisons as follows:

/var/www/misp-dashboard# ls -al
total 172
drwxr-xr-x 12 www-data www-data  4096 Nov  2 17:04 .
drwxr-xr-x  5 root     root      4096 Nov  2 17:03 ..
-rwxr-xr-x  1 www-data www-data  2573 Nov  2 17:04 clean.py
drwxr-xr-x  2 www-data www-data  4096 Nov  2 17:04 config
drwxr-xr-x  6 root     root      4096 Nov  2 17:04 DASHENV
drwxr-xr-x  3 www-data www-data  4096 Nov  2 17:04 data
drwxr-xr-x  2 www-data www-data  4096 Nov  2 17:04 doc
drwxr-xr-x  8 www-data www-data  4096 Nov  2 17:04 .git
-rwxr-xr-x  1 www-data www-data  7641 Nov  2 17:04 give_honors_to_org.py
drwxr-xr-x  2 www-data www-data  4096 Nov  2 17:04 helpers
-rwxr-xr-x  1 www-data www-data  5339 Nov  2 17:04 install_dependencies.sh
-rw-r--r--  1 www-data www-data 34520 Nov  2 17:04 LICENSE
-rw-r--r--  1 www-data www-data   185 Nov  2 17:04 misp-dashboard.wsgi
-rw-r--r--  1 www-data www-data  9243 Nov  2 17:04 README.md
-rwxr-xr-x  1 www-data www-data  2148 Nov  2 17:04 retreive_map_pic.py
drwxr-xr-x  2 www-data www-data  4096 Nov  2 17:04 screenshots
-rwxr-xr-x  1 www-data www-data 20710 Nov  2 17:04 server.py
-rwxr-xr-x  1 www-data www-data  2482 Nov  2 17:04 start_all.sh
drwxr-xr-x  6 www-data www-data  4096 Nov  2 17:04 static
drwxr-xr-x  2 www-data www-data  4096 Nov  2 17:04 templates
drwxr-xr-x  2 www-data www-data  4096 Nov  2 17:04 tests
-rw-r--r--  1 www-data www-data  3341 Nov  2 17:04 util.py
-rwxr-xr-x  1 www-data www-data  9952 Nov  2 17:04 zmq_dispatcher.py
-rwxr-xr-x  1 www-data www-data  1978 Nov  2 17:04 zmq_subscriber.py

What is the appropriate user if "misp" or "www-data" don't work?

user=misp group=misp

Originally posted by @faustus25 in #71 (comment)

MISP Monitoring page

A page showing the status/health of MISP, where the data comes from MISP/MISP#3235

Prevent the purging of the data from the live dashboard

Switching to a different view seems to purge the previous data from the live dashboard view. Could we make it persistent for the duration of the session / some other mechanism?

Filtering on trendings

System Boot

HI people,

My question is that after i make system run on apache is possible to start zqm_subscrive and dispatcher at boot system.

Thanks.

Improving ranking by user voting

Interesting feedback from Florian Roth:

We should investigate the voting per user but the original issue of signing is back...

Error spotted once

[2017-12-04 10:28:59,005] ERROR in app: Exception on /_getOrgRank [GET]
Traceback (most recent call last):
  File "/home/user/git/MISP-Dashboard/DASHENV/lib/python3.5/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/user/git/MISP-Dashboard/DASHENV/lib/python3.5/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/user/git/MISP-Dashboard/DASHENV/lib/python3.5/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/user/git/MISP-Dashboard/DASHENV/lib/python3.5/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/home/user/git/MISP-Dashboard/DASHENV/lib/python3.5/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/user/git/MISP-Dashboard/DASHENV/lib/python3.5/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "./server.py", line 387, in getOrgRank
    return jsonify(contributor_helper.getCurrentOrgRankFromRedis(org))
  File "/home/user/git/MISP-Dashboard/contributor_helper.py", line 523, in getCurrentOrgRankFromRedis
    remainingPts = self.getRemainingPoints(points)
  File "/home/user/git/MISP-Dashboard/contributor_helper.py", line 548, in getRemainingPoints
    for i in [math.floor(self.rankMultiplier**x) for x in range(1,self.levelMax+1)]:
TypeError: 'float' object cannot be interpreted as an integer

ZMQ Subscriber Error: JSONDecodeError("Expecting value")

Tried to publish events (STIX imported and also from Feeds) to MISP-Dashboad but it fails as zmq_subscriber.py crashes with error below:

Traceback (most recent call last):
  File "./zmq_subscriber.py", line 329, in <module>
    main(args.zmqname)
  File "./zmq_subscriber.py", line 306, in main
    process_log(zmq_name, content)
  File "./zmq_subscriber.py", line 291, in process_log
    jsonevent = json.loads(eventdata)
  File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Error log points to redis module not installed (which it is):

tail -f /var/www/MISP/app/tmp/logs/mispzmq.error.log
Traceback (most recent call last):
  File "/var/www/MISP/app/files/scripts/mispzmq/mispzmq.py", line 3, in <module>
    import redis
ImportError: No module named redis

Module installed:

pip3 list | grep "redis"
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
ipasn-redis (2.0)
redis (2.10.6)

Running it as:

. ./DASHENV/bin/activate
./zmq_subscriber.py
&&
. ./DASHENV/bin/activate
./server.py

Any guidance on correcting this?

Ranking on individuals

zmq_subscriber.py crashes repeatedly

I am receiving the following error when I try and publish some events to ZMQ:

AttributeError: 'StricRedis' object has no attribute 'geoadd'.

This causes the subscriber to terminate. The error appears to occur on any event containing an IP address. Additionally I am getting messages that the subscriber is unable to resolve domains to IPs.

Kind regards
Andy Dove

MISP: Error importError cannot import name “ExpandedMISP”

I am receiving the following error below. Much appreciated in advance.

ERROR: Failure: ImportError (cannot import name 'ExpandedPyMISP') ImportError: cannot import name "ExpandedPyMISP

Zoom buttons have too high z-index and overlap with the menu on the live dashboard

An Error on request:

hello, happen to see a error from server.py:

127.0.0.1 - - [06/Nov/2017 05:24:52] "GET /_logs HTTP/1.1" 500 -
Error on request:
Traceback (most recent call last):
File "/home/jessie/misp/misp-dashboard/DASHENV/lib/python3.5/site-packages/werkzeug/serving.py", line 209, in run_wsgi
execute(self.server.app)
File "/home/jessie/misp/misp-dashboard/DASHENV/lib/python3.5/site-packages/werkzeug/serving.py", line 199, in execute
for data in application_iter:
File "/home/jessie/misp/misp-dashboard/DASHENV/lib/python3.5/site-packages/werkzeug/wsgi.py", line 704, in next
return self._next()
File "/home/jessie/misp/misp-dashboard/DASHENV/lib/python3.5/site-packages/werkzeug/wrappers.py", line 81, in _iter_encoded
for item in iterable:
File "/home/jessie/misp/misp-dashboard/server.py", line 218, in event_stream_log
for msg in subscriber_log.listen():
File "/home/jessie/misp/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/client.py", line 2501, in listen
response = self.handle_message(self.parse_response(block=True))
File "/home/jessie/misp/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/client.py", line 2430, in parse_response
return self._execute(connection, connection.read_response)
File "/home/jessie/misp/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/client.py", line 2408, in _execute
return command(*args)
File "/home/jessie/misp/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/connection.py", line 624, in read_response
response = self._parser.read_response()
File "/home/jessie/misp/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/connection.py", line 284, in read_response
response = self._buffer.readline()
AttributeError: 'NoneType' object has no attribute 'readline'

Kind regards
Jessie Lee

Have a timeline of events/data based on sightings per galaxy/tag

zmq_dispatcher errors: value is not a valid float

Hi,

Dashboards appear to work fine, but when publishing event to ZMQ I get this error:

File "./zmq_dispatcher.py", line 299, in
main(args.sleeptime)
File "./zmq_dispatcher.py", line 274, in main
process_log(zmqName, content)
File "./zmq_dispatcher.py", line 256, in process_log
dico_action[topic](zmq_name, jsonevent)
File "./zmq_dispatcher.py", line 101, in handler_dispatcher
handler_event(zmq_name, jsonObj)
File "./zmq_dispatcher.py", line 168, in handler_event
trendings_helper.addTrendingEvent(eventName, timestamp)
File "/var/www/MISP/tools/misp-dashboard/helpers/trendings_helper.py", line 47, in addTrendingEvent
self.addGenericTrending(self.keyEvent, eventName, timestamp)
File "/var/www/MISP/tools/misp-dashboard/helpers/trendings_helper.py", line 43, in addGenericTrending
self.serv_redis_db.zincrby(keyname, to_save, 1)
File "/var/www/MISP/tools/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/client.py", line 2281, in zincrby
return self.execute_command('ZINCRBY', name, amount, value)
File "/var/www/MISP/tools/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/client.py", line 755, in execute_command
return self.parse_response(connection, command_name, **options)
File "/var/www/MISP/tools/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/client.py", line 768, in parse_response
response = connection.read_response()
File "/var/www/MISP/tools/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/connection.py", line 638, in read_response
raise response
redis.exceptions.ResponseError: value is not a valid float

Could anybody assist?

zmq_dispatcher errors

I people

after a install the system and start I have thies error:

ile "/var/www/misp-dashboard/zmq_dispatcher.py", line 282, in
main(args.sleeptime)
File "/var/www/misp-dashboard/zmq_dispatcher.py", line 258, in main
process_log(zmqName, content)
File "/var/www/misp-dashboard/zmq_dispatcher.py", line 240, in process_log
dico_action[topic](zmq_name, jsonevent)
File "/var/www/misp-dashboard/zmq_dispatcher.py", line 214, in handler_attribute
geo_helper.getCoordFromIpAndPublish(jsonattr['value'], jsonattr['category'])
File "/var/www/misp-dashboard/helpers/geo_helper.py", line 106, in getCoordFromIpAndPublish
self.push_to_redis_geo(self.keyCategRad, coord['lon'], coord['lat'], json.dumps(ordDic))
File "/var/www/misp-dashboard/helpers/geo_helper.py", line 166, in push_to_redis_geo
self.serv_redis_db.geoadd(keyname, lon, lat, content)
File "/var/www/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/client.py", line 2121, in geoadd
return self.execute_command('GEOADD', name, *values)
File "/var/www/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/client.py", line 668, in execute_command
return self.parse_response(connection, command_name, **options)
File "/var/www/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/client.py", line 680, in parse_response
response = connection.read_response()
File "/var/www/misp-dashboard/DASHENV/lib/python3.5/site-packages/redis/connection.py", line 629, in read_response
raise response
redis.exceptions.ResponseError: unknown command 'GEOADD'

How to start Misp-dashboard on version 2.4.100

Web server will not start

When I run server.py it hangs and does nothing.

If I try as sudo I get:
Traceback (most recent call last):
File "./zmq_subscriber.py", line 13, in
configfile = os.path.join(os.environ['DASH_CONFIG'], 'config.cfg')
File "/usr/lib/python3.5/os.py", line 725, in getitem
raise KeyError(key) from None
KeyError: 'DASH_CONFIG'

Using Ubuntu Server 16. Also have to point to a config file when starting redis and have to be in the virtual environment to it to see the .data directory. Instructions do not state this.

Geo: Click on popup redirect to concerned event in MISP

geolocation of phone numbers based on prefixes (phone-number)

geolocation of phone numbers based on prefixes

Title page should include the name of the page instead of the generic title

No option 'streamlogcachekey' in section: 'RedisLog'

Updated to latest version today:

git describe --tags
v1.1-2-gce26610

Issue once I launch the startup scripts:

./start_all.sh
dashboard virtualenv seems to exist, good
tcp        0      0 0.0.0.0:6250            0.0.0.0:*               LISTEN
root      1615  0.4  0.1 213960 60420 ?        Sl    /usr/local/src/misp-dashboard/DASHENV/bin/python ./zmq_subscriber.py
        * NOT starting Redis server, made a very unrealiable check on port 6250, and something seems to be thereâ¦ please double check if this is good!
        * NOT starting zmq subscriber, made a rather unrealiable ps -auxw | grep for zmq_subscriber.py, and something seems to be thereâ¦ please double check if this is good!
        * Launching zmq dispatcher
        * Launching flask server
user@server:/usr/local/src/misp-dashboard# Traceback (most recent call last):
  File "/usr/lib/python3.5/configparser.py", line 786, in get
    value = d[option]
  File "/usr/local/src/misp-dashboard/DASHENV/lib/python3.5/collections/__init__.py", line 878, in __getitem__
    return self.__missing__(key)            # support subclasses that define __missing__
  File "/usr/local/src/misp-dashboard/DASHENV/lib/python3.5/collections/__init__.py", line 870, in __missing__
    raise KeyError(key)
KeyError: 'maxcachehistory'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./zmq_dispatcher.py", line 49, in <module>
    live_helper = live_helper.Live_helper(serv_redis_db, cfg)
  File "/usr/local/src/misp-dashboard/helpers/live_helper.py", line 12, in __init__
    self.maxCacheHistory = cfg.get('Dashboard', 'maxCacheHistory')
  File "/usr/lib/python3.5/configparser.py", line 789, in get
    raise NoOptionError(option, section)
configparser.NoOptionError: No option 'maxcachehistory' in section: 'Dashboard'
Traceback (most recent call last):
  File "/usr/lib/python3.5/configparser.py", line 786, in get
    value = d[option]
  File "/usr/local/src/misp-dashboard/DASHENV/lib/python3.5/collections/__init__.py", line 878, in __getitem__
    return self.__missing__(key)            # support subclasses that define __missing__
  File "/usr/local/src/misp-dashboard/DASHENV/lib/python3.5/collections/__init__.py", line 870, in __missing__
    raise KeyError(key)
KeyError: 'streamlogcachekey'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./server.py", line 45, in <module>
    streamLogCacheKey = cfg.get('RedisLog', 'streamLogCacheKey')
  File "/usr/lib/python3.5/configparser.py", line 789, in get
    raise NoOptionError(option, section)
configparser.NoOptionError: No option 'streamlogcachekey' in section: 'RedisLog'

TLP White is invisble on trending page

If you can see the attached image, anything is the trendings page of the dashboard, if white in color, is not visible due to the lack of border on the graphs.

Exclude events from being published to dashboard

Hi all,

Is there any way to exclude certain events from being sent to the dashboard? We have some feeds with 100,000+attributes, which keep the dashboard pretty busy and make the statistics somewhat confusing.

ZMQ Dispatcher Error: JSONDecodeError("Expecting value")

Had the same issue with the zmq_subscriber script now the zmq_dispatcher.py one has this issue:

echo 'source /usr/local/src/misp-dashboard/DASHENV/bin/activate; /usr/bin/python3 /usr/local/src/misp-dashboard/zmq_dispatcher.py' | /bin/bash

Traceback (most recent call last):
  File "/usr/local/src/misp-dashboard/zmq_dispatcher.py", line 299, in <module>
    main(args.sleeptime)
  File "/usr/local/src/misp-dashboard/zmq_dispatcher.py", line 274, in main
    process_log(zmqName, content)
  File "/usr/local/src/misp-dashboard/zmq_dispatcher.py", line 254, in process_log
    jsonevent = json.loads(eventdata)
  File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

How are you running the "start_all.sh" script as virtualenv when passed to crontab at reboot or rc.local?

Consuming logging to perform statistics from the ZMQ audit

Consuming logging to perform statistics from the ZMQ audit + localized user based on paranoid logging output

float() argument must be a string or a number, not 'NoneType'

This happened when getting a feed. It worked for 10+ minutes, then hung on the following. Kinda hard to reconstruct. Sounds like an attribute with empty value? How would you troubleshoot this?

can't resolve ip
sending keepalive
Traceback (most recent call last):
  File "zmq_dispatcher.py", line 270, in <module>
    main(args.sleeptime)
  File "zmq_dispatcher.py", line 246, in main
    process_log(zmqName, content)
  File "zmq_dispatcher.py", line 228, in process_log
    dico_action[topic](zmq_name, jsonevent)
  File "zmq_dispatcher.py", line 76, in handler_dispatcher
    handler_event(zmq_name, jsonObj)
  File "zmq_dispatcher.py", line 157, in handler_event
    handler_attribute(zmq_name, jsoncopy)
  File "zmq_dispatcher.py", line 202, in handler_attribute
    geo_helper.getCoordFromIpAndPublish(jsonattr['value'], jsonattr['category'])
  File "/var/www/misp-dashboard/geo_helper.py", line 82, in getCoordFromIpAndPublish
    rep = self.ip_to_coord(supposed_ip)
  File "/var/www/misp-dashboard/geo_helper.py", line 162, in ip_to_coord
    lat = float(resp.location.latitude)
TypeError: float() argument must be a string or a number, not 'NoneType'

show Event tags

It seems that at the moment only attribute tags are shown.
It will be useful to show also the event tag (I think almost all users are tagging the event and not the single attribute).
I tried to modify the config file row in this way:
fieldname_order=["Event.id", "Event.Tag", "Attribute.Tag", "Attribute.category", "Attribute.type", ["Attribute.value", "Attribute.comment"]]
It doesn't seem to help (at least now I can see event tag under Most popular tags, but in dashboard the new column event.tag is not being populated)

misp / misp-dashboard Goto Github PK

misp-dashboard's Issues

Recommend Projects

Recommend Topics

Recommend Org