Comments (17)
mokaddem
commented on July 18, 2024
Hello,
To be sure that your are truly getting the MISP zmq, could you try to run this script
https://github.com/MISP/MISP/blob/2.4/tools/misp-zmq/sub.py
and see if you get an output?
from misp-dashboard.
faustus25
commented on July 18, 2024
All good:
python3 /var/www/MISP/tools/misp-zmq/sub.py
{"status": "I feel FANTASTIC and I'm still alive.", "uptime": 238180}
{"status": "I feel FANTASTIC and I'm still alive.", "uptime": 238180}
{"status": "I feel FANTASTIC and I'm still alive.", "uptime": 238180}
{"status": "I feel FANTASTIC and I'm still alive.", "uptime": 238180}
In the VirtualEnv, redis module is installed:
/usr/local/src/misp-dashboard# . ./DASHENV/bin/activate
(DASHENV) root@misp:/usr/local/src/misp-dashboard# pip3 list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
certifi (2017.11.5)
chardet (3.0.4)
click (6.7)
Flask (0.12.2)
geoip2 (2.6.0)
idna (2.6)
itsdangerous (0.24)
Jinja2 (2.10)
MarkupSafe (1.0)
maxminddb (1.3.0)
pip (9.0.1)
pkg-resources (0.0.0)
pyzmq (16.0.3)
redis (2.10.6)
requests (2.18.4)
setuptools (38.2.1)
urllib3 (1.22)
Werkzeug (0.12.2)
wheel (0.30.0)
zmq (0.0.0)
from misp-dashboard.
mokaddem
commented on July 18, 2024
Can you pull the latest changes? Lots of bugs were corrected in #15 .
Also, what is the version of the publishing MISP? Version 2.4.82 improved ZMQ format.
from misp-dashboard.
faustus25
commented on July 18, 2024
Updated misp-dashboard there and reinstall dependencies and checked for instance of redis:
HEAD is now at cd46dd3 Update README.md
MISP is also on 2.4.82.
Requirement already up-to-date: pip in ./DASHENV/lib/python3.5/site-packages
Requirement already up-to-date: argparse in ./DASHENV/lib/python3.5/site-packages
Requirement already up-to-date: redis in ./DASHENV/lib/python3.5/site-packages
Requirement already up-to-date: zmq in ./DASHENV/lib/python3.5/site-packages
Requirement already up-to-date: geoip2 in ./DASHENV/lib/python3.5/site-packages
Requirement already up-to-date: flask in ./DASHENV/lib/python3.5/site-packages
Collecting phonenumbers
Downloading phonenumbers-8.8.7-py2.py3-none-any.whl (3.0MB)
100% |████████████████████████████████| 3.0MB 299kB/s
Collecting pycountry
Downloading pycountry-17.9.23.tar.gz (9.2MB)
100% |████████████████████████████████| 9.2MB 171kB/s
Requirement already up-to-date: pyzmq in ./DASHENV/lib/python3.5/site-packages (from zmq)
Requirement already up-to-date: requests>=2.9 in ./DASHENV/lib/python3.5/site-packages (from geoip2)
Requirement already up-to-date: maxminddb>=1.2.1 in ./DASHENV/lib/python3.5/site-packages (from geoip2)
Requirement already up-to-date: Werkzeug>=0.7 in ./DASHENV/lib/python3.5/site-packages (from flask)
Requirement already up-to-date: Jinja2>=2.4 in ./DASHENV/lib/python3.5/site-packages (from flask)
Requirement already up-to-date: itsdangerous>=0.21 in ./DASHENV/lib/python3.5/site-packages (from flask)
Requirement already up-to-date: click>=2.0 in ./DASHENV/lib/python3.5/site-packages (from flask)
Requirement already up-to-date: chardet<3.1.0,>=3.0.2 in ./DASHENV/lib/python3.5/site-packages (from requests>=2.9->geoip2)
Requirement already up-to-date: certifi>=2017.4.17 in ./DASHENV/lib/python3.5/site-packages (from requests>=2.9->geoip2)
Requirement already up-to-date: idna<2.7,>=2.5 in ./DASHENV/lib/python3.5/site-packages (from requests>=2.9->geoip2)
Requirement already up-to-date: urllib3<1.23,>=1.21.1 in ./DASHENV/lib/python3.5/site-packages (from requests>=2.9->geoip2)
Requirement already up-to-date: MarkupSafe>=0.23 in ./DASHENV/lib/python3.5/site-packages (from Jinja2>=2.4->flask)
ZMQ_subscriber launches but no Keep Alive Messages as it is running.
While MISPZMQ complains about redis
tail -f /var/www/MISP/app/tmp/logs/mispzmq.error.log
Traceback (most recent call last):
File "/var/www/MISP/app/files/scripts/mispzmq/mispzmq.py", line 3, in <module>
import redis
ImportError: No module named redis
Server.py is listening while ZMQ port is also listening:
tcp 0 0 0.0.0.0:8001 0.0.0.0:* LISTEN 5085/python3.5
tcp 0 0 0.0.0.0:50000 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN -
````
Sub.py runs normally too:
````
python /var/www/MISP/tools/misp-zmq/sub.py
{"uptime": 81180, "status": "I feel FANTASTIC and I'm still alive."}
{"uptime": 81180, "status": "I feel FANTASTIC and I'm still alive."}
{"uptime": 81180, "status": "I feel FANTASTIC and I'm still alive."}
````
Publishing STIX event not pushed as zmq_subscriber not running as expected.
from misp-dashboard.
mokaddem
commented on July 18, 2024
Don't you mean 4c04bd0 instead of cd46dd3? I see that you are fetching the phonenumbers python lib which has been introduced after cd46dd3.
Also, do you have any ideas how you are receiving the feed with sub.py while the publisher appears to have crashed:
File "/var/www/MISP/app/files/scripts/mispzmq/mispzmq.py", line 3, in <module>
import redis
ImportError: No module named redis```
from misp-dashboard.
adulau
commented on July 18, 2024
@faustus25 could you install the redis module in Python 2 and 3? I suppose maybe the redis module is only installed in one version of Python on your system.
from misp-dashboard.
faustus25
commented on July 18, 2024
I check misp-dashboard to see if green icon for misp zmq is there but is not.
tail -f /var/www/MISP/app/tmp/logs/mispzmq.error.log
Traceback (most recent call last):
File "/var/www/MISP/app/files/scripts/mispzmq/mispzmq.py", line 3, in <module>
import redis
ImportError: No module named redis
Could it be an old entry in the log rather than an actual issue?
HEAD is now at 4c04bd0 Merge branch 'doc'
Python environment using:
/usr/local/src/misp-dashboard/DASHENV/bin/python3
Python libraries within DASHENV:
screen is already the newest version (4.3.1-2build1).
redis-server is already the newest version (2:3.0.6-1).
python3-virtualenv is already the newest version (15.0.1+ds-3ubuntu1).
virtualenv is already the newest version (15.0.1+ds-3ubuntu1).
./server.py
Works as expected
./zmq_subscriber.py
Doesn't work as expected (No keep alives message but runs in any case)
./zmq_dispatcher.py
Doesn't work as expected (intermittently runs and then crashes with error below
sending keepalive
sending keepalive
sending keepalive
sending keepalive
sending keepalive
Processed 5 message(s) since last sleep.
Traceback (most recent call last):
File "./zmq_dispatcher.py", line 270, in <module>
main(args.sleeptime)
File "./zmq_dispatcher.py", line 246, in main
process_log(zmqName, content)
File "./zmq_dispatcher.py", line 226, in process_log
jsonevent = json.loads(eventdata)
File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Pip installs for redis in DASHENV:
(DASHENV) :/usr/local/src/misp-dashboard$ pip2 list | grep redis
redis (2.10.6)
(DASHENV) /usr/local/src/misp-dashboard$ pip3.5 list | grep redis
redis (2.10.6)
:/usr/local/src/misp-dashboard$ pip2 list | grep redis
redis (2.10.6)
Pip installs outside DASHENV:
:/usr/local/src/misp-dashboard$ pip3.5 list | grep redis
ipasn-redis (2.0)
redis (2.10.6)
/usr/local/src/misp-dashboard$ pip2 list | grep redis
redis (2.10.6)
So just to review ZMQ in general, I added it to crontab to launch automatically at reboot as it doesn't launch as expected to listen as below:
0.0.0.0:50000
@reboot sudo -u www-data /usr/bin/python3.5 /var/www/MISP/app/files/scripts/mispzmq/mispzmq.py
Tried adding 3 separate bash scripts to /etc/rc.local for zmq server,subscriber and dispatcher scripts to run on reboot/user session too:
#!/bin/sh
cd /usr/local/src/misp-dashboard/
. ./DASHENV/bin/activate
./server.py
#!/bin/sh
cd /usr/local/src/misp-dashboard/
. ./DASHENV/bin/activate
./zmq_subscriber.py
#!/bin/sh
cd /usr/local/src/misp-dashboard/
. ./DASHENV/bin/activate
./zmq_dispatcher.py
Extract from "/etc/rc.local"
sudo -u www-data bash /var/www/MISP/app/Console/worker/start.sh
sudo -u misp /bin/bash /usr/local/bin/misp-dash-server.sh
sudo -u misp /bin/bash /usr/local/bin/misp-dash-zmqs.sh
sudo -u misp /bin/bash /usr/local/bin/misp-dash-zmqd.sh
How have you this setup to run (automatically)?
from misp-dashboard.
faustus25
commented on July 18, 2024
Updated to latest commit, receive the same error when launching "zmq_dispatcher.py":
./zmq_dispatcher.py
Traceback (most recent call last):
File "./zmq_dispatcher.py", line 270, in <module>
main(args.sleeptime)
File "./zmq_dispatcher.py", line 246, in main
process_log(zmqName, content)
File "./zmq_dispatcher.py", line 226, in process_log
jsonevent = json.loads(eventdata)
File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Which value is is missing from decoder.py?
from misp-dashboard.
mokaddem
commented on July 18, 2024
Could you paste the message received before the crash? The variable event would be nice, so that we can see the topic and eventdata.
from misp-dashboard.
faustus25
commented on July 18, 2024
No event data is pushed or other error message before it crashes.
Simply, activating the VirtualEnv for Misp-Dashboard then running "zmq_dispatcher.py" shows the Traceback message above.
cd /usr/local/src/misp-dashboard/
. ./DASHENV/bin/activate
./zmq_dispatcher.py
I run "server.py" and "zmq_dispatcher.py" in separate sessions and they run normally.
from misp-dashboard.
mokaddem
commented on July 18, 2024
If there is no event data, it is not possible to access the function process_log.
Without data, it is difficult to help you. If you could try to add more prints or explore the redis database to see the messages causing the exception, it would be great.
from misp-dashboard.
mokaddem
commented on July 18, 2024
Hi @faustus25 ,
Did you manage to solve this issue?
from misp-dashboard.
faustus25
commented on July 18, 2024
No, I haven't managed to progress this, still the same error as before:
./zmq_dispatcher.py Traceback (most recent call last): File "./zmq_dispatcher.py", line 270, in <module> main(args.sleeptime) File "./zmq_dispatcher.py", line 246, in main process_log(zmqName, content) File "./zmq_dispatcher.py", line 226, in process_log jsonevent = json.loads(eventdata) File "/usr/lib/python3.5/json/__init__.py", line 319, in loads return _default_decoder.decode(s) File "/usr/lib/python3.5/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Can you share how you have setup the misp-dashboard to run on your server correctly?
from misp-dashboard.
mokaddem
commented on July 18, 2024
Hi @faustus25,
We just fixed an issue very similar to yours on both MISP-Dashboard and MISP.
Could you pull (both projects) and try again? Thanks for your patience ;)
from misp-dashboard.
faustus25
commented on July 18, 2024
Hi @mokaddem
Running on the versions on all good apart from one potential issue:
MISP v.2.4.94
MISP-Dashboard v1.0-55-g7816abd
I will close this but before I do maybe you can assist with an ZMQ persistence error to the dashboard.
(The ZMQ active Green Icon disappears after publishing some events and events don't remain on the dashboard)
Should the ZMQ Active Icon remain live on the dashboard at all times?
Components installed and errors below:
Installed pip3 redis
pip3 list | grep redis
ipasn-redis (2.0)
redis (2.10.6)
I tested the new version which works however publishing an event to ZMQ hasn't worked since the initial setup post a reboot.
MISP
ZeroMQ settings…OK
- ZMQ server status is operational.
MISP-Dashboard
Green icon disappers on MISP dashboard for ZMQ after some events first published
tail -f /var/www/MISP/app/tmp/logs/mispzmq.error.log
Listening ports, the Redis listening ports for MISP on port 6379 and MISP Dashboard on port 6250:
netstat -tunapl | grep redis
tcp 0 0 0.0.0.0:6250 0.0.0.0:* LISTEN 943/redis-server *:
tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 1015/redis-server 1
tcp 0 0 127.0.0.1:6379 127.0.0.1:58856 ESTABLISHED 1015/redis-server 1
tcp 0 0 127.0.0.1:6379 127.0.0.1:58880 ESTABLISHED 1015/redis-server 1
tcp 0 0 127.0.0.1:6379 127.0.0.1:58864 ESTABLISHED 1015/redis-server 1
tcp 0 0 127.0.0.1:6379 127.0.0.1:40490 ESTABLISHED 1015/redis-server 1
tcp 0 0 127.0.0.1:6379 127.0.0.1:58872 ESTABLISHED 1015/redis-server 1
tcp 0 0 127.0.0.1:6379 127.0.0.1:40520 ESTABLISHED 1015/redis-server 1
tcp 0 0 127.0.0.1:6379 127.0.0.1:58898 ESTABLISHED 1015/redis-server 1
tcp 0 5 127.0.0.1:6379 127.0.0.1:58806 ESTABLISHED 1015/redis-server 1
tcp 0 0 127.0.0.1:6379 127.0.0.1:58834 ESTABLISHED 1015/redis-server 1
tcp 0 0 127.0.0.1:6379 127.0.0.1:58860 ESTABLISHED 1015/redis-server 1
tcp 0 0 127.0.0.1:6379 127.0.0.1:58798 ESTABLISHED 1015/redis-server 1
tcp 0 0 127.0.0.1:6379 127.0.0.1:40476 ESTABLISHED 1015/redis-server 1
tcp 0 0 127.0.0.1:6379 127.0.0.1:58838 ESTABLISHED 1015/redis-server 1
tcp 0 0 127.0.0.1:6379 127.0.0.1:58868 ESTABLISHED 1015/redis-server 1
tcp 0 0 127.0.0.1:6379 127.0.0.1:58876 ESTABLISHED 1015/redis-server 1
tcp 0 0 127.0.0.1:6379 127.0.0.1:58884 ESTABLISHED 1015/redis-server 1
tcp 0 0 127.0.0.1:6379 127.0.0.1:58902 ESTABLISHED 1015/redis-server 1
tcp6 0 0 :::6250 :::* LISTEN 943/redis-server *:
tcp6 0 0 ::1:6250 ::1:39394 ESTABLISHED 943/redis-server *:
tcp6 0 0 ::1:6250 ::1:39392 ESTABLISHED 943/redis-server *:
tcp6 0 0 ::1:6250 ::1:54882 ESTABLISHED 943/redis-server *:
tcp6 0 0 ::1:6250 ::1:54884 ESTABLISHED 943/redis-server *:
tcp6 0 0 ::1:6250 ::1:39396 ESTABLISHED 943/redis-server *:
tcp6 0 0 ::1:6250 ::1:54886 ESTABLISHED 943/redis-server *:
tcp6 0 0 ::1:6250 ::1:54888 ESTABLISHED 943/redis-server *:
tcp6 0 0 ::1:6250 ::1:39390 ESTABLISHED 943/redis-server *:
tcp6 0 0 ::1:6250 ::1:54880 ESTABLISHED 943/redis-server *:
tcp6 0 0 ::1:6250 ::1:39388 ESTABLISHED 943/redis-server *:
tcp6 0 0 ::1:6250 ::1:39398 ESTABLISHED 943/redis-server *:
Dashboard config.cfg:
[RedisGlobal]
host=localhost
port=6250
misp_web_url = https://192.168.x.x
#misp_web_url = http://localhost
#zmq_url=tcp://192.168.56.50:50000
zmq_url=tcp://localhost:50000
from misp-dashboard.
mokaddem
commented on July 18, 2024
Should the ZMQ Active Icon remain live on the dashboard at all times?
YES
However, we spot an issue about that. It seems that when multiple tabs are open on the dashboard page, some ZMQ messages are not sent to all tabs.
This particular issue will be solved for the next dashboard release.
But, if you have only one tab open, it is indeed a bug... It would be great if you can find the root cause...
Nevertheless, the next release will contain many improvements and better persistence.
from misp-dashboard.
faustus25
commented on July 18, 2024
Closing this as have a new issue with the latest update
from misp-dashboard.
Related Issues (20)
- Too many redirects cause inability to display HOT 3
- Auth with Misp Saml HOT 3
- pycountry module install error HOT 1
- Dashboard not showing results HOT 2
- Do I install this on my misp server or on a different server? (aka is this a standalone tool) HOT 1
- Python Script not working. HOT 4
- Unable to connect to the MISP Dashboard, zmq_Dispatcher crashes after publishing events HOT 3
- Maps do not show HOT 1
- Redis server error HOT 1
- Question on compatibility with MISP docker
- Slight syntax error in diagnostic.py file
- Getting Error in MISP Dashboard Script( install_dependencies.sh)
- Unable to populate the Map with entries HOT 3
- Support request: Don't understand redis port
- Bug: Live dashboard - attribute with tags HOT 3
- Bug: Attribute.category panel does not work as expected HOT 1
- No data in Trendings dashboard HOT 1
- Replace maxmind with GeoOpen
- Malware
- Dispatcher stop working
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from misp-dashboard.