No disk space due to excessive correlations and large tmp misp-dashboard files about misp-dashboard HOT 4 CLOSED

Sounds like you have probably ingested the same feed over and over into new events. To resolve it:

1. Truncate the correlations table
2. modify your csv/freetext feed settings to always use fixed events, not "new event each pull"
3. Delete all the duplicate events coming from the feeds (they will have the same info fields, one per feed)
4. recorrelate the database (server settings -> diagnostics -> legacy tools -> recorrelate attributes)

This should get you up and running again. Let us know if it doesn't resolve it.

from misp-dashboard.

Truncating the correlations table works but I need to delete the high volume attribute Fixed Events for certain events.

Part of the problem is the fact, that Warning List attributes are duplicated in multiple events i.e. "8.8.8.8" and should be ideally mass deleted. The second is that the MISP ZMQ filters through each single attribute for those high volume attribute Fixed Events which is time consuming and eventually the ZMQ listener stops.

from misp-dashboard.

I have remove feeds with excessive attributes but ideally an option to mass delete duplicate false positive attributes would help.

Disk space for misp-dashboard:

/usr/local/src/misp-dashboard/data ------------------------------------------------------------------------------------------------------------------------------------------
                         /..
    2.1 GiB [##########]  temp-5530.rdb
    2.1 GiB [######### ]  dump.rdb
    1.9 GiB [######### ]  temp-9849.rdb
  548.4 MiB [##        ]  temp-1612.rdb
  454.7 MiB [##        ]  temp-1404.rdb
  442.2 MiB [##        ]  temp-1339.rdb
  438.6 MiB [##        ]  temp-1582.rdb
  438.4 MiB [##        ]  temp-1326.rdb
  436.1 MiB [##        ]  temp-1455.rdb
  435.4 MiB [##        ]  temp-1415.rdb
  419.2 MiB [#         ]  temp-1435.rdb
  406.2 MiB [#         ]  temp-1405.rdb
  406.2 MiB [#         ]  temp-1492.rdb
  374.5 MiB [#         ]  temp-1342.rdb
  349.9 MiB [#         ]  temp-1412.rdb
  328.7 MiB [#         ]  temp-2020.rdb
  325.5 MiB [#         ]  temp-1286.rdb
   63.2 MiB [          ] /GeoLite2-City_20180807
   20.0 KiB [          ]  country_code_lat_long.json
@   0.0   B [          ]  GeoLite2-City

Is it recommended to delete all the temp.rdb files?

Also how do you get the ZMQ listener to move onto the next event published? The dashboard is stuck on one event that has an excessive amount of attributes and can't progress to the next one.

from misp-dashboard.

Indeed, if you publish massive events to the ZMQ channel, it will generate a lot of data.
As of now, you can't skip event that are being processed.

from misp-dashboard.