how to get at the payload data using idstools-u2json about py-idstools HOT 5 CLOSED

I'm trying to get at what analysts call "payload" data using idstools-u2json. I tried using the master branch with the additional params of --packets --extra-data, and I never see anything in extra-data field, but I do see what looks like binary data in the 'packet'(s) field.

When using Barnyard2 it places the payload data, if any, in the mysql/db table called 'data', which is represented in the column 'data_payload' as a hex string ... it would be nice to store this as both
hex and ascii in elasticsearch.

Packets is an array, where each packet is encoded as base64 so no information is lost. It should be trivial enough to turn this back into binary data and then extract the printable characters. This is why my EveBox does with Javascript (pulls the base64 from ES then represents the data as a printable). Alternatively, idstools-u2json could be just used as example code for a new tool that outputs the base64, or hex, or printable or a combination of any of the above.

You have have to look harder for an event with extra data in it. Not many events have it. And when they do, I'm still running into the occasional issue. At this time I'm not sure whether I should base64 encode that data, strip non-printable characters or what.

Also, I can't find the sensor(sid), hostname, or interface fields ... as sometimes our snort events are > gathered from multiple sensors.

I've added sensor_id to master, it makes sense as its part of the unified2 alert.

The hostname and interface fields are features of barnyard. Again, if using idstools-u2json as an example application, then you could add them yourself, or use something like Logstash to post process the data.

Something to keep in mind, as its not too clear, is that the u2json output more or less aims to be compatible with Suricata's eve output, so you can feed Suricata and Snort events to Logstash/ElasticSearch and view the events together in something like Kibana or EveBox.

from py-idstools.

Thanks for the info ... I was able to add this to u2json.py:

import binascii
def filter_non_printable(str):
  return ''.join([c if ord(c) > 31 and ord(c) < 127 else '' for c in str])

... and in:

class SuricataJsonFilter(object):
...
                encoded_packet_data = base64.b64encode(packet["data"])
                try:
                    decoded_packet_data = binascii.b2a_hex(packet["data"])
                except Exception, e:
                    decoded_packet_data = ''
                    print('decode failed!')
                output_packet = {
                    "timestamp": render_timestamp(
                        packet["packet-second"], packet["packet-microsecond"]),
                    "sensor-id": packet["sensor-id"],
                    "length": packet["length"],
                    "packet_hex": decoded_packet_data,
                    "payload": filter_non_printable(decoded_packet_data.decode('hex'))
                }

... there's probably a better/faster way to do this, but so far it works with all of the
unified2 logs I've tried. We do see some payload (packet) data, not a lot as you
say, but searching within payload data in elasticsearch was a requested feature.
Yes, everyone here seems to love Kibana, but it doesn't integrate very well with
existing rails app's (auth/login for example). I will have to check out EveBox.
Thanks, again.

from py-idstools.

uspewfoo.py, an implementation of Snort's own u2spewfoo using idstools has code to extract the printable bytes from a packet to output something like:

[   64] b5 18 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f   ..HTTP/1.1 200 O
[   80] 4b 0d 0a 44 61 74 65 3a 20 4d 6f 6e 2c 20 31 35   K..Date: Mon, 15
[   96] 20 4a 75 6c 20 32 30 31 33 20 32 31 3a 34 39 3a    Jul 2013 21:49:
[  112] 31 39 20 47 4d 54 0d 0a 53 65 72 76 65 72 3a 20   19 GMT..Server: 
[  128] 41 70 61 63 68 65 2f 32 2e 32 2e 31 35 20 28 43   Apache/2.2.15 (C
[  144] 65 6e 74 4f 53 29 0d 0a 4c 61 73 74 2d 4d 6f 64   entOS)..Last-Mod
[  160] 69 66 69 65 64 3a 20 54 68 75 2c 20 32 37 20 4a   ified: Thu, 27 J

But its not very generic.

Yeah, Kibana is nice, but its lack of authentication and authorization make it a no-go for some.

from py-idstools.

I have the exact same layout of payload data in our rails app's ... I think I got that code from snorby.
Which the users like, but they insisted on making it a searchable ascii field in elasticsearch, so the
payload/packet data has to be put into elasticsearch in both hex (not so searchable) and ascii (searchable). I will check out uspewfoo.py. I always struggle with these snort apps and some of the odd stuff going on, or rather lesser encountered coding required. Thanks again, and good job on idstools.

from py-idstools.

@curtbraz, I don't have a u2json.py script (my comments above are a reference to that file in idstools),
but rather I have code similar to that in:
https://github.com/cleesmith/uni2espy
... which uses idstools to tail and index unified2 files into elasticsearch
... so they can be shown/searched using EveBox, Rorshack, or Kibana
Hope that helps.

from py-idstools.