Create JSON emitter to send on to ELK etc. about libipfix HOT 3 CLOSED

This would be a better use-case for the 'files' output (not the database-backed collector). The only benefit to using the database-backed connector is that we would drop data messages post restart (when we've effectively forgotten what templates we had previously learned about).

Perhaps it would be long-term best to create a persistence mechanism for learned templates. Perhaps writing them as JSON files (one file for each template, with the filename based on the enterprise number and attribute number). Long-term, it would best to divorce the PEN IEs from the code and put that into a data-file, so you can add more without modifying the code.

Short-term, it would be easiest to use the database collector, but only use the database for the templates, and have data updates streamed out via JSON. I think that works nicely, and doesn't preclude storing the templates otherwise in the future, so that is what I shall do.

Note that the output will be one JSON document per line, so the entire file will not be valid JSON. In logstash terms, this is 'jsonlines' input.

• Add an argument --data-jsonlines=FILENAME (5ab7a4f)
• Create a function in lib/ipfix_col_db.c called ipfix_export_drecord_jsonfile similar to ipfix_export_drecord_db, which will write out the data record as JSON to a file
• parsing of timestamps will be the most difficult thing (this is not currently done in libipfix, and timestamps of various types, including NTP are frequently seen from the Netscaler) (d22a064)
• extend ipfixe_data_db_t to add a filename (and FILE* preferably, although we need to be able to rotate the file...) (08b2389)
• modify ipfix_export_init_db to take jsonfile argument and initialises the jsonfile particulars (08b2389)
• alter ipfix_col_init_mysqlexport to add jsonfile argument (NULL if not used), call ipfix_export_init_jsonfile and set g_colinfo->export_drecord to ipfix_export_drecord_jsonfile. (7b21bfb )
• alter collector/collector.c to add the jsonfile command-line option, and add par.jsonfile as an argument to ipfix_col_init_mysqlexport (5ab7a4f )

from libipfix.

There are a number of 64-bit integers being emitted in the Netscaler. Kibana, being written in JavaScript can't natively support a 64-bit integer. This will be true in Kibana 4 also. Elasticsearch can support them, but you would need to ensure they are mapped as 'long'

from libipfix.

This is about as complete as it is likely to get in this milestone; unless the Netscaler sends me FLOAT or BOOLEAN types...

from libipfix.