Handle repeated record types about aushape HOT 3 CLOSED

Apart from handling known repeated record types, we need to protect from other repeated records. Perhaps keep track of what types were encountered in an event and report an optional and recoverable error if a record is repeated unexpectedly.

from aushape.

Comparing approaches

NOTE: Execve records need to be transformed in any case, as we cannot have
numbered field names in a schema.

• No explicit events, records only
  • Searching is complicated
  • Very simple code (execve records still need modification to fit a schema)
  • New duplicated record types are no problem
  • Transformation is straight-forward
  • Getting all records for each event is a two-step search, as joining is not supported in ElasticSearch
  • Searching execve records is problematic
• Event object with record array inside
  • Searching is complicated
  • Straight-forward, simple code
  • New duplicated record types are no problem
  • Transformation is straight-forward
  • Needs nested data type use on record level, which is supported very poorly by Kibana. Otherwise fields from different record types mix.
  • Needs explicit record type mention with each search
  • Many cases affected, because fields mix between record types, and they can have different meanings
• Event objects with record type map, repeated records are distributed between distributed events
  • Searching is complicated
  • A little complicated code
  • New duplicated record types are no problem
  • Transformation is confusing
  • It is possible that only some or one record with repeated type will show up in results. Requires extra searching step to reveal all the records, which gets more complicated if more than one event is matched.
  • Breaks one-to-one conversion relationship on top (event) level, confusing
  • Introduces irregularity into the design right from the start
• Event objects with record type map, duplicated records aggregated (current)
  • Simple searching in most cases
  • Complicated code, growing with number of repeated record types
  • New duplicated record types need manual handling
  • Transformation is straight-forward with duplicated records an exception
  • Still requires some nested data types, but only for a few record types
  • Needs extra error handling for unexpected duplicated records.

from aushape.

There is no confirmation to repeated AVC records.

from aushape.