ElasticSearch Unified Index about benchmark-wrapper HOT 3 OPEN

I think that this discussion becomes even more important with #298 and #296. Right now we are exporting a flat-JSON document, which is fine for now, but should we think about revisiting this since we are adding more fields?

Thoughts?
@whitleykeith
@jtaleric
@dry923

from benchmark-wrapper.

Elasticsearch is built on top of Lucene, which stores data in an inverted index. Simply put, Lucene maps terms -> documents instead of documents -> terms. Lucene doesn't really support nesting as it only supports numeric, binary, and text fields. ES does a lot of optimizations on top of Lucene to support nesting but I'm not sure about the index/search performance of them as I've only ever used Solr. But in general I would default to not nesting given the base technology ES uses (to a point).

ES has some guidelines on doc structure which is generally pretty solid. I think we should start here and look at defining the common core fields for all benchmarks and then specific core fields for each benchmark.

I think of the common core fields as things like uuid, run_id, start_time, end_time, .... Things that every benchmark should have.

I think every common core field should be left at the document root with no nesting. Every doc should have these fields defined and we should reject docs that don't. After that, we should look at how we search the data and fields that are heavily used in queries should be moved to the root of the doc. For instance I think all the environment information of a run (i.e. cluster_name, platform, etc.) should be as close to the root of the doc as possible to make searches easier and more performant.

The counter to that is when fields may or may not exist doc to doc, or are specific to the environment/benchmark. For instance not every snafu run may be running in k8s, so cluster_name, etc. may not be collected. For those we don't necessarily want them to be flat because we want a uniform root doc structure for better readability and also indexing.

Given that I think we can afford to nest somewhat. I think a good starting schema would look something like this:

{
    "uuid": "str",
    "run_id": "str",
    "start_time": "datetime",
    "end_time": "datetime",
    "duration": "Number",
    "type": "string",
    "iteration": "Number",
    "kubernetes": {
        "cluster_version": "str"
    },
    "openshift": {
        "cluster_version": "str",
        "cluster_name": "str",
        "platform": "str",
        "network_type": "str"
    },
    "config": {
        "foo": "bar"
    },
    "data": {
        "foo": "bar"
    }

}

I like that structure, and I think you have a really good point about modeling after ECS. I think it might be worth doing something like this:

{
    "uuid": "str",
    "run_id": "str",
    "start_time": "datetime",
    "end_time": "datetime",
    "duration": "Number",
    "type": "string",
    "iteration": "Number",
    "environment": "flattened",
    "config": "flattened",
    "data": "flattened
}