Support array values for dimension fields about elasticsearch HOT 5 OPEN

Pinging @elastic/es-storage-engine (Team:StorageEngine)

from elasticsearch.

Stepping back, mapping all attributes as dimensions seems problematic, since they're also used in tsid calculations etc. What's the reasoning behind storing all these IPs along with metric values? Do we anticipate filtering for a particular IP, as opposed to hostname etc? More so, how much value do we get by storing an array of IPs, instead of the first one as suggested, without additional info on what each IP represents?

from elasticsearch.

OTel's metrics data model define a time series by an entity consisting of several metadata properties, which includes all attributes.

We can't make a generic assumption that the time series can still be uniquely identified when we just store the first value of an attribute array.

I do agree that in the specific case of the host.ip field, it may not always be an identifying attribute. But at this time, there's no concept of a non-identifying attribute in OpenTelemetry. This may change in the future and in that case we can store these non-identifying attributes in a namespace that is not a container for dimensions (time_series_dimension: false). However, that doesn't change the fact that there can still be multi-valued identifying attributes. It's something that the OTel data model allows for but the TSDS data model doesn't, which creates some kind of impedance mismatch when trying to map OTel metrics to TSDS.

While I do think that we need to allow for multi-valued dimensions at some point, I also think that we can live with the workaround of storing a comma-separated string and having no support for automatically mapping IP fields as the ip field type.

from elasticsearch.

Makes sense, thanks. While we do want to eventually support all corner cases, having concrete data about how much of an issue this is in practice can help us prioritize it.

from elasticsearch.

I don't have data on that at the moment but I don't expect this to be a critical issue. We'll probably learn more along the way and maybe users can chime in here when and if this becomes a bigger problem than anticipated. As long as we're aligned that we want to fix this eventually, I'm good with putting this on the shelf for the time being.

from elasticsearch.