sidecar, stores, and query container memory spikes about thanos HOT 4 OPEN

I've seen the same yesterday. The thanos-sidecar started to eat all the memory available on the node, until OOM and crash.

It's the very first time we're noticing that behavior. The logs didn't show anything out of the ordinary.

We're also running 0.33.

from thanos.

I'm also seeing the same issue. Does upgrading to 0.34.x fix this? We upgraded from 0.31.0 to 0.33.0 and started noticing it a couple days later.

from thanos.

still seeing this with 0.34.1. I'm convinced it is a bad query or two being called from a Grafana panel, but not sure how to track it down exactly.

from thanos.

still seeing this with 0.34.1. I'm convinced it is a bad query or two being called from a Grafana panel, but not sure how to track it down exactly.

Can you try configuring:

      --query.active-query-path=""
                                 Directory to log currently active queries in
                                 the queries.active file.

? On restart of the querier it should log which queries were active when it crashed hopefully.

from thanos.