prometheus_tsdb_head_series increases to double previous value during WAL replay then drops about prometheus HOT 8 OPEN

And you said in Slack that you have zero churn, so this should not be caused by #12286.

from prometheus.

This is partly as much a question as a bug report; this happens so consistently that I wouldn't be surprised at all if this is just expected behaviour and a lack of understanding of WAL replay on my part, but I could swear I haven't noticed this before, and I'm interested to see if there is an explanation - cc. @bboreham

from prometheus.

First suspicion would be this is caused by #12286. Since you are running 2.43, you get all the series from 3 compaction cycles back. 2.45 reduced this to 1 extra.

from prometheus.

First suspicion would be this is caused by #12286. Since you are running 2.43, you get all the series from 3 compaction cycles back. 2.45 reduced this to 1 extra.

I did see that one and we tried 2.45.2, but got the same result. It's unlikely we'll get a chance to prioritise more upgrades soon, but let me know if there's anything else we can look at. This isn't impacting us or high priority, more of a curiosity - it's good to know it's not expected behaviour at least!

from prometheus.

You would get the same result on first start; the fix only helps after one or two compaction cycles.

from prometheus.

Had the same on 2.43 Prometheus, so pretty old (re: https://cloud-native.slack.com/archives/C01AUBA4PFE/p1711572195620209). Will try to check newest one.

from prometheus.

I can reproduce the same with the latest v2.51.1 as well.

Same memory bump. Goes away on the next compaction cycle.

from prometheus.

Ok, I realized my setup caused some April Fool's Day prank on me -- I think we are chasing a bug that does not exists.. all my "issues" were due to bad interpretation of my data. Was using sum without instance label and staleness markers are not working as they should in my env (not a Prometheus bug).

Double check your PromQL @milesbxf

from prometheus.