KubernetesPodNotHealthy expr problem about awesome-prometheus-alerts HOT 6 CLOSED

Hi @youpai, thanks for reporting this issue.

I tested the initial query, and I got the following error:

Error executing query: parse error at char 107: range specification must be preceded by a metric selector, but follows a *promql.AggregateExpr instead

Regarding this message, I think min_over_time does not support subquery.

Adding ':' does not work either, on my Prometheus instance.

What version of Prometheus are you using ?

from awesome-prometheus-alerts.

The version of Prometheus is 2.16.0。
Maybe there are some differences between different versions，but I think min_over_time does support subquery,the ref "https://prometheus.io/blog/2019/01/28/subquery-support/"

from awesome-prometheus-alerts.

Ok, my Prometheus server was too old then.

I'll prepare a PR ;)

from awesome-prometheus-alerts.

Hi! I think this expression query still needs some love. We just began using it and we're getting false positives. That means, short-lived pods that go trough the following phases: Pending, Running, Succeeded, Failed, Unknown get marked as unhealthy even if they did their work. Example:

trafficdatalakeingestionespdrivingsensor-12de660039ef47959b6b90046c978ff2                0/1     Pending             0          0s
trafficdatalakeingestionespdrivingsensor-12de660039ef47959b6b90046c978ff2                0/1     ContainerCreating   0          1s
trafficdatalakeingestionespdrivingsensor-12de660039ef47959b6b90046c978ff2                1/1     Running             0          3s
trafficdatalakeingestionespdrivingsensor-12de660039ef47959b6b90046c978ff2                0/1     Completed           0          16s
trafficdatalakeingestionespdrivingsensor-12de660039ef47959b6b90046c978ff2                0/1     Terminating         0          17s

From: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase

Failed | All containers in the Pod have terminated, and at least one container has terminated in failure. That is, the container either exited with non-zero status or was terminated by the system.

In our case they are terminated by the system, but everything was correct. I'll be thinking on a better expression but if you have any ideas I'm all eyes!

from awesome-prometheus-alerts.

@lgg42 I'm running into this too. I have gitlab CI runners and unit tests running on my cluster and they all trigger this because they are short lived. Did you ever come up with a better query?

from awesome-prometheus-alerts.

@djhoese Nope, hadn't the time yet. But is good to see I'm not the only one, sorry you're also suffering it 🙃

from awesome-prometheus-alerts.