NATS cluster has different values of the same metric about prometheus-nats-exporter HOT 6 OPEN

Hey,
we have the same issue with jetstream_consumer_num_pending .
As a workaround I added != 0 in my Grafana dashboard and set "Connect null values" to "Always" in the Time series panel. You might be able to use this for alerting if you dont alert on "null values", just keep in mind that you have less data points as prometheus sometimes scrapes the wrong value (for us the wrong value is always 0 ).
I am not sure I would trust such an alert 100% though.

from prometheus-nats-exporter.

Same behaviour for us with jetstream_consumer_num_pending. This already happened with the exporter in version 0.9.1. Upgraded it to 0.11.0 but it still shows the same behaviour.

from prometheus-nats-exporter.

Hello @wallyqs, sorry for ping you, but in general, it is difficult to understand which server displays the real information.
we have different values from each nats server(0 / 8 / 0):

nats_consumer_num_pending{account="TEST",account_id="ID",cluster="nats",consumer_desc="",consumer_leader="nats-1",
consumer_name="monitor",domain="",is_consumer_leader="false",is_meta_leader="false",is_stream_leader="false",
meta_leader="nats-2",server_name="nats-0",stream_leader="nats-2",stream_name="TEST"} 0

nats_consumer_num_pending{account="TEST",account_id="ID",cluster="nats",consumer_desc="",consumer_leader="nats-1",
consumer_name="monitor",domain="",is_consumer_leader="true",is_meta_leader="false",is_stream_leader="false",
meta_leader="nats-2",server_name="nats-1",stream_leader="nats-2",stream_name="TEST"} 8

nats_consumer_num_pending{account="TEST",account_id="ID",cluster="nats",consumer_desc="",consumer_leader="nats-1",
consumer_name="monitor",domain="",is_consumer_leader="false",is_meta_leader="true",is_stream_leader="true",
meta_leader="nats-2",server_name="nats-2",stream_leader="nats-2",stream_name="TEST"} 0

and in this case nats-1 is the leader.
result of nats consumer info:

nats consumer info TEST monitor |grep -E 'Leader|Unprocessed'                                                                                                                                                     
              Leader: nats-1
     Unprocessed Messages: 8

and it's difficult to say what exactly is true, since the leader displays 8, but the other two servers are 0.
Could you say where the error is possible and I could prepare a PR.

from prometheus-nats-exporter.

A few new points, I used the promql query:
count(nats_consumer_num_pending > 0) by (cluster_id, account, consumer_name, stream_name, consumer_leader) > 0
and I found that if there is a difference in the same metric between different servers, then the metric difference is always on consumer_leader side.

and the second point is that when I try to restart the prometheus-nats-exporter container inside the nats server pod(with metric differences) by:
kill -HUP $(ps aufx |grep '[p]rometheus-nats-exporter' |awk '{print $1}')
prometheus-nats-exporter container is successfully restarted, but the metric value doesnt change. I tried restarting the whole pod, but the result is the same, nothing changes.
apparently, the error is not with the exporter, as if the nats server displays another metric value.
it looks like consumer replicas don't replicate these metrics from the consumer_leader.

from prometheus-nats-exporter.

as far as I understand, when using the nats consumer info command, information about "Unprocessed Messages" is always given by the consumer leader. Is there any way to view this metric on each nats server? there is a desire to connect to each server and see the list of unprocessed messages and compare their number with the metric, to understand where the error is

from prometheus-nats-exporter.

after testing, it turned out that the nats pod, which is consumer_leader at the moment, always shows the correct value for pending messages and for ack pending messages. I added the label is_consumer_leader="true" to Grafana dashboard and it solved the problem of incorrect data display.
the same for alerts expression:

nats_consumer_num_pending{env="stage", is_consumer_leader="true"} > 0

it will always be triggered only when the current values are.

@jlange-koch, != 0 is not always true, as I have observed situations that replicas show != 0, but in fact there are no pending messages and the leader correctly displays 0.

from prometheus-nats-exporter.