How to set up ALERT rule of prometheus. With gauge, final groked value is repeatedly reported forever even the new line will never occurs. about grok_exporter HOT 2 CLOSED

I think grok_exporter is not the right tool for monitoring CPU usage, you should try the node_exporter instead.

It is correct that grok_exporter values don't change when no new log lines are produced. The most common usage for grok_exporter is to count error messages in log files. For that purpose the behaviour is fine: The error counter metric does not increase unless there are new errors in the log file.

The usage of gauge or other metrics with grok_exporter is for special purposes only. Maybe if you have a tool logging some number at a periodic time interval (like a periodic job logging the number of items in a queue or something like that), you might use grok_exporter to get a gauge metric from that number. However, there are not a lot of scenarios where this is useful.

from grok_exporter.

I'm sorry for late reply.
I've found the workaround makes grok_exporter forget obsolete metrics: reboot or restart grok_exporter periodically. Although it seems to work well, I guess any kind of metric expiration option can be useful in some case.

It is correct that grok_exporter values don't change when no new log lines are produced.

I've understood my use case is not in the scope of grok_exporter now, but I'd like to share my experience. It might be useful for someone to try same thing as I did. I'm glad if you think about this once.

• I was trying to monitor container applications on our CloudFoundry(PaaS).
• To get HTTP response time and code of these applications, we need to use logs from cloudfoundry like below. It's comes from log function of cloudfoundry called firehose originally.

<6> 2017-04-13T03:19:03Z cd6af9910932 doppler[1]: {"cf_app_id":"10cf23cc-fb9b-423a-8271-e16979582f3c","cf_app_name":"logs","cf_ignored_app":false,"cf_org_id":"49e28c50-df1e-40f0-9269-74f7bbdff147","cf_org_name":"admin","cf_origin":"firehose","cf_space_id":"15f7b4fd-3de0-4c89-8382-e0880385268f","cf_space_name":"admin_space","content_length":27,"deployment":"cf","duration_ms":3,"event_type":"HttpStartStop","forwarded":["10.0.0.1"],"instance_id":"6df0c850-be49-48bc-4e46-8deae8b56dcd","instance_index":0,"ip":"172.16.0.1","job":"router","job_index":"a799da5e-c893-46f2-9290-e48912ca3788","level":"info","method":"GET","msg":"","origin":"gorouter","peer_type":"Server","remote_addr":"10.0.0.2:44429","request_id":"b0e7a638-0660-40b4-7901-d2bc661b72a8","start_timestamp":1492053543249780071,"status_code":302,"stop_timestamp":1492053543253082944,"time":"2017-04-13T03:19:03Z","uri":"http://example.com/xxxx/yyyy","user_agent":"curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2"}

• With grok_exporter, I could get HTTP response time and code from this log. It's great!
• Applications on PaaS can be crated, deleted or recreated on a daily basis by users, so its GUID and any kind of information are changed dynamically.
• Grok_exporter could find new application rapidly, but it never forget metrics of applications deleted already. In this case, prometheus returns same value repeatedly. It wastes disk volume gradually. At this point, I guess that implementing any kind of metrics expiration option to grok_exporter is worthful.
• Additionally, in some case, grafana shows the last value of deleted applications forever. I coundn't distinguish it's active or has gone already.
• After restarting grok_exporter, it has forgotten old useless metrics. (workaround)
• For example, firehose_exporter has doppler.metric-expiration option. It determines "How long Cloud Foundry metrics received from the Firehose are valid". (However, this exporter doesn't match my requirement. It doesn't include Org information of apps in its metrics.)
  https://github.com/cloudfoundry-community/firehose_exporter

I believe this exporter has great potential. I wish for the further development. Thanks. :-)

from grok_exporter.