Comments (7)
Please attach your config file, I'll have a look.
from grok_exporter.
Attached please find the conf file.
The reason I'm asking this question is due the fact I observed:
When using Prometheus query, no data returned for those expired metrics.
But from port 9144/metrics, the metrics are all there.
I assume grok_exporter should not display those expired metrics in port 9144?
Is there any way to get debug info.
It's seems some expired metrics are removed, but some are kept.
Not sure if it's my input file issue or not.
from grok_exporter.
global:
config_version: 2
retention_check_interval: 53s
input:
type: file
path: /path/to/input
readall: false # Read from the beginning of the file? False means we start at the end of the file and read only new lines.
grok:
patterns_dir: /path/to/pattern
metrics:
- type: gauge
name: worker_task_duration
help: task running duration of given task.
match: mypattern
value: '{{.run_time}}'
cumulative: false
labels:
WorkingQueue: '{{.queue}}'
Status: '{{.status}}'
StartTime: '{{.start_time}}'
TaskID: '{{.task_id}}'
retention: 5m
- type: gauge
name: scramble_task_duration
help: scrambling task running time.
match: mypattern
value: '{{.run_time}}'
cumulative: false
labels:
Status: '{{.status}}'
Product: '{{.product}}'
StartTime: '{{.start_time}}'
TaskID: '{{.task_id}}'
AttachmentID: '{{.attachment_id}}'
TsJobID: '{{.ts_id}}'
ErrorMessage: '{{.error_msg}}'
retention: 5m
- type: gauge
name: encryption_task_duration
help: encryption task running time.
match: mypattern
value: '{{.run_time}}'
cumulative: false
labels:
Status: '{{.status}}'
StartTime: '{{.start_time}}'
TaskID: '{{.task_id}}'
AttachmentID: '{{.attachment_id}}'
retention: 5m
server:
port: 9144
from grok_exporter.
Your config looks ok. The expected behavior is: If a metric is not updated for 5:53 minutes (retention time plus check interval), the metric should disappear from http://localhost:9144/metrics. Are you sure that no new log lines for these metrics are written?
Quick experiment to verify that retention works in general: The grok_exporter
distribution has an example in ./example/config.yaml
. The example metric is a counter named exim_rejected_rcpt_total
. I copied and pasted the metric to create exim_rejected_rcpt_total2
, which is exactly the same as exim_rejected_rcpt_total
but with retention: 2m
:
global:
config_version: 2
input:
type: file
path: ./example/exim-rejected-RCPT-examples.log
readall: true # Read from the beginning of the file? False means we start at the end of the file and read only new lines.
grok:
patterns_dir: ./logstash-patterns-core/patterns
additional_patterns:
- 'EXIM_MESSAGE [a-zA-Z ]*'
metrics:
- type: counter
name: exim_rejected_rcpt_total
help: Total number of rejected recipients, partitioned by error message.
match: '%{EXIM_DATE} %{EXIM_REMOTE_HOST} F=<%{EMAILADDRESS}> rejected RCPT <%{EMAILADDRESS}>: %{EXIM_MESSAGE:message}'
labels:
error_message: '{{.message}}'
- type: counter
name: exim_rejected_rcpt_total2
help: Total number of rejected recipients, partitioned by error message.
match: '%{EXIM_DATE} %{EXIM_REMOTE_HOST} F=<%{EMAILADDRESS}> rejected RCPT <%{EMAILADDRESS}>: %{EXIM_MESSAGE:message}'
labels:
error_message: '{{.message}}'
retention: 2m
server:
host: localhost
port: 9144
Now I run grok_exporter -config ./example/config.yml
. Initially, I see the same matches for both metrics:
exim_rejected_rcpt_total{error_message="Sender verify failed"} 2000
exim_rejected_rcpt_total{error_message="Unrouteable address"} 32
exim_rejected_rcpt_total{error_message="relay not permitted"} 165
exim_rejected_rcpt_total2{error_message="Sender verify failed"} 2000
exim_rejected_rcpt_total2{error_message="Unrouteable address"} 32
exim_rejected_rcpt_total2{error_message="relay not permitted"} 165
Obviously no new log messages are written, the logfile is unchanged. After 3 minutes, I see the exim_rejected_rcpt_total2
disappear and only the exim_rejected_rcpt_total
metrics are left:
exim_rejected_rcpt_total{error_message="Sender verify failed"} 2000
exim_rejected_rcpt_total{error_message="Unrouteable address"} 32
exim_rejected_rcpt_total{error_message="relay not permitted"} 165
You should have similar behavior with your config.
from grok_exporter.
I have the following observation these days, the finding is:
only one metrics retention didn't work, which means it's there all the time.
the other metrics could be deleted by retention timer.
For the un-deleted metrics, I have some empty lable value "" in the metrics, not sure if it's the cause.
I will try to verify this way and feedback later.
from grok_exporter.
I replace those lable value from "" to "-", then the retention works normal now.
Will the team fix this issue?
So far, I'm using prometheus client lib to push metrics to resolve this issue.
from grok_exporter.
Thanks for the analysis, I'll look into this.
from grok_exporter.
Related Issues (20)
- Feature Request: Option to specify ciphers or minimum TLS version HOT 2
- fail_on_missing_logfile for missing directories HOT 3
- Feature Request : Config option to expose metrics as a .prom file
- Issue reading Multiple files:
- how to match regex upper case/lower case insensitively
- Is grok able to watch only the latest log in dir?
- how to use regex group match and use matched group name label? HOT 1
- Can the latest version cover multi-line pattern match?
- if string exists HOT 7
- Use Gauge metric to find DB connectivity status HOT 1
- Exclude char HOT 2
- Does it support Elastic Search Open Distro HOT 1
- Configure for AWS ALB/CLB Logs or Cloudfront
- grok timestamp diffrent
- grok fails on insufficient permissions when fail_on_missing_logfile is false
- mutiple depth doc support HOT 1
- How can I use FuncMap to help transform data from the log, inside a template?
- Always report counter metric (initialize to zero) HOT 1
- match fails after encountering binary text within file
- Update or let someone else maintain this repo HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from grok_exporter.