Comments (6)
boernd
commented on September 26, 2024
Looks like the timestamp is missing in the sample, thought this is more or less mandatory. Can you share your config (especially which data is sent via remote_write)?
from prometheusbeat.
cabrinoob
commented on September 26, 2024
here is my prometheus config file. I did not found any reference to "timestamp" in it.
global:
evaluation_interval: 1m
scrape_interval: 1m
scrape_timeout: 10s
rule_files:
- /etc/config/rules
- /etc/config/alerts
scrape_configs:
- job_name: prometheus
static_configs:
- targets:
- localhost:9090
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
job_name: kubernetes-apiservers
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- action: keep
regex: default;kubernetes;https
source_labels:
- __meta_kubernetes_namespace
- __meta_kubernetes_service_name
- __meta_kubernetes_endpoint_port_name
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
job_name: kubernetes-nodes
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- replacement: kubernetes.default.svc:443
target_label: __address__
- regex: (.+)
replacement: /api/v1/nodes/${1}/proxy/metrics
source_labels:
- __meta_kubernetes_node_name
target_label: __metrics_path__
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
job_name: kubernetes-nodes-cadvisor
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- replacement: kubernetes.default.svc:443
target_label: __address__
- regex: (.+)
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
source_labels:
- __meta_kubernetes_node_name
target_label: __metrics_path__
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
- job_name: kubernetes-service-endpoints
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- action: keep
regex: true
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_scrape
- action: replace
regex: (https?)
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_scheme
target_label: __scheme__
- action: replace
regex: (.+)
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_path
target_label: __metrics_path__
- action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
source_labels:
- __address__
- __meta_kubernetes_service_annotation_prometheus_io_port
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: kubernetes_namespace
- action: replace
source_labels:
- __meta_kubernetes_service_name
target_label: kubernetes_name
- honor_labels: true
job_name: prometheus-pushgateway
kubernetes_sd_configs:
- role: service
relabel_configs:
- action: keep
regex: pushgateway
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_probe
- job_name: kubernetes-services
kubernetes_sd_configs:
- role: service
metrics_path: /probe
params:
module:
- http_2xx
relabel_configs:
- action: keep
regex: true
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_probe
- source_labels:
- __address__
target_label: __param_target
- replacement: blackbox
target_label: __address__
- source_labels:
- __param_target
target_label: instance
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels:
- __meta_kubernetes_namespace
target_label: kubernetes_namespace
- source_labels:
- __meta_kubernetes_service_name
target_label: kubernetes_name
- job_name: kubernetes-pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: keep
regex: true
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape
- action: replace
regex: (.+)
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_path
target_label: __metrics_path__
- action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
source_labels:
- __address__
- __meta_kubernetes_pod_annotation_prometheus_io_port
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: kubernetes_namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: kubernetes_pod_name
remote_write:
- url: "http://prombeat.prometheus.svc.cluster.local:8080/prometheus"
alerting:
alertmanagers:
- kubernetes_sd_configs:
- role: pod
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_namespace]
regex: prometheus
action: keep
- source_labels: [__meta_kubernetes_pod_label_app]
regex: prometheus
action: keep
- source_labels: [__meta_kubernetes_pod_label_component]
regex: alertmanager
action: keep
- source_labels: [__meta_kubernetes_pod_container_port_number]
regex:
action: drop
from prometheusbeat.
boernd
commented on September 26, 2024
I have made some changes, can you test infonova/prometheusbeat:conversion-panic if this fixes your issue? In case the timestamp is missing I set the current timestamp manually.
from prometheusbeat.
cabrinoob
commented on September 26, 2024
Ok, I've just tried your updated image and I always have an error :
2018-10-23T14:17:21.691Z INFO instance/beat.go:544 Home path: [/] Config path: [/] Data path: [//data] Logs path: [//logs]
2018-10-23T14:17:21.694Z INFO instance/beat.go:551 Beat UUID: 41152cf8-31f8-4364-95d8-55bb31d4b3e9
2018-10-23T14:17:21.694Z INFO [seccomp] seccomp/seccomp.go:116 Syscall filter successfully installed
2018-10-23T14:17:21.694Z INFO [beat] instance/beat.go:768 Beat info {"system_info": {"beat": {"path": {"config": "/", "data": "/data", "home": "/", "logs": "/logs"}, "type": "prometheusbeat", "uuid": "41152cf8-31f8-4364-95d8-55bb31d4b3e9"}}}
2018-10-23T14:17:21.694Z INFO [beat] instance/beat.go:777 Build info {"system_info": {"build": {"commit": "unknown", "libbeat": "6.4.1", "time": "1754-08-30T22:43:41.128Z", "version": "6.4.1"}}}
2018-10-23T14:17:21.694Z INFO [beat] instance/beat.go:780 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.11"}}}
2018-10-23T14:17:21.695Z INFO [beat] instance/beat.go:784 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2018-06-13T12:55:31Z","containerized":true,"name":"prombeat-5999cb4bbc-gcbmc","ip":["127.0.0.1/8","::1/128","100.96.11.56/24","fe80::1072:fdff:fe1e:aa9/64"],"kernel_version":"4.4.121-k8s","mac":["0a:58:64:60:0b:38"],"os":{"family":"debian","platform":"debian","name":"Debian GNU/Linux","version":"9 (stretch)","major":9,"minor":0,"patch":0,"codename":"stretch"},"timezone":"UTC","timezone_offset_sec":0,"id":"3e608929fbd39b959f388bf468c9f0b1"}}}
2018-10-23T14:17:21.695Z INFO [beat] instance/beat.go:813 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/", "exe": "/prometheusbeat", "name": "prometheusbeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter"}, "start_time": "2018-10-23T14:17:20.660Z"}}}
2018-10-23T14:17:21.696Z INFO instance/beat.go:273 Setup Beat: prometheusbeat; Version: 6.4.1
2018-10-23T14:17:21.696Z INFO elasticsearch/client.go:163 Elasticsearch url: https://esclient.mydomain.com:443
2018-10-23T14:17:21.696Z INFO pipeline/module.go:98 Beat name: prombeat-5999cb4bbc-gcbmc
2018-10-23T14:17:21.696Z INFO [monitoring] log/log.go:114 Starting metrics logging every 30s
2018-10-23T14:17:21.697Z INFO instance/beat.go:367 prometheusbeat start running.
2018-10-23T14:17:21.697Z INFO beater/prometheusbeat.go:38 prometheusbeat is running! Hit CTRL-C to stop it.
2018-10-23T14:17:40.525Z INFO [monitoring] log/log.go:149 Total non-zero metrics {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":0,"time":{"ms":4}},"total":{"ticks":20,"time":{"ms":24},"value":20},"user":{"ticks":20,"time":{"ms":20}}},"info":{"ephemeral_id":"9c8b9d98-89ad-4980-ab01-3394771ff856","uptime":{"ms":18838}},"memstats":{"gc_next":4194304,"memory_alloc":1891792,"memory_total":3657736,"rss":24854528}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"system":{"cpu":{"cores":2},"load":{"1":0.31,"15":0.11,"5":0.16,"norm":{"1":0.155,"15":0.055,"5":0.08}}}}}}
2018-10-23T14:17:40.525Z INFO [monitoring] log/log.go:150 Uptime: 18.83850674s
2018-10-23T14:17:40.525Z INFO [monitoring] log/log.go:127 Stopping metrics logging.
2018-10-23T14:17:40.525Z INFO instance/beat.go:373 prometheusbeat stopped.
prometheusbeat stopped
from prometheusbeat.
boernd
commented on September 26, 2024
Hm, that's actually just an INFO log. It looks like the container is receiving a SIGTERM and shuts down gracefully. Can you run in debug mode (./prometheusbeat -e -c <path-to-config>/prometheusbeat.yml -d '*') to maybe see more details? Does it also shutdown when prometheus is not sending any data? What does your k8s deployment.yml look like?
from prometheusbeat.
cabrinoob
commented on September 26, 2024
Ok, it works now! I think that infonova/prometheusbeat:conversion-panic had fix the main problem, but I had a wrong configuration port in a k8s service.
Thank you for your help
from prometheusbeat.
Related Issues (19)
- Get remote write version from header
- windows binary? HOT 4
- connection es refused when prometheus beat run in one or two days. HOT 2
- Failed to encode event: unsupported float value: -Inf HOT 1
- make issue HOT 1
- make setup issue "ERROR: field <@timestamp> is duplicated, remove it or set 'overwrite: true' HOT 5
- Unable to send data to elasticsearch due to error - 'net/http: request canceled' HOT 7
- Missing metric name HOT 4
- accept4: too many open files HOT 3
- Add support for Elasticsearch 7 HOT 3
- Missing metadata fields in v7.x HOT 1
- the data didn't have time HOT 1
- prometheusbeat can not send data to ES HOT 1
- replicasets resource is now in "apps" apigroup HOT 1
- Make error when installing repository HOT 1
- Add vendoring
- Dockerfile HOT 1
- Do you have support for remote_read ? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from prometheusbeat.