Git Product home page Git Product logo

prometheusbeat's Introduction

Prometheusbeat

Prometheusbeat is an elastic beat that can receive Prometheus metrics via the remote write feature.

Example Prometheusbeat configuration:

prometheusbeat:
  listen: ":8080"
  context: "/prometheus"

[...]

Example Prometheus configuration:

[...]

remote_write:
  url: "http://localhost:8080/prometheus"

[...]

Example Prometheusbeat event:

{
  "@timestamp": "2019-08-26T11:34:04.253Z",
  "@metadata": {
    "beat": "prometheusbeat",
    "type": "_doc",
    "version": "7.3.1"
  },
  "labels": {
    "instance": "localhost:9090",
    "job": "prometheus"
  },
  "ecs": {
    "version": "1.0.1"
  },
  "host": {
    "containerized": false,
    "hostname": "test",
    "architecture": "x86_64",
    "os": {
      "kernel": "4.14.14-1.el7.elrepo.x86_64",
      "codename": "Core",
      "platform": "centos",
      "version": "7 (Core)",
      "family": "redhat",
      "name": "CentOS Linux"
    },
    "id": "338bc9f83bf343dfa1983fc2bc43bd0f",
    "name": "test"
  },
  "agent": {
    "version": "7.3.1",
    "type": "prometheusbeat",
    "ephemeral_id": "51cadc55-5746-4ad2-a278-b0dcac12942b",
    "hostname": "test",
    "id": "f9c5188e-a1ea-4fbb-adf6-8494f56c59bf"
  },
  "name": "scrape_series_added",
  "value": 349
}

Getting Started with Prometheusbeat

Requirements

Init Project

To get running with Prometheusbeat and also install the dependencies, run the following command:

make setup

It will create a clean git history for each major step. Note that you can always rewrite the history if you wish before pushing your changes.

To push Prometheusbeat in the git repository, run the following commands:

git remote set-url origin https://github.com/infonova/prometheusbeat
git push origin master

For further development, check out the beat developer guide.

Build

To build the binary for Prometheusbeat run the command below. This will generate a binary in the same directory with the name prometheusbeat.

mage build

Run

To run Prometheusbeat with debugging output enabled, run:

./prometheusbeat -c prometheusbeat.yml -e -d "*"

Test

To test Prometheusbeat, run the following command:

make testsuite

alternatively:

make unit-tests
make system-tests
make integration-tests
make coverage-report

The test coverage is reported in the folder ./build/coverage/

Update

Each beat has a template for the mapping in elasticsearch and a documentation for the fields which is automatically generated based on etc/fields.yml. To generate etc/prometheusbeat.template.json and etc/prometheusbeat.asciidoc

make update

Cleanup

To clean Prometheusbeat source code, run the following command:

make fmt

To clean up the build directory and generated artifacts, run:

make clean

Clone

To clone Prometheusbeat from the git repository, run the following commands:

mkdir -p ${GOPATH}/github.com/infonova
cd ${GOPATH}/github.com/infonova
git clone https://github.com/infonova/prometheusbeat

For further development, check out the beat developer guide.

Packaging

The beat frameworks provides tools to crosscompile and package your beat for different platforms. This requires docker and vendoring as described above. To build packages of your beat, run the following command:

make release

This will fetch and create all images required for the build process. The whole process to finish can take several minutes.

Building Docker Image

Prometheusbeat Docker image can be customized and build to run on any environments

docker build . -t infonova/prometheusbeat:latest

Running Docker Image

Prometheusbeat Docker image can be run using below command

docker run -d  --name prometheusbeat -p <<dockerhost-port>>:8080 -v <<host-config-path>>:/prometheusbeat.yml infonova/prometheusbeat:latest

prometheusbeat's People

Contributors

boernd avatar vikasshiva avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

prometheusbeat's Issues

the data didn't have time

{
"_index": "prometheusbeat-7.3.1-2019.09.10",
"_type": "doc",
"_id": "UcjkGG0Bq4bH-F3rF0q1",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2019-09-10T01:39:28.491Z",
"ecs": {
"version": "1.0.1"
},
"host": {
"os": {
"platform": "debian",
"version": "9 (stretch)",
"family": "debian",
"name": "Debian GNU/Linux",
"kernel": "5.0.7-1.el7.elrepo.x86_64",
"codename": "stretch"
},
"id": "689cfabc30776e6bfe2e7477e81eaa6d",
"containerized": false,
"hostname": "prometheusbeat-b4d994dd-d89xb",
"architecture": "x86_64",
"name": "prometheusbeat-b4d994dd-d89xb"
},
"agent": {
"version": "7.3.1",
"type": "prometheusbeat",
"ephemeral_id": "063b8cbb-45ef-49f7-bb9b-7af21ebdf94d",
"hostname": "prometheusbeat-b4d994dd-d89xb",
"id": "3b0510d4-100d-433d-a77b-15cad1ccc940"
},
"name": "node_cpu_seconds_total",
"value": 0,
"labels": {
"cpu": "6",
"endpoint": "https",
"instance": "*****:9100",
"namespace": "monitoring",
"service": "node-exporter-outk8s",
"job": "node-exporter-outk8s",
"mode": "steal",
"prometheus": "monitoring/k8s",
"prometheus_replica": "prometheus-k8s-0"
}
},
"fields": {
"@timestamp": [
"2019-09-10T01:39:28.491Z"
]
},
"sort": [
1568079568491
]
}

the data prometheusbeat remote write into es ,didn't hava the time,is the doc normal?

Add vendoring

For libraries used within prometheus/prometheus.go

Missing metric name

I'm using the Docker version of Prometheusbeat with ES 6.2. Prometheus is using cadvisor to gather Docker metrics. The name of the metrics does not seem to be arriving to Elasticsearch.

This is a sample as taken from Prometheus:

Mon, 04 Mar 2019 13:23:18 GMTP
container_spec_cpu_shares: 1024

container_label_com_docker_compose_service: prometheus,
container_label_com_docker_compose_slug: 41c1993279644d72a7854d00f3121e226fa39ed87583f23e0dc1ffae7c4237e,
container_label_com_docker_compose_oneoff: False,
container_label_maintainer: The Prometheus Authors <[email protected]>, 
container_label_com_docker_compose_container_number: 1,
container_label_com_docker_compose_version: 1.23.1,
container_label_com_docker_compose_config_hash: 167f95bd6d7f050d957025016fb2c77620d525554cd24ef9a3dc8c13380f35bf,
container_label_com_docker_compose_project: 01,
job: cadvisor
instance: cadvisor:8080,
id: /docker/5675a33223e73fba11cbbee3a234451814d2bced7bc3afb8da4796505d4a1c7f,
image: prom/prometheus:v2.7.1,
name: prometheus

However, what arrives to Elasticsearch misses the name of the metric. See the source field in the ES documents:

{
    "@timestamp": "2019-03-04T14:08:19.903Z",
    "value": 1024,
    "labels": {
      "id": "/docker/5675a33223e73fba11cbbee3a234451814d2bced7bc3afb8da4796505d4a1c7f",
      "image": "prom/prometheus:v2.7.1",
      "container_label_com_docker_compose_service": "prometheus",
      "container_label_com_docker_compose_slug": "41c1993279644d72a7854d00f3121e226fa39ed87583f23e0dc1ffae7c4237e",
      "instance": "cadvisor:8080",
      "container_label_com_docker_compose_oneoff": "False",
      "container_label_maintainer": "The Prometheus Authors <[email protected]>",
      "state": "iowaiting",
      "container_label_com_docker_compose_container_number": "1",
      "container_label_com_docker_compose_version": "1.23.1",
      "container_label_com_docker_compose_config_hash": "167f95bd6d7f050d957025016fb2c77620d525554cd24ef9a3dc8c13380f35bf",
      "container_label_com_docker_compose_project": "01",
      "job": "cadvisor"
    },
    "name": "prometheus"
}

container_spec_cpu_shares is missing. What am I doing wrong?

replicasets resource is now in "apps" apigroup

https://github.com/infonova/prometheusbeat/blob/master/vendor/github.com/elastic/beats/deploy/kubernetes/metricbeat/metricbeat-role.yaml

in the source code file above, the kubernetes resource "replicasets" is defined as an "extensions" instead of "app". this causes error when running metric-beat in AWS EKS 1.15 cluster.

  • apiGroups: ["extensions"]
    resources:
    • replicasets
      verbs: ["get", "list", "watch"]

I added the "replicasets" resource to the apiGroups "apps" in order to fix the error that I got.

  • apiGroups: ["apps"]
    resources:
    • statefulsets
    • deployments
    • replicasets
      verbs: ["get", "list", "watch"]

Fatal error

Hi,
I'am runing prometheusbeat in a K8S environment. I have deployed it through Helm. I have this FATAL ERROR at runtime :

2018-10-23T09:59:44.857Z	INFO	instance/beat.go:544	Home path: [/] Config path: [/] Data path: [//data] Logs path: [//logs]
2018-10-23T09:59:44.862Z	INFO	instance/beat.go:551	Beat UUID: 883dfa0d-82b0-4f17-97a2-fcb708eaa4b1
2018-10-23T09:59:44.862Z	INFO	[seccomp]	seccomp/seccomp.go:116	Syscall filter successfully installed
2018-10-23T09:59:44.862Z	INFO	[beat]	instance/beat.go:768	Beat info	{"system_info": {"beat": {"path": {"config": "/", "data": "/data", "home": "/", "logs": "/logs"}, "type": "prometheusbeat", "uuid": "883dfa0d-82b0-4f17-97a2-fcb708eaa4b1"}}}
2018-10-23T09:59:44.862Z	INFO	[beat]	instance/beat.go:777	Build info	{"system_info": {"build": {"commit": "unknown", "libbeat": "6.4.1", "time": "1754-08-30T22:43:41.128Z", "version": "6.4.1"}}}
2018-10-23T09:59:44.862Z	INFO	[beat]	instance/beat.go:780	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.11"}}}
2018-10-23T09:59:44.863Z	INFO	[beat]	instance/beat.go:784	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2018-10-10T13:56:05Z","containerized":true,"name":"prombeat-778894dcfd-pqx27","ip":["127.0.0.1/8","::1/128","100.117.221.82/32","fe80::fc8e:37ff:fe77:b550/64"],"kernel_version":"4.4.121-k8s","mac":["fe:8e:37:77:b5:50"],"os":{"family":"debian","platform":"debian","name":"Debian GNU/Linux","version":"9 (stretch)","major":9,"minor":0,"patch":0,"codename":"stretch"},"timezone":"UTC","timezone_offset_sec":0,"id":"3e608929fbd39b959f388bf468c9f0b1"}}}
2018-10-23T09:59:44.864Z	INFO	[beat]	instance/beat.go:813	Process info	{"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/", "exe": "/prometheusbeat", "name": "prometheusbeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter"}, "start_time": "2018-10-23T09:59:44.540Z"}}}
2018-10-23T09:59:44.864Z	INFO	instance/beat.go:273	Setup Beat: prometheusbeat; Version: 6.4.1
2018-10-23T09:59:44.864Z	INFO	elasticsearch/client.go:163	Elasticsearch url: https://esclient.mydomain.com:80
2018-10-23T09:59:44.865Z	INFO	pipeline/module.go:98	Beat name: prombeat-778894dcfd-pqx27
2018-10-23T09:59:44.865Z	INFO	instance/beat.go:367	prometheusbeat start running.
2018-10-23T09:59:44.865Z	INFO	beater/prometheusbeat.go:38	prometheusbeat is running! Hit CTRL-C to stop it.
2018-10-23T09:59:44.865Z	INFO	[monitoring]	log/log.go:114	Starting metrics logging every 30s
2018-10-23T10:00:03.727Z	INFO	[monitoring]	log/log.go:149	Total non-zero metrics	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":0,"time":{"ms":8}},"total":{"ticks":20,"time":{"ms":32},"value":20},"user":{"ticks":20,"time":{"ms":24}}},"info":{"ephemeral_id":"91d792f9-af5d-4e50-9a22-d470adb2ed14","uptime":{"ms":18875}},"memstats":{"gc_next":4194304,"memory_alloc":1895960,"memory_total":3658672,"rss":24375296}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"system":{"cpu":{"cores":2},"load":{"1":0.27,"15":0.24,"5":0.27,"norm":{"1":0.135,"15":0.12,"5":0.135}}}}}}
2018-10-23T10:00:03.727Z	INFO	[monitoring]	log/log.go:150	Uptime: 18.877659338s
2018-10-23T10:00:03.727Z	INFO	[monitoring]	log/log.go:127	Stopping metrics logging.
2018-10-23T10:00:03.727Z	INFO	runtime/asm_amd64.s:523	prometheusbeat stopped.
2018-10-23T10:00:03.727Z	FATAL	[prometheusbeat]	instance/beat.go:161	Failed due to panic.	{"panic": "interface conversion: interface {} is nil, not int64", "stack": "github.com/infonova/prometheusbeat/vendor/github.com/elastic/beats/libbeat/cmd/instance.Run.func1.1\n\t/go/src/github.com/infonova/prometheusbeat/vendor/github.com/elastic/beats/libbeat/cmd/instance/beat.go:162\nruntime.call32\n\t/usr/local/go/src/runtime/asm_amd64.s:522\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:513\nruntime.panicdottypeE\n\t/usr/local/go/src/runtime/iface.go:248\ngithub.com/infonova/prometheusbeat/beater.(*Prometheusbeat).Run\n\t/go/src/github.com/infonova/prometheusbeat/beater/prometheusbeat.go:60\ngithub.com/infonova/prometheusbeat/vendor/github.com/elastic/beats/libbeat/cmd/instance.(*Beat).launch\n\t/go/src/github.com/infonova/prometheusbeat/vendor/github.com/elastic/beats/libbeat/cmd/instance/beat.go:373\ngithub.com/infonova/prometheusbeat/vendor/github.com/elastic/beats/libbeat/cmd/instance.Run.func1\n\t/go/src/github.com/infonova/prometheusbeat/vendor/github.com/elastic/beats/libbeat/cmd/instance/beat.go:188\ngithub.com/infonova/prometheusbeat/vendor/github.com/elastic/beats/libbeat/cmd/instance.Run\n\t/go/src/github.com/infonova/prometheusbeat/vendor/github.com/elastic/beats/libbeat/cmd/instance/beat.go:189\ngithub.com/infonova/prometheusbeat/vendor/github.com/elastic/beats/libbeat/cmd.genRunCmd.func1\n\t/go/src/github.com/infonova/prometheusbeat/vendor/github.com/elastic/beats/libbeat/cmd/run.go:36\ngithub.com/infonova/prometheusbeat/vendor/github.com/spf13/cobra.(*Command).execute\n\t/go/src/github.com/infonova/prometheusbeat/vendor/github.com/spf13/cobra/command.go:766\ngithub.com/infonova/prometheusbeat/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/go/src/github.com/infonova/prometheusbeat/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/infonova/prometheusbeat/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/go/src/github.com/infonova/prometheusbeat/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/go/src/github.com/infonova/prometheusbeat/main.go:12\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:201"}

I have checked that my ES cluster is reachable from inside the pod. I have no other idea about the reason of this pb

make setup issue "ERROR: field <@timestamp> is duplicated, remove it or set 'overwrite: true'

Not sure if this is the correct place for this issue. I cloned the repo to go/src/github.com instead of go/github.com, since the src directory is where all the other packages were installed. Any help is appreciated. Thank you,

ubuntu % make setup
mkdir -p vendor/github.com/elastic/
cp -R /home/bwillcox/go/src/github.com/elastic/beats vendor/github.com/elastic/
rm -rf vendor/github.com/elastic/beats/.git
make update
make[1]: Entering directory '/home/bwillcox/go/src/github.com/infonova/prometheusbeat/prometheusbeat'
Running virtualenv with interpreter /usr/bin/python2
New python executable in /home/bwillcox/go/src/github.com/infonova/prometheusbeat/prometheusbeat/build/python-env/bin/python2
Also creating executable in /home/bwillcox/go/src/github.com/infonova/prometheusbeat/prometheusbeat/build/python-env/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
Generated fields.yml for prometheusbeat to /home/bwillcox/go/src/github.com/infonova/prometheusbeat/prometheusbeat/fields.yml
Updating generated files for prometheusbeat
mkdir -p include
go run ./vendor/github.com/elastic/beats/dev-tools/cmd/asset/asset.go -license ASL2 -pkg include -in fields.yml -out include/fields.go prometheusbeat
if [ -d /home/bwillcox/go/src/github.com/infonova/prometheusbeat/prometheusbeat/_meta/kibana ]; then
cp -pr /home/bwillcox/go/src/github.com/infonova/prometheusbeat/prometheusbeat/_meta/kibana/* /home/bwillcox/go/src/github.com/infonova/prometheusbeat/pr
ometheusbeat/_meta/kibana.generated ;
fi
ERROR: field <@timestamp> is duplicated, remove it or set 'overwrite: true'
exit status 1
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:315: recipe for target 'update' failed
make[1]: *** [update] Error 1
make[1]: Leaving directory '/home/bwillcox/go/src/github.com/infonova/prometheusbeat/prometheusbeat'
Makefile:18: recipe for target 'setup' failed
make: *** [setup] Error 2
ubuntu % go version
go version go1.10.4 linux/amd64
ubuntu % uname -a
Linux ubuntu 4.15.0-42-generic #45-Ubuntu SMP Thu Nov 15 19:32:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

make issue

# cat /etc/*release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
# go version
go version go1.7.4 linux/amd64

Got the following issues:

# make setup
mkdir -p vendor/github.com/elastic/
cp -R /home/projects/src/src/github.com/elastic/beats vendor/github.com/elastic/
rm -rf vendor/github.com/elastic/beats/.git
make update
make[1]: Entering directory '/home/projects/src/src/github.com/infonova/prometheusbeat'
make[2]: Entering directory '/home/projects/src/src/github.com/infonova/prometheusbeat/vendor/github.com/elastic/beats/libbeat'
echo /home/projects/src/src/github.com/infonova/prometheusbeat/vendor/github.com/elastic/beats/libbeat
/home/projects/src/src/github.com/infonova/prometheusbeat/vendor/github.com/elastic/beats/libbeat
Generated fields.yml for libbeat
make[2]: Leaving directory '/home/projects/src/src/github.com/infonova/prometheusbeat/vendor/github.com/elastic/beats/libbeat'
Updating generated files for prometheusbeat
mkdir -p include
go run  ./vendor/github.com/elastic/beats/dev-tools/cmd/asset/asset.go -pkg include fields.yml prometheusbeat > include/fields.go
Invalid file path: %s fields.yml
exit status 1
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:319: recipe for target 'update' failed
make[1]: *** [update] Error 1
make[1]: Leaving directory '/home/projects/src/src/github.com/infonova/prometheusbeat'
Makefile:18: recipe for target 'setup' failed
make: *** [setup] Error 2
# make
fatal: Not a git repository (or any parent up to mount point /home)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
go build -i -ldflags "-X github.com/elastic/beats/libbeat/version.buildTime=2018-06-08T09:03:40Z -X github.com/elastic/beats/libbeat/version.commit="
import cycle not allowed
package github.com/infonova/prometheusbeat
	imports github.com/infonova/prometheusbeat/beater
	imports github.com/elastic/beats/libbeat/publisher
	imports github.com/elastic/beats/libbeat/outputs
	imports github.com/elastic/beats/libbeat/publisher
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:112: recipe for target 'prometheusbeat' failed
make: *** [prometheusbeat] Error 1

Make error when installing repository

Hi, I´m trying to install the prometheusbeat but when I run make setup after downloading repository with git clone, I get this output in the terminal:

~/Desktop/prometheusbeat$ make setup
mkdir -p vendor/github.com/elastic
cp -R /src/github.com/elastic/beats vendor/github.com/elastic/
cp: cannot stat '/src/github.com/elastic/beats': No such file or directory
make: *** [Makefile:28: copy-vendor] Error 1

Am I missing something? Thanks in advance

accept4: too many open files

few seconds after running : ~/work/src/github.com/infonova/prometheusbeat# ./prometheusbeat -c prometheusbeat.yml

the output says and don't data to Elastic

2019/03/28 13:57:27 http: Accept error: accept tcp [::]:9080: accept4: too many open files; retrying in 80ms
2019/03/28 13:57:27 http: Accept error: accept tcp [::]:9080: accept4: too many open files; retrying in 160ms
2019/03/28 13:57:27 http: Accept error: accept tcp [::]:9080: accept4: too many open files; retrying in 320ms
2019/03/28 13:57:28 http: Accept error: accept tcp [::]:9080: accept4: too many open files; retrying in 5ms
2019/03/28 13:57:28 http: Accept error: accept tcp [::]:9080: accept4: too many open files; retrying in 10ms
2019/03/28 13:57:28 http: Accept error: accept tcp [::]:9080: accept4: too many open files; retrying in 20ms
2019/03/28 13:57:28 http: Accept error: accept tcp [::]:9080: accept4: too many open files; retrying in 40ms
2019/03/28 13:57:28 http: Accept error: accept tcp [::]:9080: accept4: too many open files; retrying in 80ms
2019/03/28 13:57:28 http: Accept error: accept tcp [::]:9080: accept4: too many open files; retrying in 5ms
2019/03/28 13:57:28 http: Accept error: accept tcp [::]:9080: accept4: too many open files; retrying in 10ms
2019/03/28 13:57:28 http: Accept error: accept tcp [::]:9080: accept4: too many open files; retrying in 5ms
2019/03/28 13:57:28 http: Accept error: accept tcp [::]:9080: accept4: too many open files; retrying in 10ms
2019/03/28 13:57:28 http: Accept error: accept tcp [::]:9080: accept4: too many open files; retrying in 20ms
2019/03/28 13:57:28 http: Accept error: accept tcp [::]:9080: accept4: too many open files; retrying in 40ms

Unable to send data to elasticsearch due to error - 'net/http: request canceled'

Error observed in prometheus beat logs:
2019-02-07T13:40:56.848Z INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":140,"time":141},"total":{"ticks":530,"time":539,"value":530},"user":{"ticks":390,"time":398}},"info":{"ephemeral_id":"16da06d7-11fa-4512-93f0-5fe4eb48ca2b","uptime":{"ms":180010}},"memstats":{"gc_next":22617376,"memory_alloc":18169120,"memory_total":38017664,"rss":1413120}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":10168,"published":1688,"total":1688}}},"system":{"load":{"1":2.12,"15":2.42,"5":2.29,"norm":{"1":0.0883,"15":0.1008,"5":0.0954}}}}}}
2019-02-07T13:39:29.731Z ERROR elasticsearch/client.go:299 Failed to perform any bulk index operations: Post http://192.168.194.143:9200/_bulk: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2019-02-07T13:39:30.732Z ERROR pipeline/output.go:92 Failed to publish events: Post http://192.168.194.143:9200/_bulk: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Error observed in prometheus:
Feb 08 05:29:50 miq-sap-prod-mgt-01.gvs.ggn prometheus[16515]: level=warn ts=2019-02-08T05:29:50.593010061Z caller=queue_manager.go:531 component=remote queue=0:http://192.168.162.71:10080/prometheus msg="Error sending samples to remote storage" count=20 err="context deadline exceeded"
Feb 08 05:29:53 miq-sap-prod-mgt-01.gvs.ggn prometheus[16515]: level=warn ts=2019-02-08T05:29:53.342480882Z caller=queue_manager.go:230 component=remote queue=0:http://192.168.162.71:10080/prometheus msg="Remote storage queue full, discarding sample. Multiple subsequent messages of this kind may be suppressed."

Logs prometheus beat in debug mode:
2019-02-08T05:37:47.903Z DEBUG [event] common/event.go:55 Dropped nil value from event where key=name
2019-02-08T05:37:47.903Z DEBUG [event] common/event.go:55 Dropped nil value from event where key=tags
2019-02-08T05:37:47.904Z DEBUG [publish] pipeline/processor.go:275 Publish event: {
"@timestamp": "2019-02-08T05:37:47.780Z",
"@metadata": {
"beat": "prometheusbeat",
"type": "doc",
"version": "6.2.4"
},
"value": 2.147483648e+09,
"labels": {
"name": "jvm_memory_bytes_max",
"environment": "****",
"instance": "",
"job": "kubernetes-pods"
},
"beat": {
"version": "6.2.4",
"name": "
",
"hostname": "*****"
}
}

ES version is also 6.2.4

Can you help with resolving the issue?

Failed to encode event: unsupported float value: -Inf

Hi, I'm running prometheus 2.5 inside K8s and prometheusbeat on a different host behind nginx,
It seems prometheusbeat can't write metrics to elasticsearch, It keeps repeating the following error:

    ERROR   elasticsearch/client.go:374     Failed to encode event: unsupported float value: -Inf

connection es refused when prometheus beat run in one or two days.

==========prometheus logs:
level=warn ts=2018-11-26T07:44:39.128148501Z caller=queue_manager.go:530 component=remote queue=0:http://172.32.149.137:8089/prometheus msg="Error sending samples to remote storage" count=100 err="Post http://172.32.149.137:8089/prometheus: dial tcp 172.32.149.137:8089: connect: connection refused"
==========prometheus beat logs:
2018-11-24T16:23:44.944Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://172.32.150.100:9200)): Get http://172.32.150.100:9200: dial tcp 172.32.150.100:9200: connect: connection refused
2018-11-24T16:24:13.992Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(http://172.32.150.100:9200)) with 86 reconnect attempt(s)
2018-11-24T16:24:35.465Z ERROR pipeline/output.go:121 Failed to publish events: temporary bulk send failure
2018-11-24T16:25:06.044Z INFO pipeline/output.go:95 Connecting to backoff(elasticsearch(http://172.32.150.98:9200))
2018-11-24T16:25:16.773Z ERROR pipeline/output.go:121 Failed to publish events: temporary bulk send failure
2018-11-24T16:28:20.393Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://172.32.150.100:9200)): Get http://172.32.150.100:9200: dial tcp 172.32.150.100:9200: connect: connection refused
2018-11-24T16:26:23.369Z INFO pipeline/output.go:95 Connecting to backoff(elasticsearch(http://172.32.150.99:9200))
2018-11-24T16:29:03.007Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(http://172.32.150.100:9200)) with 87 reconnect attempt(s)
2018-11-24T16:29:12.393Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://172.32.150.98:9200)): Get http://172.32.150.98:9200: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2018-11-24T16:32:43.126Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(http://172.32.150.98:9200)) with 1 reconnect attempt(s)
2018-11-24T16:34:19.741Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://172.32.150.99:9200)): Get http://172.32.150.99:9200: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2018-11-24T16:35:43.159Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://172.32.150.100:9200)): Get http://172.32.150.100:9200: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2018-11-24T16:35:26.607Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(http://172.32.150.99:9200)) with 1 reconnect attempt(s)
2018-11-24T16:36:03.726Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(http://172.32.150.100:9200)) with 88 reconnect attempt(s)
2018-11-24T16:36:07.941Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://172.32.150.98:9200)): Get http://172.32.150.98:9200: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2018-11-24T16:37:00.245Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(http://172.32.150.98:9200)) with 2 reconnect attempt(s)
2018-11-24T16:38:49.730Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://172.32.150.99:9200)): Get http://172.32.150.99:9200: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2018-11-24T16:39:00.350Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(http://172.32.150.99:9200)) with 2 reconnect attempt(s)
2018-11-24T16:39:53.643Z INFO elasticsearch/client.go:712 Connected to Elasticsearch version 5.2.2
2018-11-24T16:39:55.999Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://172.32.150.100:9200)): Get http://172.32.150.100:9200: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2018-11-24T16:40:53.891Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(http://172.32.150.100:9200)) with 89 reconnect attempt(s)

Add support for Elasticsearch 7

Can you please add support for Elasticsearch 7.x?

Exporting the template with:

prometheusbeat -c "$prometheusbeatHome\prometheusbeat.yml" export template

And putting it in ES:

PUT _template/prometheusbeat
{
  "index_patterns": [
    "prometheusbeat-*"
  ],
  "mappings": {
    "doc": {
      "_meta": {
        "version": "6.4.1"
      },
      "date_detection": false,
      "dynamic_templates": [
        {
          "strings_as_keyword": {
            "mapping": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "match_mapping_type": "string"
          }
        }
      ],
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "name": {
          "ignore_above": 1024,
          "type": "keyword"
        },
        "value": {
          "type": "double"
        }
      }
    }
  },
  "order": 1,
  "settings": {
    "index": {
      "mapping": {
        "total_fields": {
          "limit": 10000
        }
      },
      "number_of_routing_shards": 30,
      "refresh_interval": "5s"
    }
  }
}

Gives the following error:

{
    "error": {
        "root_cause": [{
            "type": "mapper_parsing_exception",
            "reason": "Root mapping definition has unsupported parameters:  [doc : {_meta={version=6.4.1}, dynamic_templates=[{strings_as_keyword={mapping={ignore_above=1024, type=keyword}, match_mapping_type=string}}], date_detection=false, properties={@timestamp={type=date}, name={ignore_above=1024, type=keyword}, value={type=double}}}]"
        }],
        "type": "mapper_parsing_exception",
        "reason": "Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters:  [doc : {_meta={version=6.4.1}, dynamic_templates=[{strings_as_keyword={mapping={ignore_above=1024, type=keyword}, match_mapping_type=string}}], date_detection=false, properties={@timestamp={type=date}, name={ignore_above=1024, type=keyword}, value={type=double}}}]",
        "caused_by": {
            "type": "mapper_parsing_exception",
            "reason": "Root mapping definition has unsupported parameters:  [doc : {_meta={version=6.4.1}, dynamic_templates=[{strings_as_keyword={mapping={ignore_above=1024, type=keyword}, match_mapping_type=string}}], date_detection=false, properties={@timestamp={type=date}, name={ignore_above=1024, type=keyword}, value={type=double}}}]"
        }
    },
    "status": 400
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.