ingensi / dockbeat Goto Github PK

View Code? Open in Web Editor NEW

287.0 24.0 70.0 25.7 MB

Dockbeat - the elastic Beat for docker daemon monitoring

License: Other

Makefile 1.79% Go 96.75% Python 0.82% Shell 0.64%

dockbeat's Introduction

Dockbeat

(if you're on the fast lane, check the TL;DR at the bottom of the readme)

Build status :

Test coverage :

Dockbeat is the new Dockerbeat name. We had to rename the project due to the Docker trademarking policy. Dockbeat is a Beat used for docker daemon monitoring. It is a lightweight agent that installed on your servers, reads periodically docker container statistics and indexes them in Elasticsearch.

Exported document types

There are five types of documents exported:

type: container: container attributes
type: cpu: container CPU usage statistics. One document per container is generated.
type: net: container network statistics. One document per network container is generated.
type: memory: container memory statistics. One document per container is generated.
type: blkio: container io access statistics. One document per container is generated.
type: log: dockbeat status information. One document per tick is generated if an error occurred.

To get a detailed list of all generated fields, please read the fields documentation page.

Elasticsearch template

To apply Dockbeat template (recommended but not required) :

curl -XPUT 'http://elastic:9200/_template/dockbeat' -d@etc/dockbeat.template.json

Build Dockbeat

Ensure that this folder is at the following location: ${GOPATH}/github.com/ingensi

Requirements

Golang 1.7
Glide >= 0.10.0

Build

To build the binary for Dockbeat run the command below. This will generate a binary in the same directory with the name dockbeat.

make

Run dockbeat

Project compilation generate a dockbeat executable file in the root directory. To launch dockbeat, run the following command:

./dockbeat -c etc/dockbeat.yml

Run in a docker container

The easiest way to launch dockbeat is to run it in a container. To achieve this, use the ingensi/dockbeat docker image, available on the docker hub.

Docker run command should:

mount the target Docker socket to /var/run/docker.sock
link an Elasticsearch node as elasticsearch

Example:

docker run -d -v /var/run/docker.sock:/var/run/docker.sock \
  --link elastic:elasticsearch ingensi/dockbeat:1.0.0-rc3

To override the default configuration, just link yours to /etc/dockbeat/dockbeat.yml:

docker run -d --link elastic:elasticsearch \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /volumes/dockbeat-config/:/etc/dockbeat \
  ingensi/dockbeat:1.0.0-rc3

By default, when dockbeat is running from this image, it logs into the /var/log/dockbeat directory. To access this logs from the host, link a directory to the dockbeat logging directory:

docker run -d --link elastic:elasticsearch \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /volumes/dockbeat-config/:/etc/dockbeat \
  -v /volumes/dockbeat-logs/:/var/logs/dockbeat \
  ingensi/dockbeat:1.0.0-rc3

Configuring Dockbeat

Dockbeat configuration file is located at etc/dockbeat.yml. This default template provides the following environment variable mapping:

How often to read server statistics
- ENV : PERIOD
- Beats variable : input.period
- Default value : 5
Docker socket path
- ENV : DOCKER_SOCKET
- Beats variable : input.socket
- Default value : unix:///var/run/docker.sock
Enable TLS encryption
- ENV : DOCKER_ENABLE_TLS
- Beats variable : input.tls.enable
- Default value : false
Path to the CA file (when TLS is enabled)
- ENV : DOCKER_CA_PATH
- Beats variable : input.tls.ca_path
- Default value : no default value
Path to the CERT file (when TLS is enabled)
- ENV : DOCKER_CERT_PATH
- Beats variable : input.tls.cert_path
- Default value : no default value
Path to the KEY file (when TLS is enabled)
- ENV : DOCKER_KEY_PATH
- Beats variable : input.tls.key_path
- Default value : no default value

When launching it inside a docker container, you can modify the environment variables using the -e flag :

docker run -d \
  -v /var/run/docker.sock:/another/path.sock  \
  --link elastic1:es1 \
  --link elastic2:es2 \
  -e PERIOD=30 \
  -e DOCKER_SOCKET=unix:///another/path.sock \
  ingensi/dockbeat:1.0.0-rc3

Contribute to the project

All contribs are welcome! Read the CONTRIBUTING documentation to get more information.

TL;DR

I want to monitor a host : (If kibana can't join elastic, check its network configuration.)

$ docker network create dockernet

$ docker run -d --net=dockernet --name=elastic \
  -v /mnt/volumes/elastic/config:/usr/share/elasticsearch/config \
  -v /mnt/volumes/elastic/data:/usr/share/elasticsearch/data \
  elasticsearch:2.2.0

$ docker run -d --net=dockernet --name=kibana -p 5601:5601 \
  -e ELASTICSEARCH_URL=http://elastic:9200 \
  kibana:4.4.1

$ docker run -d --net=dockernet --name=dockbeat \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /mnt/dv/dockbeat:/etc/dockbeat ingensi/dockbeat:latest

dockbeat's People

Contributors

Stargazers

Watchers

dockbeat's Issues

Gather and/or duplicate metrics container

We have to choose between:

gather all statistics (cpu, blkio, container, etc.) into a unique and big event
remove the container event, and duplicate container information to all other metrics

We should compare to topbeat and packetbeat to see their logics.

Should Dockerbeat be able to monitor many docker daemons?

The question is: if we have many docker daemons to monitor,

should we have to launch as many dockerbeat as docker daemon
or should we configure one dockerbeat to monitor many docker daemons (socket configuration property could be modified to sockets and be an array of sockets)

This case could append when there is many docker daemons deployed on several machines and listening on TCP. It could be usefull to centralize stats collect on one dedicated host with only one dockerbeat instance, pulling many TCP sockets.

Related to #58 (to identify the docker daemon) and #58 (to allow remote connection to docker daemon).

Connection to Docker via TCP/TLS

Dockerbeat only allows connection to docker daemon socket. It should also be able to connect to listening TCP daemon.

Change to new Libbeat Github repo

All the beats have merged into one github repos. The Libbeat deps should be changed to that repos.

See https://www.elastic.co/blog/the-beats-are-moving-together-into-single-git-repository for annoucement.
See https://github.com/elastic/beats for new repos

Blkio stats should gather "per second" statistics

Currently, dockerbeat sends blkio data as number of reads/writes/total minus previous tick data. It should divides by the number of seconds to emit a "per second" data (like network and proc data).

panic after 30 minutes running

Hi,

We are trying to run develop branch on our servers, but are getting after 30 minutes panics, and the container stops. Seems a deadlock situation..

We are running this command:
docker run -d -v /var/run/docker.sock:/var/run/docker.sock --link elasticsearch-001:elasticsearch ingensi/dockerbeat:develop

docker version:

Client:
Version: 1.9.1
API version: 1.21
Go version: go1.4.2
Git commit: a34a1d5
Built: Fri Nov 20 13:12:04 UTC 2015
OS/Arch: linux/amd64
Server:
Version: 1.9.1
API version: 1.21
Go version: go1.4.2
Git commit: a34a1d5
Built: Fri Nov 20 13:12:04 UTC 2015
OS/Arch: linux/amd64[/quote]

Attached you find the error log
dockerbeat-fail.zip

Fatal error: concurrent map writes

I've just launched your last development version and in a few minutes it threw following exception:

$ sudo ./dockerbeat
fatal error: concurrent map writes

goroutine 2529 [running]:
runtime.throw(0xa32d20, 0x15)
    /usr/local/go/src/runtime/panic.go:530 +0x90 fp=0xc8203c3b00 sp=0xc8203c3ae8
runtime.mapassign1(0x8459a0, 0xc82018e720, 0xc82008d290, 0xc8203c3d90)
    /usr/local/go/src/runtime/hashmap.go:445 +0xb1 fp=0xc8203c3ba8 sp=0xc8203c3b00
github.com/ingensi/dockerbeat/beat.(*EventGenerator).getBlkioEvent(0xc82008c0d8, 0xc82008d290, 0xc820211b80, 0xc82032ff20)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/beat/EventGenerator.go:225 +0x93f fp=0xc8203c3e28 sp=0xc8203c3ba8
github.com/ingensi/dockerbeat/beat.(*Dockerbeat).exportContainerStats.func2(0xc82034efc0, 0xc82034f080, 0xc82008c090, 0xc82008d290)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/beat/Dockerbeat.go:184 +0x1c5 fp=0xc8203c3f90 sp=0xc8203c3e28
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc8203c3f98 sp=0xc8203c3f90
created by github.com/ingensi/dockerbeat/beat.(*Dockerbeat).exportContainerStats
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/beat/Dockerbeat.go:197 +0x210

goroutine 1 [chan receive, 22 minutes]:
github.com/elastic/beats/libbeat/beat.Run(0x9e07e0, 0xa, 0x9d6d30, 0xb, 0x7fe11aaa0bd0, 0xc82008c090, 0x0, 0x0)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/beat/beat.go:126 +0xc2
main.main()
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/main.go:14 +0xa5

goroutine 17 [syscall, 22 minutes, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 5 [syscall, 22 minutes]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 34 [select]:
github.com/elastic/beats/libbeat/publisher.(*messageWorker).run(0xc82016e300)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/publisher/worker.go:61 +0x44a
created by github.com/elastic/beats/libbeat/publisher.(*messageWorker).init
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/publisher/worker.go:55 +0x11e

goroutine 7 [select]:
github.com/ingensi/dockerbeat/beat.(*Dockerbeat).Run(0xc82008c090, 0xc82008c120, 0x0, 0x0)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/beat/Dockerbeat.go:101 +0x4e8
github.com/elastic/beats/libbeat/beat.(*Beat).Run(0xc82008c120, 0x0, 0x0)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/beat/beat.go:250 +0x335
github.com/elastic/beats/libbeat/beat.(*Beat).Start(0xc82008c120, 0x0, 0x0)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/beat/beat.go:162 +0x10f
github.com/elastic/beats/libbeat/beat.Run.func1(0xc82008c120)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/beat/beat.go:110 +0x42
created by github.com/elastic/beats/libbeat/beat.Run
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/beat/beat.go:122 +0x8d

goroutine 35 [select]:
github.com/elastic/beats/libbeat/publisher.(*bulkWorker).run(0xc8201d2000)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/publisher/bulk.go:57 +0x3cb
created by github.com/elastic/beats/libbeat/publisher.newBulkWorker
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/publisher/bulk.go:41 +0x20a

goroutine 36 [select, 22 minutes, locked to thread]:
runtime.gopark(0xacf2b8, 0xc8201d8f28, 0x9d2b78, 0x6, 0x18, 0x2)
    /usr/local/go/src/runtime/proc.go:262 +0x163
runtime.selectgoImpl(0xc8201d8f28, 0x0, 0x18)
    /usr/local/go/src/runtime/select.go:392 +0xa67
runtime.selectgo(0xc8201d8f28)
    /usr/local/go/src/runtime/select.go:215 +0x12
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal1_unix.go:279 +0x358
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 37 [chan receive, 22 minutes]:
github.com/elastic/beats/libbeat/service.HandleSignals.func1(0xc8201a30e0, 0xc82018cb40, 0xc82018cb30)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/service/service.go:29 +0x3a
created by github.com/elastic/beats/libbeat/service.HandleSignals
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/service/service.go:32 +0x21e

goroutine 133 [select]:
net/http.(*persistConn).writeLoop(0xc8202ac1a0)
    /usr/local/go/src/net/http/transport.go:1273 +0x472
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:854 +0x10cb

goroutine 132 [IO wait]:
net.runtime_pollWait(0x7fe11aaa1478, 0x72, 0xc820229000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc820054300, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820054300, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8200542a0, 0xc820229000, 0x1000, 0x1000, 0x0, 0x7fe11aa9c028, 0xc82000a120)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc820028020, 0xc820229000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7fe11aa202c0, 0xc820028020, 0xc8202ac208, 0xc820229000, 0x1000, 0x1000, 0x404df3, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc820168ac0, 0xc820229000, 0x1000, 0x1000, 0xc8203c7d1d, 0x0, 0x0)
    <autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc820184420)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc820184420, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8202ac1a0)
    /usr/local/go/src/net/http/transport.go:1069 +0x177
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:853 +0x10a6

goroutine 1875 [select]:
net/http.(*persistConn).writeLoop(0xc820246270)
    /usr/local/go/src/net/http/transport.go:1273 +0x472
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:854 +0x10cb

goroutine 1874 [IO wait]:
net.runtime_pollWait(0x7fe11aaa1538, 0x72, 0xc820295000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc820121fe0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820121fe0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc820121f80, 0xc820295000, 0x1000, 0x1000, 0x0, 0x7fe11aa9c028, 0xc82000a120)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc82025eb40, 0xc820295000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7fe11aa5c000, 0xc82025eb40, 0xc8202462d8, 0xc820295000, 0x1000, 0x1000, 0x404df3, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc820446960, 0xc820295000, 0x1000, 0x1000, 0xc820335d1d, 0x0, 0x0)
    <autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc820263d40)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc820263d40, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc820246270)
    /usr/local/go/src/net/http/transport.go:1069 +0x177
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:853 +0x10a6
[cloud-user@mi-worker-001 dockerbeat]$ sudo ./dockerbeat
fatal error: concurrent map writes

goroutine 924 [running]:
runtime.throw(0xa32d20, 0x15)
    /usr/local/go/src/runtime/panic.go:530 +0x90 fp=0xc820247b00 sp=0xc820247ae8
runtime.mapassign1(0x8459a0, 0xc820061f50, 0xc8202cefc0, 0xc820247d90)
    /usr/local/go/src/runtime/hashmap.go:445 +0xb1 fp=0xc820247ba8 sp=0xc820247b00
github.com/ingensi/dockerbeat/beat.(*EventGenerator).getBlkioEvent(0xc820176048, 0xc8202cefc0, 0xc82030eb00, 0xc8202ca390)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/beat/EventGenerator.go:225 +0x93f fp=0xc820247e28 sp=0xc820247ba8
github.com/ingensi/dockerbeat/beat.(*Dockerbeat).exportContainerStats.func2(0xc8202686c0, 0xc820268780, 0xc820176000, 0xc8202cefc0)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/beat/Dockerbeat.go:184 +0x1c5 fp=0xc820247f90 sp=0xc820247e28
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820247f98 sp=0xc820247f90
created by github.com/ingensi/dockerbeat/beat.(*Dockerbeat).exportContainerStats
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/beat/Dockerbeat.go:197 +0x210

goroutine 1 [chan receive, 7 minutes]:
github.com/elastic/beats/libbeat/beat.Run(0x9e07e0, 0xa, 0x9d6d30, 0xb, 0x7f7ac2b291e0, 0xc820176000, 0x0, 0x0)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/beat/beat.go:126 +0xc2
main.main()
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/main.go:14 +0xa5

goroutine 17 [syscall, 7 minutes, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 5 [syscall, 7 minutes]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 18 [select, 1 minutes]:
github.com/ingensi/dockerbeat/beat.(*Dockerbeat).Run(0xc820176000, 0xc820176090, 0x0, 0x0)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/beat/Dockerbeat.go:101 +0x4e8
github.com/elastic/beats/libbeat/beat.(*Beat).Run(0xc820176090, 0x0, 0x0)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/beat/beat.go:250 +0x335
github.com/elastic/beats/libbeat/beat.(*Beat).Start(0xc820176090, 0x0, 0x0)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/beat/beat.go:162 +0x10f
github.com/elastic/beats/libbeat/beat.Run.func1(0xc820176090)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/beat/beat.go:110 +0x42
created by github.com/elastic/beats/libbeat/beat.Run
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/beat/beat.go:122 +0x8d

goroutine 22 [IO wait, 1 minutes]:
net.runtime_pollWait(0x7f7ac2b29cc8, 0x72, 0xc8201c4000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc820147f70, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820147f70, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc820147f10, 0xc8201c4000, 0x1000, 0x1000, 0x0, 0x7f7ac2ba9028, 0xc82000a120)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc82013c0e8, 0xc8201c4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f7ac2b2a608, 0xc82013c0e8, 0xc820145cd8, 0xc8201c4000, 0x1000, 0x1000, 0x404df3, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc8201c20a0, 0xc8201c4000, 0x1000, 0x1000, 0xc820034d1d, 0x0, 0x0)
    <autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc820143740)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc820143740, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc820145c70)
    /usr/local/go/src/net/http/transport.go:1069 +0x177
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:853 +0x10a6

goroutine 23 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc820145c70)
    /usr/local/go/src/net/http/transport.go:1273 +0x472
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:854 +0x10cb

goroutine 7 [select, 1 minutes]:
github.com/elastic/beats/libbeat/publisher.(*messageWorker).run(0xc820001b00)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/publisher/worker.go:61 +0x44a
created by github.com/elastic/beats/libbeat/publisher.(*messageWorker).init
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/publisher/worker.go:55 +0x11e

goroutine 8 [runnable]:
github.com/elastic/beats/libbeat/publisher.(*bulkWorker).onEvents(0xc820054d20, 0x0, 0x0, 0xc8202be4a0, 0x4, 0x4)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/publisher/bulk.go:83
github.com/elastic/beats/libbeat/publisher.(*bulkWorker).run(0xc820054d20)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/publisher/bulk.go:63 +0x352
created by github.com/elastic/beats/libbeat/publisher.newBulkWorker
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/publisher/bulk.go:41 +0x20a

goroutine 9 [select, 7 minutes, locked to thread]:
runtime.gopark(0xacf2b8, 0xc820027f28, 0x9d2b78, 0x6, 0x18, 0x2)
    /usr/local/go/src/runtime/proc.go:262 +0x163
runtime.selectgoImpl(0xc820027f28, 0x0, 0x18)
    /usr/local/go/src/runtime/select.go:392 +0xa67
runtime.selectgo(0xc820027f28)
    /usr/local/go/src/runtime/select.go:215 +0x12
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal1_unix.go:279 +0x358
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 10 [chan receive, 7 minutes]:
github.com/elastic/beats/libbeat/service.HandleSignals.func1(0xc82005f3e0, 0xc82021a250, 0xc82021a240)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/service/service.go:29 +0x3a
created by github.com/elastic/beats/libbeat/service.HandleSignals
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/service/service.go:32 +0x21e

goroutine 79 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc8202540d0)
    /usr/local/go/src/net/http/transport.go:1273 +0x472
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:854 +0x10cb

goroutine 78 [IO wait, 1 minutes]:
net.runtime_pollWait(0x7f7ac2b29c08, 0x72, 0xc8204ba000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8204a2300, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8204a2300, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8204a22a0, 0xc8204ba000, 0x1000, 0x1000, 0x0, 0x7f7ac2ba9028, 0xc82000a120)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc820028658, 0xc8204ba000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f7ac2bb1c20, 0xc820028658, 0xc820254138, 0xc8204ba000, 0x1000, 0x1000, 0x404df3, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc8200c08c0, 0xc8204ba000, 0x1000, 0x1000, 0xc820256d1d, 0x0, 0x0)
    <autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc82043f860)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc82043f860, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8202540d0)
    /usr/local/go/src/net/http/transport.go:1069 +0x177
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:853 +0x10a6
[cloud-user@mi-worker-001 dockerbeat]$ sudo ./dockerbeat
fatal error: concurrent map writes

goroutine 1323 [running]:
runtime.throw(0xa32d20, 0x15)
    /usr/local/go/src/runtime/panic.go:530 +0x90 fp=0xc820269b00 sp=0xc820269ae8
runtime.mapassign1(0x8459a0, 0xc8201c0720, 0xc82017e750, 0xc820269d90)
    /usr/local/go/src/runtime/hashmap.go:445 +0xb1 fp=0xc820269ba8 sp=0xc820269b00
github.com/ingensi/dockerbeat/beat.(*EventGenerator).getBlkioEvent(0xc82016c048, 0xc82017e750, 0xc8201a18c0, 0xc82012e9f0)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/beat/EventGenerator.go:225 +0x93f fp=0xc820269e28 sp=0xc820269ba8
github.com/ingensi/dockerbeat/beat.(*Dockerbeat).exportContainerStats.func2(0xc820523f80, 0xc8200da060, 0xc82016c000, 0xc82017e750)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/beat/Dockerbeat.go:184 +0x1c5 fp=0xc820269f90 sp=0xc820269e28
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820269f98 sp=0xc820269f90
created by github.com/ingensi/dockerbeat/beat.(*Dockerbeat).exportContainerStats
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/beat/Dockerbeat.go:197 +0x210

goroutine 1 [chan receive, 11 minutes]:
github.com/elastic/beats/libbeat/beat.Run(0x9e07e0, 0xa, 0x9d6d30, 0xb, 0x7fe48f96f150, 0xc82016c000, 0x0, 0x0)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/beat/beat.go:126 +0xc2
main.main()
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/main.go:14 +0xa5

goroutine 17 [syscall, 11 minutes, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 5 [syscall, 11 minutes]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 34 [select]:
github.com/elastic/beats/libbeat/publisher.(*messageWorker).run(0xc820060300)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/publisher/worker.go:61 +0x44a
created by github.com/elastic/beats/libbeat/publisher.(*messageWorker).init
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/publisher/worker.go:55 +0x11e

goroutine 18 [select]:
github.com/ingensi/dockerbeat/beat.(*Dockerbeat).Run(0xc82016c000, 0xc82016c090, 0x0, 0x0)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/beat/Dockerbeat.go:101 +0x4e8
github.com/elastic/beats/libbeat/beat.(*Beat).Run(0xc82016c090, 0x0, 0x0)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/beat/beat.go:250 +0x335
github.com/elastic/beats/libbeat/beat.(*Beat).Start(0xc82016c090, 0x0, 0x0)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/beat/beat.go:162 +0x10f
github.com/elastic/beats/libbeat/beat.Run.func1(0xc82016c090)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/beat/beat.go:110 +0x42
created by github.com/elastic/beats/libbeat/beat.Run
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/beat/beat.go:122 +0x8d

goroutine 35 [runnable]:
github.com/elastic/beats/libbeat/publisher.(*bulkWorker).run(0xc820204000)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/publisher/bulk.go:57 +0x134
created by github.com/elastic/beats/libbeat/publisher.newBulkWorker
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/publisher/bulk.go:41 +0x20a

goroutine 36 [select, 11 minutes, locked to thread]:
runtime.gopark(0xacf2b8, 0xc82020af28, 0x9d2b78, 0x6, 0x18, 0x2)
    /usr/local/go/src/runtime/proc.go:262 +0x163
runtime.selectgoImpl(0xc82020af28, 0x0, 0x18)
    /usr/local/go/src/runtime/select.go:392 +0xa67
runtime.selectgo(0xc82020af28)
    /usr/local/go/src/runtime/select.go:215 +0x12
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal1_unix.go:279 +0x358
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 37 [chan receive, 11 minutes]:
github.com/elastic/beats/libbeat/service.HandleSignals.func1(0xc8201d50e0, 0xc8201beb40, 0xc8201beb30)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/service/service.go:29 +0x3a
created by github.com/elastic/beats/libbeat/service.HandleSignals
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/service/service.go:32 +0x21e

goroutine 123 [select]:
net/http.(*persistConn).writeLoop(0xc8204d40d0)
    /usr/local/go/src/net/http/transport.go:1273 +0x472
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:854 +0x10cb

goroutine 122 [IO wait]:
net.runtime_pollWait(0x7fe48f96fb78, 0x72, 0xc82015a000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc820054290, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820054290, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc820054230, 0xc82015a000, 0x1000, 0x1000, 0x0, 0x7fe48f9af028, 0xc82000a120)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc8202de000, 0xc82015a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7fe48f9706c8, 0xc8202de000, 0xc8204d4138, 0xc82015a000, 0x1000, 0x1000, 0x404df3, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc8201b8280, 0xc82015a000, 0x1000, 0x1000, 0xc82026ad1d, 0x0, 0x0)
    <autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc8201d44e0)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc8201d44e0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8204d40d0)
    /usr/local/go/src/net/http/transport.go:1069 +0x177
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:853 +0x10a6

goroutine 941 [select]:
net/http.(*persistConn).writeLoop(0xc8204d41a0)
    /usr/local/go/src/net/http/transport.go:1273 +0x472
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:854 +0x10cb

goroutine 940 [IO wait]:
net.runtime_pollWait(0x7fe48f96fc38, 0x72, 0xc8202cd000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc820205800, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820205800, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8202057a0, 0xc8202cd000, 0x1000, 0x1000, 0x0, 0x7fe48f9af028, 0xc82000a120)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc8202df930, 0xc8202cd000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7fe48f970578, 0xc8202df930, 0xc8204d4208, 0xc8202cd000, 0x1000, 0x1000, 0x404df3, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc8202d1020, 0xc8202cd000, 0x1000, 0x1000, 0xc8203cfd1d, 0x0, 0x0)
    <autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc82040be00)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc82040be00, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8204d41a0)
    /usr/local/go/src/net/http/transport.go:1069 +0x177
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:853 +0x10a6

goroutine 1339 [runnable]:
github.com/fsouza/go-dockerclient.(*Client).Stats.func3(0xc8201ad300, 0x40, 0xc8200da0c0, 0xc8200da000, 0xc8200da120, 0xffffffffffffffff, 0xc8202de670, 0xc82017cc00)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/fsouza/go-dockerclient/container.go:798 +0xe4
created by github.com/fsouza/go-dockerclient.(*Client).Stats
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/fsouza/go-dockerclient/container.go:804 +0x3a7

goroutine 1327 [runnable]:
github.com/fsouza/go-dockerclient.(*Client).Stats.func3(0xc8201ad240, 0x40, 0xc820523f80, 0xc820523f00, 0xc8200da000, 0xffffffffffffffff, 0xc820134cd8, 0xc8200da240)
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/fsouza/go-dockerclient/container.go:798 +0xe4
created by github.com/fsouza/go-dockerclient.(*Client).Stats
    /home/cloud-user/.go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/fsouza/go-dockerclient/container.go:804 +0x3a7

Issue starting dockerbeat with ES_HOSTS

Hello,

We are getting an error when starting dockerbeat with the ES_HOSTS parameter in our docker run (our elasticsearch cluster is not running in docker and on other machines, so we use ES_HOSTS):
docker run --net bridge -m 0b --detach=true -e ES_HOSTS=xxx.xxxxxx.xxx.domain.tld -v /var/run/docker.sock:/var/run/docker.sock --name dockerbeat-app00 ingensi/dockerbeat:develop

Start error: loading config file error: YAML config parsing failed on dockerbeat.yml: yaml: unmarshal errors:
line 29: cannot unmarshal !!str xxx-xx... into []string. Exiting.

panic: close of nil channel

goroutine 1 [running]:
github.com/ingensi/dockerbeat/beat.(_Dockerbeat).Stop(0xc82014ae10)
/go/src/github.com/ingensi/dockerbeat/beat/Dockerbeat.go:122 +0x24
github.com/elastic/beats/libbeat/beat.(_Beat).Stop(0xc82014aea0)
/go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/beat/beat.go:262 +0x77
github.com/elastic/beats/libbeat/beat.Run(0x95e790, 0xa, 0x955710, 0xb, 0x7fe7940a51b8, 0xc82014ae10, 0x0, 0x0)
/go/src/github.com/ingensi/dockerbeat/Godeps/_workspace/src/github.com/elastic/beats/libbeat/beat/beat.go:127 +0xd0
main.main()
/go/src/github.com/ingensi/dockerbeat/main.go:14 +0xa4

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 5 [syscall]:
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
/usr/local/go/src/os/signal/signal_unix.go:28 +0x37

Kr,
Jarko

Next release?

Hi @marminthibaut

Do you know ETA for next release?

Add a "dockerSocket" field

Add the dockerSocket field to all shipped events. It should be usefull when many dockerbeat are launched on the same host to monitor different dockers daemons.

Per CPU stat is just an array

It's currently impossible to create a chart to visualize each cpu consumption, because the metric 'perCpuUsage' is only an array.

It can be possible to construct a map with the cpu name and its value.

Code refactoring

The app should be tested...

Set some common parameters via environment

Instead of creating custom a config file just to set a non-default elasticsearch port for example, it would be more convenient to be able to set it via the environment variables.
Like this:

docker run -d ... -e HOST=192.168.0.1 -e PORT=9999

See the example of this approach here: https://github.com/athieriot/docker-elastic-topbeat

I suggest to support this list of variables: PERIOD, HOST, PORT, SHIPPER_NAME, SHIPPER_TAGS.

What about Dockerbeat and Swarm?

See #59 @gpolaert comment:

I'm wondering if there is a place for swarm and dockerbeat. Swarm can provide some information about nodes, ip, cpu, and so on, plus containers
2 thoughts:

Can we handles beats via swarm/dockerbeat? Is it interesting (not sure)?
Do we have to provide a json swarm event if the socket is a instance of swarm node?

First network stats does not push well named fields

The first network stats after dockerbeat startup is not well formatted:

event = common.MapStr{
            "timestamp":      common.Time(stats.Read),
            "type":           "net",
            "containerID":    container.ID,
            "containerNames": container.Names,
            "net": common.MapStr{
                "rxBytes":   0,
                "rxDropped": 0,
                "rxErrors":  0,
                "rxPackets": 0,
                "txBytes":   0,
                "txDropped": 0,
                "txErrors":  0,
                "txPackets": 0,
            },
        }

Fields should be suffixed by "_ps" to match with the template.

Manage errors due to a too old version of Docker

Dockerbeat uses docker stats api, which has been introduced from 1.5 version.

Dockerbeat should check docker version on startup and display compatibility error message if docker daemon is not up to date.

Network type output.

Can't find information about network type in a container (like bridge or host). Is there any support for this function?

Docker version constraint

Currently, development version has the constraint for docker version upper than 1.9. What is the reason of this? I've tried to run it without this constraint and everything works fine.

Network stats could be enhanced

Network stats should include a bandwidth instead of Rx/Tx values.

Clean code :)

Clean the code:

remove useless log debug
...

Percentage fields should be between 0.0 and 1.0

Currently, percentage fields are between 0.0 and 100.0, to use kibana field formatting properly, it should be between 0.0 and 1.0.

No network stats

We are running dockerbeat v1.0.0-beta2, and we've found that there are no network statistics, and no any error message.
logs with debug level: https://gist.github.com/hcoa/335c1c26ec011adc4394
docker api version 1.20

Minimal docker version should be updated to 1.9.x

Reference for #71

Dockerbeat should display an error message when docker is too old. Required version is 1.9.x (API version > 1.21)

Sample Kibana dashboards

It would be very useful if you could provide some Kibana templates for dockerbeat.
Like Elastic does for topbeat: https://github.com/elastic/beats-dashboards

Memory metrics should show a percentage

It could be usefull to have percent of memory used by container, instead of raw values.

cpu,blkio,network value type conflicts with 1.0.0-beta2

Using Docker image ingensi/dockerbeat:1.0.0-beta2.

Seems the input data has long type, while the index mapping requires double.

logs are:

elasticsearch_1 | [2016-01-28 02:33:45,085][DEBUG][action.admin.indices.mapping.put] [Cloud] failed to put mappings on indices [[dockerbeat-2016.01.28]], typ
e [cpu]
elasticsearch_1 | MergeMappingException[Merge failed with failures {[mapper [cpu.percpuUsage.cpu3] of different type, current_type [double], merged_type [lon
g], mapper [cpu.totalUsage] of different type, current_type [double], merged_type [long]]}]
elasticsearch_1 | at org.elasticsearch.cluster.metadata.MetaDataMappingService$2.execute(MetaDataMappingService.java:392)
elasticsearch_1 | at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:388)
elasticsearch_1 | at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsT
hreadPoolExecutor.java:231)
elasticsearch_1 | at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoo
lExecutor.java:194)
elasticsearch_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
elasticsearch_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
elasticsearch_1 | at java.lang.Thread.run(Thread.java:745)
elasticsearch_1 | [2016-01-28 02:33:45,088][DEBUG][action.bulk ] [Cloud] [dockerbeat-2016.01.28][2] failed to execute bulk item (index) index {[
dockerbeat-2016.01.28][cpu][AVKGFE_njViCxbrU49UJ], source[{"@timestamp":"2016-01-28T02:33:38.179Z","beat":{"hostname":"0802849db28f","name":"0802849db28f"},"
containerID":"21d8b2f4b250012a731ebb68f8458713f492c9247fb58b24da2092302aa0600f","containerName":"dockerbeatek_kibana_1","count":1,"cpu":{"percpuUsage":{"cpu0
":0,"cpu1":0,"cpu2":0,"cpu3":0},"totalUsage":0,"usageInKernelmode":0,"usageInUsermode":0},"type":"cpu"}]}
elasticsearch_1 | MergeMappingException[Merge failed with failures {[mapper [cpu.percpuUsage.cpu3] of different type, current_type [double], merged_type [lon
g], mapper [cpu.totalUsage] of different type, current_type [double], merged_type [long]]}]
elasticsearch_1 | at org.elasticsearch.cluster.metadata.MetaDataMappingService$2.execute(MetaDataMappingService.java:392)
elasticsearch_1 | at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:388)
elasticsearch_1 | at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsT
hreadPoolExecutor.java:231)
elasticsearch_1 | at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoo
lExecutor.java:194)
elasticsearch_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
elasticsearch_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
elasticsearch_1 | at java.lang.Thread.run(Thread.java:745)
elasticsearch_1 | [2016-01-28 02:33:45,098][DEBUG][action.admin.indices.mapping.put] [Cloud] failed to put mappings on indices [[dockerbeat-2016.01.28]], typ
e [blkio]
elasticsearch_1 | MergeMappingException[Merge failed with failures {[mapper [blkio.total_ps] of different type, current_type [long], merged_type [double], ma
pper [blkio.write_ps] of different type, current_type [long], merged_type [double]]}]
elasticsearch_1 | at org.elasticsearch.cluster.metadata.MetaDataMappingService$2.execute(MetaDataMappingService.java:392)
elasticsearch_1 | at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:388)
elasticsearch_1 | at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsT
hreadPoolExecutor.java:231)
elasticsearch_1 | at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoo
lExecutor.java:194)
elasticsearch_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
elasticsearch_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
elasticsearch_1 | at java.lang.Thread.run(Thread.java:745)

Percentage values should be set between 0 and 1

Currently, percentage values are set between 0 and 100, but kibana field format only supports percentages between 0 and 1 out of the box...

CPU stats should show load percentage

Dockerbeat should send cpu load (percentage) instead of raw values (usages).

Logstash output?

Didn't find this one in README, are we able to output to logstash?

Dependency (go-dockerclient) has been updated

The go-dockerclient project have change for the new Docker API (1.22).

The Stat object have change : see here

The impact on Dockerbeat is that it changes the event struct.

Setup doc field generation

Generate asciidoc from scripts/generate_fields_doc.py script.

Report Docker label

Hi all,

I would like to use dockerbeat in a multitenant environement.
Do you plan to return Label keys (https://docs.docker.com/engine/userguide/labels-custom-metadata/) ?

That's will be helpfull to differentiate customers

Best regards,
Hug

Improve README content

add a TLDR
rewrite some parts
etc.

Display proper error message when docker daemon is unreachable

When dockerbeat successfully runs and docker daemon does not respond, dockerbeat should properly log error message with logp.

Some message should be displayed:

when dockerbeat cannot access to docker socket
when dockerbeat can access to docker socker, but an error occurred on listing containers
when dockerbeat can access to docker socker, but an error occurred on getting container stats

Errors due to an unreachable elasticsearch node should be managed by libbeat.

Should dockerbeat allow to monitor container logs?

The question is: Does the dockerbeat role is to monitor only container statistics or should also ship container logs ?

Dockerbeat crashes when docker API is not reachable

Related issue: #60

Reproduce: launch dockerbeat monitoring a running docker daemon and then stop the docker daemon.

Solve problem: check nil values in EventGenerator

Mapping should set fields as "not analyzed"

To avoid splitting field problem :)

see topbeat project

Add new document type for docker daemon status

Dockerbeat could ship an event type for docker daemon status by pulling the /info endpoint : https://docs.docker.com/engine/reference/api/docker_remote_api_v1.21/#2-3-misc

When API is unreachable, we could ship an event with special status. (need discuss)

Wrong cpu type event in README file

Found this line in README:
type: cpu for per process container statistics.
but example type: cpu output don't have anything related to processes

Can you clarify this?
Thanks

Sanitize docker labels containing dots

Docker-compose generates some container labels like 'com.docker.compose.container-number' and 'com.docker.compose.project' among others. These are invalid field names since ES 2.0 does not support dot in field names. It would be nice if those names are sanitized using '_' or any other valid character.

I checked docker-compose documentation and source and I wasn't able to find a way to disable that label generation.

Code comments

Code should includes some comments

Add blkio stats

Dockerbeat should collect blkio statistics. Stats should be human-readable i.e should be read/write Bytes per second.

Mapping volume to config directory causes executable to be lost

Hey, I am just getting started with dockerbeat, but I like the way it looks. Ran into one issue right off the bat. In your documentation, it says that to map a custom config file simply mount a volume:

-v /src/beats/dockerbeat/config/:/etc/dockerbeat

However, when you do that, you simply overwrite the executable, and then it cannot launch. Not a huge ordeal, I can simply use COPY in the Dockerfile or change the shared directory and update the entrypoint. Just thought I would let you all know. Thanks

scripts/generate_field_docs.py
scripts/generate_template.py

Python scripts used to generate the etc/dockerbeat.template.json file.

Identify container with human readable names

It will allow to understand more easily kibana dashboards...

Container type event should parse status field

Container events have a status field, formatted as string ("Up 36 minutes").

Dockerbeat could parse this string to extract two nested fields:

state ("up", "stopped", etc.)
time (in seconds: "2160")

Fields names are not definitive.

Add dockerbeat to AUR repo

Could we push dockerbeat to AUR repo (for Arch distrib) like topbeat, filebeat and so on ?

container.ports (container type) is malformed when container does not have any published port

When a monitored container does not publish any port, then the container.ports field take the string value "-". It should be an empty array [].

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.