bwalex / fluent-plugin-docker-format Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT License
License: MIT License
When time allows could you please bump rubygems, thanks!
The last tag doesn't include changes (that are already on the development branch) to handle config.v2.json files. Is it possible a new tag and release could be taken?
So I'm trying to get proper json log combining with fluentd and I'm wondering: is this to be used separately from the fluentd docker log driver?
Your examples require path /var/lib/docker/containers/*/*-json.log
and I'm not sure if those files are created when the fluentd log driver is used instead of the json file driver. If this needs the json-file log driver, do I need to mount the path
into the fluentd container?
Thanks!
The name of the container config JSON file seems to have changed as of docker 1.10.1. It's now called config.v2.json
. See https://github.com/bwalex/fluent-plugin-docker-format/blob/master/lib/fluent/plugin/out_docker_format.rb#L41
2016-09-20 12:54:06 +0000 [warn]: emit transaction failed: error_class=RuntimeError error="BUG: use router.emit instead of Engine.emit" tag="debian:latest"
2016-09-20 12:54:06 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.1/lib/fluent/engine.rb:132:in `emit'
2016-09-20 12:54:06 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-docker-format-0.2.3/lib/fluent/plugin/out_docker_format.rb:17:in `block in emit'
2016-09-20 12:54:06 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.1/lib/fluent/event.rb:96:in `each'
2016-09-20 12:54:06 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-docker-format-0.2.3/lib/fluent/plugin/out_docker_format.rb:16:in `emit'
2016-09-20 12:54:06 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.1/lib/fluent/compat/output.rb:151:in `process'
2016-09-20 12:54:06 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.1/lib/fluent/plugin/output.rb:520:in `emit_sync'
2016-09-20 12:54:06 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.1/lib/fluent/event_router.rb:90:in `emit_stream'
2016-09-20 12:54:06 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.1/lib/fluent/event_router.rb:81:in `emit'
2016-09-20 12:54:06 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.1/lib/fluent/plugin/in_forward.rb:205:in `on_message'
2016-09-20 12:54:06 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.1/lib/fluent/plugin/in_forward.rb:295:in `call'
2016-09-20 12:54:06 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.1/lib/fluent/plugin/in_forward.rb:295:in `block in on_read_msgpack'
2016-09-20 12:54:06 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.1/lib/fluent/plugin/in_forward.rb:294:in `feed_each'
2016-09-20 12:54:06 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.1/lib/fluent/plugin/in_forward.rb:294:in `on_read_msgpack'
2016-09-20 12:54:06 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.1/lib/fluent/plugin/in_forward.rb:280:in `call'
2016-09-20 12:54:06 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.1/lib/fluent/plugin/in_forward.rb:280:in `on_read'
2016-09-20 12:54:06 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/cool.io-1.4.4/lib/cool.io/io.rb:123:in `on_readable'
2016-09-20 12:54:06 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/cool.io-1.4.4/lib/cool.io/io.rb:186:in `on_readable'
2016-09-20 12:54:06 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/cool.io-1.4.4/lib/cool.io/loop.rb:88:in `run_once'
2016-09-20 12:54:06 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/cool.io-1.4.4/lib/cool.io/loop.rb:88:in `run'
2016-09-20 12:54:06 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.1/lib/fluent/plugin/in_forward.rb:117:in `run'
2016-09-20 12:54:06 +0000 [error]: forward error error=#<RuntimeError: BUG: use router.emit instead of Engine.emit> error_class=RuntimeError```
I'm having an issue with td-agent and fluent-plugin-docker-format.
td-agent: 2.1.4-0
fluent-plugin-docker-format: 0.2.3
The issue I am seeing is when I run a docker ps on a server running a container, clearly there is a name for the container.
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c36b69fd1481 itriage/bamboo:0.2.4 "/var/bamboo/bamboo 13 days ago Up 51 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:8000->8000/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:9000->9000/tcp bamboo
When I see the logs in ES I see this:
{"log":"2015/02/17 22:05:15.495912 [c36b69fd1481/yDNlspq6bt-000364] Started GET "/" from 10.40.70.96:48837\n","stream":"stderr","container_id":"c36b69fd1481e5a28e8b4ccbe34361e25d08f361562b0d28056505b9c166af64-json","container_name":"","@timestamp":"2015-02-17T22:05:15+00:00"}
Any way to debug 'unknown' container ID?
Here is my td-agent.conf:
<source>
type tail
path /mnt/docker/containers/*/*-json.log
pos_file /var/log/td-agent/fluentd-docker.pos
time_format %Y-%m-%dT%H:%M:%S
tag docker.*
format json
</source>
<match docker.mnt.docker.containers.*.*.log>
type docker_format
container_id ${tag_parts[5]}
tag docker.all
</match>
<match docker.all>
type elasticsearch
logstash_format true
flush_interval 10s # for testing
hosts 10.10.10.10
</match>
Hello,
I have a problem using your plugin with an EFK stack.
Every thing seems OK at startup, the conf is exactly the same as yours, I can see everything up, logs are forwarded to elasticsearch and Kibana retrieves them.
But my container_name remains at value.
I checked your code and did not notice anything where you get container_name.
One thing i noticed is that i have to use sudo to open config.v2.json. Maybe your plugin does not have read access on it ? (see the CONTAINER FOLDER RIGHTS part).
Can you have a look please, and give me some tips to debug your pluign (i am a Java developer...)
@log_name e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b
@timestamp October 18th 2016, 16:31:06.000
t_id AVfYNAJ2sUxcrcA5hnKg
t_index logstash-2016.10.18
t_type fluentd
tcontainer_id e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b
tcontainer_name
tlog �[2m2016-10-18 14:31:06.083�[0;39m �[32m INFO�[0;39m �[35m7�[0;39m �[2m---�[0;39m �[2m[ main]�[0;39m �[36mo.a.camel.spring.SpringCamelContext �[0;39m �[2m:�[0;39m StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html
tstream stdout
fluentd_2 | 2016-10-18T14:29:57.190742756Z Fluentd waithing TCP connection :...
fluentd_2 | 2016-10-18T14:29:58.194248978Z Start fluentd
fluentd_2 | 2016-10-18T14:29:58.461565888Z 2016-10-18 14:29:58 +0000 [info]: fluent/supervisor.rb:471:read_config: reading config file path="/fluentd/etc/fluent.conf"
fluentd_2 | 2016-10-18T14:29:58.464357222Z 2016-10-18 14:29:58 +0000 [info]: fluent/supervisor.rb:337:supervise: starting fluentd-0.12.29
fluentd_2 | 2016-10-18T14:29:58.543209510Z 2016-10-18 14:29:58 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-mixin-config-placeholders' version '0.4.0'
fluentd_2 | 2016-10-18T14:29:58.543257242Z 2016-10-18 14:29:58 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-docker-format' version '0.2.3'
fluentd_2 | 2016-10-18T14:29:58.543317650Z 2016-10-18 14:29:58 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-elasticsearch' version '1.7.0'
fluentd_2 | 2016-10-18T14:29:58.543408305Z 2016-10-18 14:29:58 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-s3' version '0.7.1'
fluentd_2 | 2016-10-18T14:29:58.543525763Z 2016-10-18 14:29:58 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluentd' version '0.12.29'
fluentd_2 | 2016-10-18T14:29:58.543638143Z 2016-10-18 14:29:58 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluentd' version '0.10.61'
fluentd_2 | 2016-10-18T14:29:58.543915201Z 2016-10-18 14:29:58 +0000 [info]: fluent/agent.rb:129:add_match: adding match pattern="docker.var.lib.docker.containers...log" type="docker_format"
fluentd_2 | 2016-10-18T14:29:58.546189569Z 2016-10-18 14:29:58 +0000 [info]: fluent/agent.rb:129:add_match: adding match pattern="" type="copy"
fluentd_2 | 2016-10-18T14:29:58.546478263Z 2016-10-18 14:29:58 +0000 [debug]: plugin/out_copy.rb:44:block in configure: adding store type="elasticsearch"
fluentd_2 | 2016-10-18T14:29:58.654633862Z 2016-10-18 14:29:58 +0000 [debug]: plugin/out_copy.rb:44:block in configure: adding store type="s3"
fluentd_2 | 2016-10-18T14:29:58.706809710Z 2016-10-18 14:29:58 +0000 [info]: fluent/root_agent.rb:147:add_source: adding source type="tail"
fluentd_2 | 2016-10-18T14:29:58.717655521Z 2016-10-18 14:29:58 +0000 [info]: fluent/engine.rb:133:configure: using configuration file:
fluentd_2 | 2016-10-18T14:29:58.717667174Z
fluentd_2 | 2016-10-18T14:29:58.717670825Z log_level debug
fluentd_2 | 2016-10-18T14:29:58.717673963Z
fluentd_2 | 2016-10-18T14:29:58.717677094Z
fluentd_2 | 2016-10-18T14:29:58.717680053Z type tail
fluentd_2 | 2016-10-18T14:29:58.717683043Z path /var/lib/docker/containers//-json.log
fluentd_2 | 2016-10-18T14:29:58.717686178Z pos_file fluentd-docker.pos
fluentd_2 | 2016-10-18T14:29:58.717689108Z time_format %Y-%m-%dT%H:%M:%S
fluentd_2 | 2016-10-18T14:29:58.717692020Z tag docker.*
fluentd_2 | 2016-10-18T14:29:58.717694865Z format json
fluentd_2 | 2016-10-18T14:29:58.717697657Z
fluentd_2 | 2016-10-18T14:29:58.717700640Z <match docker.var.lib.docker.containers...log>
fluentd_2 | 2016-10-18T14:29:58.717703595Z type docker_format
fluentd_2 | 2016-10-18T14:29:58.717706470Z container_id ${tag_parts[5]}
fluentd_2 | 2016-10-18T14:29:58.717718503Z tag docker.${name}
fluentd_2 | 2016-10-18T14:29:58.717724682Z docker_containers_path /var/lib/docker/containers
fluentd_2 | 2016-10-18T14:29:58.717727907Z
fluentd_2 | 2016-10-18T14:29:58.717730840Z <match docker.
fluentd_2 | 2016-10-18T14:29:58.717733834Z @type copy
fluentd_2 | 2016-10-18T14:29:58.717736595Z
fluentd_2 | 2016-10-18T14:29:58.717739495Z @type elasticsearch
fluentd_2 | 2016-10-18T14:29:58.717742347Z host elasticsearch
fluentd_2 | 2016-10-18T14:29:58.717745172Z port 9200
fluentd_2 | 2016-10-18T14:29:58.717747949Z include_tag_key true
fluentd_2 | 2016-10-18T14:29:58.717750819Z tag_key @log_name
fluentd_2 | 2016-10-18T14:29:58.717753656Z logstash_format true
fluentd_2 | 2016-10-18T14:29:58.717756494Z flush_interval 5s
fluentd_2 | 2016-10-18T14:29:58.717759227Z
fluentd_2 | 2016-10-18T14:29:58.717762106Z
fluentd_2 | 2016-10-18T14:29:58.717764943Z @type s3
fluentd_2 | 2016-10-18T14:29:58.717767631Z aws_key_id xxxxxx
fluentd_2 | 2016-10-18T14:29:58.717770376Z aws_sec_key xxxxxx
fluentd_2 | 2016-10-18T14:29:58.717773109Z s3_bucket fluentd
fluentd_2 | 2016-10-18T14:29:58.717775893Z s3_region eu-central-1
fluentd_2 | 2016-10-18T14:29:58.717778631Z path logs/
fluentd_2 | 2016-10-18T14:29:58.717781389Z buffer_path /var/log/fluent/s3
fluentd_2 | 2016-10-18T14:29:58.717784193Z time_slice_format %Y%m%d%H
fluentd_2 | 2016-10-18T14:29:58.717786958Z time_slice_wait 5s
fluentd_2 | 2016-10-18T14:29:58.717789730Z utc
fluentd_2 | 2016-10-18T14:29:58.717793708Z buffer_chunk_limit 256m
fluentd_2 | 2016-10-18T14:29:58.717796698Z
fluentd_2 | 2016-10-18T14:29:58.717799677Z
fluentd_2 | 2016-10-18T14:29:58.717802648Z
fluentd_2 | 2016-10-18T14:29:59.612179183Z 2016-10-18 14:29:59 +0000 [info]: plugin/in_tail.rb:557:initialize: following tail of /var/lib/docker/containers/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b-json.log
[...]$ sudo ls -al /var/lib/docker/containers/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b
total 4644
drwx------ 3 root root 4096 Oct 18 14:30 .
drwx------ 12 root root 20480 Oct 18 14:31 ..
-rw-rw-rw- 1 root root 9708 Oct 18 14:30 config.v2.json
-rw-r----- 1 root root 4692269 Oct 18 14:31 e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b-json.log
-rw-rw-rw- 1 root root 1282 Oct 18 14:30 hostconfig.json
-rw-r--r-- 1 root root 13 Oct 18 14:30 hostname
-rw-r--r-- 1 root root 174 Oct 18 14:30 hosts
-rw-r--r-- 1 root root 109 Oct 18 14:30 resolv.conf
-rw-r--r-- 1 root root 71 Oct 18 14:30 resolv.conf.hash
drwxrwxrwt 2 root root 40 Oct 18 14:30 shm
[...]$ sudo cat /var/lib/docker/containers/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b/config.v2.json
{"State":{"Running":true,"Paused":false,"Restarting":false,"OOMKilled":false,"RemovalInProgress":false,"Dead":false,"Pid":6820,"StartedAt":"2016-10-18T14:30:40.006984652Z","FinishedAt":"2016-10-18T14:30:35.585791768Z","Health":null},"ID":"e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b","Created":"2016-10-17T17:42:42.568174797Z","Managed":false,"Path":"/bin/sh", ........,"Config":{"Hostname":"e47bced8e497","Domainname":"","User":"",
[...]
,"LogPath":"/var/lib/docker/containers/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b-json.log","Name":"/r-stack_my-container","Driver":"overlay","MountLabel":"","ProcessLabel":"","RestartCount":0,"HasBeenStartedBefore":false,"HasBeenManuallyStopped":false,"MountPoints":{"/tmp":{"Source":"","Destination":"/tmp","RW":true,"Name":"3f92ecaad901e05cc4ce24f998a4605a2861590ff22f3924abe594e79c8b8262","Driver":"local","Relabel":"","Propagation":"","Named":false,"ID":"13a83ceabb05a0128342ba0b52db7443de61bd03ed95a36ee9356c66c658af72"}},"AppArmorProfile":"","HostnamePath":"/var/lib/docker/containers/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b/hostname","HostsPath":"/var/lib/docker/containers/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b/hosts","ShmPath":"/var/lib/docker/containers/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b/shm","ResolvConfPath":"/var/lib/docker/containers/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b/resolv.conf","SeccompProfile":"","NoNewPrivileges":false}
[...]$ cat /var/lib/docker/containers/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b
cat: can't open '/var/lib/docker/containers/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b': Permission denied
WHEREAS
sudo cat /var/lib/docker/containers/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b-json.
shows file content
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.