When time allows could you please bump rubygems, thanks!

The last tag doesn't include changes (that are already on the development branch) to handle config.v2.json files. Is it possible a new tag and release could be taken?

So I'm trying to get proper json log combining with fluentd and I'm wondering: is this to be used separately from the fluentd docker log driver?

Your examples require path /var/lib/docker/containers/*/*-json.log and I'm not sure if those files are created when the fluentd log driver is used instead of the json file driver. If this needs the json-file log driver, do I need to mount the path into the fluentd container?

Thanks!

The name of the container config JSON file seems to have changed as of docker 1.10.1. It's now called config.v2.json. See https://github.com/bwalex/fluent-plugin-docker-format/blob/master/lib/fluent/plugin/out_docker_format.rb#L41

I'm having an issue with td-agent and fluent-plugin-docker-format.

td-agent: 2.1.4-0
fluent-plugin-docker-format: 0.2.3

The issue I am seeing is when I run a docker ps on a server running a container, clearly there is a name for the container.

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c36b69fd1481 itriage/bamboo:0.2.4 "/var/bamboo/bamboo 13 days ago Up 51 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:8000->8000/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:9000->9000/tcp bamboo

When I see the logs in ES I see this:

{"log":"2015/02/17 22:05:15.495912 [c36b69fd1481/yDNlspq6bt-000364] Started GET "/" from 10.40.70.96:48837\n","stream":"stderr","container_id":"c36b69fd1481e5a28e8b4ccbe34361e25d08f361562b0d28056505b9c166af64-json","container_name":"","@timestamp":"2015-02-17T22:05:15+00:00"}

Any way to debug 'unknown' container ID?

Here is my td-agent.conf:

<source>
  type tail
  path /mnt/docker/containers/*/*-json.log
  pos_file /var/log/td-agent/fluentd-docker.pos
  time_format %Y-%m-%dT%H:%M:%S
  tag docker.*
  format json
</source>

<match docker.mnt.docker.containers.*.*.log>
  type docker_format
  container_id ${tag_parts[5]}
  tag docker.all
</match>

<match docker.all>
  type elasticsearch
  logstash_format true
  flush_interval 10s # for testing
  hosts 10.10.10.10
</match>

Hello,

I have a problem using your plugin with an EFK stack.

Every thing seems OK at startup, the conf is exactly the same as yours, I can see everything up, logs are forwarded to elasticsearch and Kibana retrieves them.
But my container_name remains at value.
I checked your code and did not notice anything where you get container_name.
One thing i noticed is that i have to use sudo to open config.v2.json. Maybe your plugin does not have read access on it ? (see the CONTAINER FOLDER RIGHTS part).

Can you have a look please, and give me some tips to debug your pluign (i am a Java developer...)

CONF

@log_name e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b
@timestamp October 18th 2016, 16:31:06.000
t_id AVfYNAJ2sUxcrcA5hnKg
t_index logstash-2016.10.18

_score

t_type fluentd
tcontainer_id e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b
tcontainer_name
tlog �[2m2016-10-18 14:31:06.083�[0;39m �[32m INFO�[0;39m �[35m7�[0;39m �[2m---�[0;39m �[2m[ main]�[0;39m �[36mo.a.camel.spring.SpringCamelContext �[0;39m �[2m:�[0;39m StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html
tstream stdout

FLUENTD

fluentd_2 | 2016-10-18T14:29:57.190742756Z Fluentd waithing TCP connection :...
fluentd_2 | 2016-10-18T14:29:58.194248978Z Start fluentd
fluentd_2 | 2016-10-18T14:29:58.461565888Z 2016-10-18 14:29:58 +0000 [info]: fluent/supervisor.rb:471:read_config: reading config file path="/fluentd/etc/fluent.conf"
fluentd_2 | 2016-10-18T14:29:58.464357222Z 2016-10-18 14:29:58 +0000 [info]: fluent/supervisor.rb:337:supervise: starting fluentd-0.12.29
fluentd_2 | 2016-10-18T14:29:58.543209510Z 2016-10-18 14:29:58 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-mixin-config-placeholders' version '0.4.0'
fluentd_2 | 2016-10-18T14:29:58.543257242Z 2016-10-18 14:29:58 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-docker-format' version '0.2.3'
fluentd_2 | 2016-10-18T14:29:58.543317650Z 2016-10-18 14:29:58 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-elasticsearch' version '1.7.0'
fluentd_2 | 2016-10-18T14:29:58.543408305Z 2016-10-18 14:29:58 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-s3' version '0.7.1'
fluentd_2 | 2016-10-18T14:29:58.543525763Z 2016-10-18 14:29:58 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluentd' version '0.12.29'
fluentd_2 | 2016-10-18T14:29:58.543638143Z 2016-10-18 14:29:58 +0000 [info]: fluent/engine.rb:126:block in configure: gem 'fluentd' version '0.10.61'
fluentd_2 | 2016-10-18T14:29:58.543915201Z 2016-10-18 14:29:58 +0000 [info]: fluent/agent.rb:129:add_match: adding match pattern="docker.var.lib.docker.containers...log" type="docker_format"
fluentd_2 | 2016-10-18T14:29:58.546189569Z 2016-10-18 14:29:58 +0000 [info]: fluent/agent.rb:129:add_match: adding match pattern="" type="copy"
fluentd_2 | 2016-10-18T14:29:58.546478263Z 2016-10-18 14:29:58 +0000 [debug]: plugin/out_copy.rb:44:block in configure: adding store type="elasticsearch"
fluentd_2 | 2016-10-18T14:29:58.654633862Z 2016-10-18 14:29:58 +0000 [debug]: plugin/out_copy.rb:44:block in configure: adding store type="s3"
fluentd_2 | 2016-10-18T14:29:58.706809710Z 2016-10-18 14:29:58 +0000 [info]: fluent/root_agent.rb:147:add_source: adding source type="tail"
fluentd_2 | 2016-10-18T14:29:58.717655521Z 2016-10-18 14:29:58 +0000 [info]: fluent/engine.rb:133:configure: using configuration file:
fluentd_2 | 2016-10-18T14:29:58.717667174Z
fluentd_2 | 2016-10-18T14:29:58.717670825Z log_level debug
fluentd_2 | 2016-10-18T14:29:58.717673963Z
fluentd_2 | 2016-10-18T14:29:58.717677094Z
fluentd_2 | 2016-10-18T14:29:58.717680053Z type tail
fluentd_2 | 2016-10-18T14:29:58.717683043Z path /var/lib/docker/containers//-json.log
fluentd_2 | 2016-10-18T14:29:58.717686178Z pos_file fluentd-docker.pos
fluentd_2 | 2016-10-18T14:29:58.717689108Z time_format %Y-%m-%dT%H:%M:%S
fluentd_2 | 2016-10-18T14:29:58.717692020Z tag docker.*
fluentd_2 | 2016-10-18T14:29:58.717694865Z format json
fluentd_2 | 2016-10-18T14:29:58.717697657Z
fluentd_2 | 2016-10-18T14:29:58.717700640Z <match docker.var.lib.docker.containers...log>
fluentd_2 | 2016-10-18T14:29:58.717703595Z type docker_format
fluentd_2 | 2016-10-18T14:29:58.717706470Z container_id ${tag_parts[5]}
fluentd_2 | 2016-10-18T14:29:58.717718503Z tag docker.${name}
fluentd_2 | 2016-10-18T14:29:58.717724682Z docker_containers_path /var/lib/docker/containers
fluentd_2 | 2016-10-18T14:29:58.717727907Z
fluentd_2 | 2016-10-18T14:29:58.717730840Z <match docker.>
fluentd_2 | 2016-10-18T14:29:58.717733834Z @type copy
fluentd_2 | 2016-10-18T14:29:58.717736595Z
fluentd_2 | 2016-10-18T14:29:58.717739495Z @type elasticsearch
fluentd_2 | 2016-10-18T14:29:58.717742347Z host elasticsearch
fluentd_2 | 2016-10-18T14:29:58.717745172Z port 9200
fluentd_2 | 2016-10-18T14:29:58.717747949Z include_tag_key true
fluentd_2 | 2016-10-18T14:29:58.717750819Z tag_key @log_name
fluentd_2 | 2016-10-18T14:29:58.717753656Z logstash_format true
fluentd_2 | 2016-10-18T14:29:58.717756494Z flush_interval 5s
fluentd_2 | 2016-10-18T14:29:58.717759227Z
fluentd_2 | 2016-10-18T14:29:58.717762106Z
fluentd_2 | 2016-10-18T14:29:58.717764943Z @type s3
fluentd_2 | 2016-10-18T14:29:58.717767631Z aws_key_id xxxxxx
fluentd_2 | 2016-10-18T14:29:58.717770376Z aws_sec_key xxxxxx
fluentd_2 | 2016-10-18T14:29:58.717773109Z s3_bucket fluentd
fluentd_2 | 2016-10-18T14:29:58.717775893Z s3_region eu-central-1
fluentd_2 | 2016-10-18T14:29:58.717778631Z path logs/
fluentd_2 | 2016-10-18T14:29:58.717781389Z buffer_path /var/log/fluent/s3
fluentd_2 | 2016-10-18T14:29:58.717784193Z time_slice_format %Y%m%d%H
fluentd_2 | 2016-10-18T14:29:58.717786958Z time_slice_wait 5s
fluentd_2 | 2016-10-18T14:29:58.717789730Z utc
fluentd_2 | 2016-10-18T14:29:58.717793708Z buffer_chunk_limit 256m
fluentd_2 | 2016-10-18T14:29:58.717796698Z
fluentd_2 | 2016-10-18T14:29:58.717799677Z
fluentd_2 | 2016-10-18T14:29:58.717802648Z
fluentd_2 | 2016-10-18T14:29:59.612179183Z 2016-10-18 14:29:59 +0000 [info]: plugin/in_tail.rb:557:initialize: following tail of /var/lib/docker/containers/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b-json.log

CONTAINER FOLDER RIGHTS

[...]$ sudo ls -al /var/lib/docker/containers/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b
total 4644
drwx------ 3 root root 4096 Oct 18 14:30 .
drwx------ 12 root root 20480 Oct 18 14:31 ..
-rw-rw-rw- 1 root root 9708 Oct 18 14:30 config.v2.json
-rw-r----- 1 root root 4692269 Oct 18 14:31 e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b-json.log
-rw-rw-rw- 1 root root 1282 Oct 18 14:30 hostconfig.json
-rw-r--r-- 1 root root 13 Oct 18 14:30 hostname
-rw-r--r-- 1 root root 174 Oct 18 14:30 hosts
-rw-r--r-- 1 root root 109 Oct 18 14:30 resolv.conf
-rw-r--r-- 1 root root 71 Oct 18 14:30 resolv.conf.hash
drwxrwxrwt 2 root root 40 Oct 18 14:30 shm

CONFIG.V2.JSON

[...]$ sudo cat /var/lib/docker/containers/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b/config.v2.json
{"State":{"Running":true,"Paused":false,"Restarting":false,"OOMKilled":false,"RemovalInProgress":false,"Dead":false,"Pid":6820,"StartedAt":"2016-10-18T14:30:40.006984652Z","FinishedAt":"2016-10-18T14:30:35.585791768Z","Health":null},"ID":"e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b","Created":"2016-10-17T17:42:42.568174797Z","Managed":false,"Path":"/bin/sh", ........,"Config":{"Hostname":"e47bced8e497","Domainname":"","User":"",
[...]
,"LogPath":"/var/lib/docker/containers/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b-json.log","Name":"/r-stack_my-container","Driver":"overlay","MountLabel":"","ProcessLabel":"","RestartCount":0,"HasBeenStartedBefore":false,"HasBeenManuallyStopped":false,"MountPoints":{"/tmp":{"Source":"","Destination":"/tmp","RW":true,"Name":"3f92ecaad901e05cc4ce24f998a4605a2861590ff22f3924abe594e79c8b8262","Driver":"local","Relabel":"","Propagation":"","Named":false,"ID":"13a83ceabb05a0128342ba0b52db7443de61bd03ed95a36ee9356c66c658af72"}},"AppArmorProfile":"","HostnamePath":"/var/lib/docker/containers/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b/hostname","HostsPath":"/var/lib/docker/containers/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b/hosts","ShmPath":"/var/lib/docker/containers/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b/shm","ResolvConfPath":"/var/lib/docker/containers/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b/resolv.conf","SeccompProfile":"","NoNewPrivileges":false}

CAT

[...]$ cat /var/lib/docker/containers/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b
cat: can't open '/var/lib/docker/containers/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b': Permission denied

WHEREAS

sudo cat /var/lib/docker/containers/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b/e47bced8e497f96b6da082c40c530b184a4ff33cdd45664386aca078c8b75e0b-json.
shows file content