Git Product home page Git Product logo

fluentd's Introduction

Fluentd: Open-Source Log Collector

Testing on Ubuntu Testing on Windows Testing on macOS Code Climate CII Best Practices

Fluentd collects events from various data sources and writes them to files, RDBMS, NoSQL, IaaS, SaaS, Hadoop and so on. Fluentd helps you unify your logging infrastructure (Learn more about the Unified Logging Layer).

Quick Start

$ gem install fluentd
$ fluentd -s conf
$ fluentd -c conf/fluent.conf &
$ echo '{"json":"message"}' | fluent-cat debug.test

Development

Branch

Prerequisites

  • Ruby 2.7 or later
  • git

git should be in PATH. On Windows, you can use Github for Windows and GitShell for easy setup.

Install dependent gems

Use bundler:

$ gem install bundler
$ bundle install --path vendor/bundle

Run test

$ bundle exec rake test

You can run specified test via TEST environment variable:

$ bundle exec rake test TEST=test/test_specified_path.rb
$ bundle exec rake test TEST=test/test_*.rb

More Information

Security

A third party security audit was performed by Cure53, you can see the full report here.

See SECURITY to contact us about vulnerability.

Contributors:

Patches contributed by great developers.

fluentd's People

Contributors

abetomo avatar abicky avatar ashie avatar cosmo0920 avatar daipom avatar edsiper avatar frsyuki avatar fujimotos avatar ganmacs avatar garfield96 avatar hotchpotch avatar isimluk avatar itiut avatar kazegusuri avatar kenhys avatar kentaro avatar kiyoto avatar kzk avatar llby avatar masayukikiyota avatar naritta avatar nurse avatar okahashi117 avatar okkez avatar pranjal-gupta2 avatar repeatedly avatar ryysud avatar sonots avatar tagomoris avatar toyama0919 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

fluentd's Issues

Encoding errors when processing some log messages

Hello,

I've noticed that fluentd will blow when processing certain messages that use non ASCII-8BIT characters.

This is a message coming from syslog. It blows up in the s3 plugin.

Peter

2012-11-16 16:01:26 +0000: emit transaction failed error=""\xC2" from ASCII-8BIT to UTF-8"
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-s3-0.2.2/lib/fluent/plugin/out_s3.rb:42:in e ncode' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-s3-0.2.2/lib/fluent/plugin/out_s3.rb:42:int
o_json'
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-s3-0.2.2/lib/fluent/plugin/out_s3.rb:42:in f ormat' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/output.rb:515:inblock in emit'
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/event.rb:52:in call' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/event.rb:52:ineach'
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/output.rb:506:in emit' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-forest-0.1.2/lib/fluent/plugin/out_forest.rb: 116:inemit'
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/match.rb:38:in emit' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/engine.rb:111:inemit_stream'
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/engine.rb:96:in emit' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-rewrite-0.0.8/lib/fluent/plugin/out_rewrite.r b:55:inblock in emit'
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/event.rb:52:in call' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/event.rb:52:ineach'
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-rewrite-0.0.8/lib/fluent/plugin/out_rewrite.rb:53:in emit' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/match.rb:38:inemit'
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/engine.rb:111:in emit_stream' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/engine.rb:96:inemit'
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/plugin/in_syslog.rb:181:in emit' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/plugin/in_syslog.rb:167:inreceive_data'
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/plugin/in_syslog.rb:196:in call' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/plugin/in_syslog.rb:196:inon_readable'
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.0/lib/cool.io/io.rb:170:in on_readable' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.0/lib/cool.io/loop.rb:96:inrun_once'
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.0/lib/cool.io/loop.rb:96:in run' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/plugin/in_syslog.rb:113:inrun'
2012-11-16 16:01:26 +0000: emit transaction failed error=""\xC2" from ASCII-8BIT to UTF-8"
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-s3-0.2.2/lib/fluent/plugin/out_s3.rb:42:in encode' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-s3-0.2.2/lib/fluent/plugin/out_s3.rb:42:into_json'
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-s3-0.2.2/lib/fluent/plugin/out_s3.rb:42:in format' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/output.rb:515:inblock in emit'
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/event.rb:52:in call' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/event.rb:52:ineach'
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/output.rb:506:in emit' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-forest-0.1.2/lib/fluent/plugin/out_forest.rb:116:inemit'
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/match.rb:38:in emit' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/engine.rb:111:inemit_stream'
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/engine.rb:96:in emit' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-rewrite-0.0.8/lib/fluent/plugin/out_rewrite.rb:55:inblock in emit'
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/event.rb:52:in call' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/event.rb:52:ineach'
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-rewrite-0.0.8/lib/fluent/plugin/out_rewrite.rb:53:in emit' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/match.rb:38:inemit'
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/engine.rb:111:in emit_stream' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/engine.rb:96:inemit'
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/plugin/in_syslog.rb:181:in emit' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/plugin/in_syslog.rb:167:inreceive_data'
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/plugin/in_syslog.rb:196:in call' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/plugin/in_syslog.rb:196:inon_readable'
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.0/lib/cool.io/io.rb:170:in on_readable' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.0/lib/cool.io/loop.rb:96:inrun_once'
2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.0/lib/cool.io/loop.rb:96:in run' 2012-11-16 16:01:26 +0000: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.27/lib/fluent/plugin/in_syslog.rb:113:inrun'
2012-11-16 16:01:26 +0000: "<142>Nov 16 16:01:26 mingle-test2-web-1a7zfu test2: INFO [2012-11-16 16:01:26,377] [btpool0-2] [com.thoughtworks.mingle] Parameters: {"authenticity_token"=>"VMG/dV4ACJpfZlZqejAyrpT+1/E/+0P05CJcQ84ImkI=", "user"=>{"login"=>"solutions", "password"=>"[FILTERED]"}, "commit"=>"Sign in \xC2\xBB", "controller"=>"profile", "action"=>"login"}\n" error=""\xC2" from ASCII-8BIT to UTF-8"

Invalid regexp

cat /etc/td-agent/td-agent.conf

type tail format /^([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})\s+\[(._)\]\s+\"(._)\"\s+\"(.*)\"$/ path /data/logs/www.test.com.log tag nginx.access type file path /var/log/td-agent/access_log

[root@nginx ~]# /etc/init.d/td-agent restart
Shutting down td-agent: [FAILED]
Starting td-agent: 2012-09-18 08:18:52 +0800: fluent/supervisor.rb:187:rescue in main_process: config error file="/etc/td-agent/td-agent.conf" error="Invalid regexp '^([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})\s+\[(.)\]\s+\"(.)\"\s+\"(.*)\"$': No named captures"
[FAILED]

Where did we go wrong?

custom_require.rb:36: stack level too deep (SystemStackError)

when i run :

$ sudo fluentd -c /etc/fluent/fluent.conf -vv

it show:


2013-03-15 09:42:17 +0800 [info]: fluent/supervisor.rb:153:supervise: starting fluentd-0.10.32
2013-03-15 09:42:17 +0800 [info]: fluent/supervisor.rb:235:read_config: reading config file path="/etc/fluent/fluent.conf"
2013-03-15 09:42:17 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered buffer plugin 'file'
2013-03-15 09:42:17 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered buffer plugin 'memory'
2013-03-15 09:42:17 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered input plugin 'debug_agent'
2013-03-15 09:42:17 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered input plugin 'exec'
2013-03-15 09:42:17 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered input plugin 'forward'
2013-03-15 09:42:17 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered input plugin 'gc_stat'
2013-03-15 09:42:17 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered input plugin 'http'
2013-03-15 09:42:17 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered input plugin 'object_space'
2013-03-15 09:42:17 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered input plugin 'status'
2013-03-15 09:42:17 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered input plugin 'tcp'
2013-03-15 09:42:17 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered input plugin 'unix'
2013-03-15 09:42:17 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered input plugin 'syslog'
2013-03-15 09:42:17 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered input plugin 'tail'
2013-03-15 09:42:17 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered output plugin 'copy'
2013-03-15 09:42:17 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered output plugin 'exec'
2013-03-15 09:42:17 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered output plugin 'exec_filter'
2013-03-15 09:42:17 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered output plugin 'file'
2013-03-15 09:42:17 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered output plugin 'forward'
2013-03-15 09:42:17 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered output plugin 'null'
2013-03-15 09:42:17 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered output plugin 'roundrobin'
2013-03-15 09:42:17 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered output plugin 'stdout'
2013-03-15 09:42:17 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered output plugin 'tcp'
2013-03-15 09:42:17 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered output plugin 'unix'
2013-03-15 09:42:17 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered output plugin 'test'
2013-03-15 09:42:17 +0800 [info]: fluent/engine.rb:61:configure: using configuration file: <ROOT>
<source>
    type forward
  </source>
  <source>
    type http
    port 8888
  </source>
  <source>
    type tail
    format apache2
    path /private/var/log/apache2/access_log
    pos_file /var/log/td-agent/apache2.access_log.pos
    tag mongo.apache
  </source>
  <source>
    type debug_agent
    port 24230
  </source>
  <match debug.**>
    type stdout
  </match>
  <match mongo.**>
    type mongo
    database apache
    collection access
    host 127.0.0.1
    port 27017
    flush_interval 10s
  </match>
</ROOT>
2013-03-15 09:42:17 +0800 [info]: fluent/engine.rb:70:block in configure: adding source type="forward"
2013-03-15 09:42:17 +0800 [info]: fluent/engine.rb:70:block in configure: adding source type="http"
2013-03-15 09:42:17 +0800 [info]: fluent/engine.rb:70:block in configure: adding source type="tail"
2013-03-15 09:42:17 +0800 [info]: fluent/engine.rb:70:block in configure: adding source type="debug_agent"
2013-03-15 09:42:17 +0800 [info]: fluent/engine.rb:86:block in configure: adding match pattern="debug.**" type="stdout"
2013-03-15 09:42:17 +0800 [info]: fluent/engine.rb:86:block in configure: adding match pattern="mongo.**" type="mongo"
2013-03-15 09:42:17 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered output plugin 'mongo'
/usr/local/Cellar/ruby/1.9.3-p286/lib/ruby/1.9.1/rubygems/custom_require.rb:36: stack level too deep (SystemStackError)
2013-03-15 09:42:17 +0800 [info]: fluent/supervisor.rb:170:supervise: process finished code=256
2013-03-15 09:42:17 +0800 [warn]: fluent/supervisor.rb:173:supervise: process died within 1 second. exit.

what's the meaning :
2013-03-14 23:21:25 +0800 [trace]: fluent/plugin.rb:89:register_impl: registered output plugin 'mongo'

/usr/local/Cellar/ruby/1.9.3-p286/lib/ruby/1.9.1/rubygems/custom_require.rb:36: stack level too deep (SystemStackError)

how to solve this problem ?

my environment:
mac os 10.7.5
$ sudo gem list
*** LOCAL GEMS ***
fluent-plugin-mongo (0.6.13)
fluentd (0.10.32)
mongo (1.6.4)

$ ruby -v
ruby 1.9.3p286 (2012-10-12 revision 37165) [x86_64-darwin11.4.2]

コンフィグ内の#文字の取り扱い

fluentd (0.10.31)をrubygemsからインストールしています。
コンフィグ内の#文字をコメントでなく扱いたいと思っています。

mongodbとの接続でfluent-plugin-mongoを利用しているのですが、mongodbとの接続情報に"#"が含まれているケースでパスワードが指定できなくなります。

user username
password #hoge+-hoge

上記のような設定ですと#以降がコメント扱いとなり、Mongodbの認証で弾かれています。
エスケープ等できると良いのですが。

TimeSlicedOutput should have hooks like 'expire_timeslice(chunk_key)'

fluentd TimeSlicedOutput plugins should have hooks for each time slice, for called when thats time_slice expires.

Many of data storage or reduce-like output needs termination. Without such hooks, to terminate each slices, we must make threads on each plugins. This is waste of CPU/Memory.

buf_file plugin doesn't support key which contains '/'

Bug in buf_file.rb:

def encode_key(key)
    URI.encode(key, /[^-_.a-zA-Z0-9]/n)
end

def decode_key(encoded_key)
    URI.decode(encoded_key, /[^-_.a-zA-Z0-9]/n)
end

URI.encode('/', /[^-_.a-zA-Z0-9]/n) returns "%2F", and URI.decode('%2F', /[^-_.a-zA-Z0-9]/n) returns "\u00002F".
so the fluent-s3-plugin cannot create sub directories.

I tested in ruby 1.9.2.

out_exec_filter fails tests

Using 0.10.9, fails with:

Finished tests in 4.680640s, 1.0682 tests/s, 4.2729 assertions/s.

  1) Failure:
test_emit_3(ExecFilterOutputTest) [out_exec_filter.rb:130]:
<2> expected but was
<0>.

  2) Failure:
test_emit_4(ExecFilterOutputTest) [out_exec_filter.rb:155]:
<2> expected but was
<0>.

5 tests, 20 assertions, 2 failures, 0 errors, 0 skips

I can't get it to work with anything more complex than cat. The idea is that it calls a command, feeds it events as input, and reads the output of that command to generate new events, right? That seems to be the idea but the documentation isn't very clear...

I'm using Scientific Linux 6.1, which is apparently a RHEL variant.

Ruby version: ruby 1.9.3p0 (2011-10-30) [x86_64-linux]

Can not "Tail" a file using tail plugin

Hi, I found that the tail plugin can not "tail" a file, here's the configuration:

< source >
type tail
path /home/ld-desktop/tmp/test.txt
tag test
format /^( ? < request > .)$/
< /source >
< match test.
* >
type stdout
< /match >

And I added to "test.txt" LINE BY LINE:

request:hello
request:echo server
request:blabla

However, I can see the outputs of fluentd is not a "Tail" one:

2012-01-04 15:33:37 +0800 test: {"request":"request:hello"}
2012-01-04 15:33:37 +0800 test: {"request":""}

2012-01-04 15:33:58 +0800 test: {"request":"request:hello"}
2012-01-04 15:33:58 +0800 test: {"request":"request:echo server"}
2012-01-04 15:33:58 +0800 test: {"request":""}

2012-01-04 15:35:02 +0800 test: {"request":"request:hello"}
2012-01-04 15:35:02 +0800 test: {"request":"request:echo server"}
2012-01-04 15:35:02 +0800 test: {"request":"request:blabla"}
2012-01-04 15:35:02 +0800 test: {"request":""}

Am I wrong in configuring the fluentd, or the "tail" plugin has some bugs?

log duplication by using cronolog

cronologを用いて出力すると、パイプを繋ぎ直す時に、fluentdがlog rotationを検知してしまいます。
なんですが、実際にはファイルはrotationされていないので、ログのduplicateが発生してしまっています。

apacheの出力をcornologで行う場合では、apacheの再起動をして、その後始めてアクセスが来た場合にrotationが検知されるようです。こちらでも、ログのduplicateが発生したのを確認しています。

再現コード

➜  cat /etc/td-agent/td-agent.conf

#省略

<source>
  type tail
  format /^(?<time>[^\t]*)\t(?<message>.*)$/
  path /tmp/cronolog.log
  tag cronolog
</source>

<match cronolog>
  type file
  path /var/log/td-agent/cronolog/log
</match>

# 省略


#2012-08-11T01:38:02
➜  echo "`date +"%Y/%m/%e %H:%M:%S"`\tmessage1" | cronolog -H /tmp/cronolog.log /tmp/cronolog.%Y%m%d.log
#2012-08-11T01:38:15
➜  echo "`date +"%Y/%m/%e %H:%M:%S"`\tmessage2" | cronolog -H /tmp/cronolog.log /tmp/cronolog.%Y%m%d.log


➜  tail -n 6 /var/log/td-agent/td-agent.log
2012-08-11 01:37:19 +0000: plugin/in_forward.rb:60:listen: listening fluent socket on 0.0.0.0:24224
2012-08-11 01:37:19 +0000: plugin/in_http.rb:74:start: listening http on 0.0.0.0:8888
2012-08-11 01:38:02 +0000: plugin/in_tail.rb:262:initialize: following tail of /tmp/cronolog.log
2012-08-11 01:38:15 +0000: plugin/in_tail.rb:203:on_rotate: detected rotation of /tmp/cronolog.log; waiting 5 seconds  # <= 誤検知 
2012-08-11 01:38:15 +0000: plugin/in_tail.rb:199:on_rotate: detected rotation of /tmp/cronolog.log
2012-08-11 01:38:17 +0000: plugin/in_tail.rb:262:initialize: following tail of /tmp/cronolog.log

➜  cat /var/log/td-agent/cronolog/log.20120811.b4c6f382e4f80168d
2012-08-11T01:38:02+00:00       cronolog        {"message":"message1"}
2012-08-11T01:38:15+00:00       cronolog        {"message":"message2"}
2012-08-11T01:38:02+00:00       cronolog        {"message":"message1"}  # <=  duplicated log
2012-08-11T01:38:15+00:00       cronolog        {"message":"message2"}  # <=  duplicated log
</addr>

DEPENDENCY ISSUE

Unable to install cool.io 1.1.0 dependency required for fluentd installation in windows 7. It is under maintenance in github.

rpmbuild make Compiler error

./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu
--target=x86_64-redhat-linux-gnu
--program-prefix=
--prefix=/usr
--exec-prefix=/usr
--bindir=/usr/bin
--sbindir=/usr/sbin
--sysconfdir=/etc
--datadir=/usr/share
--includedir=/usr/include
--libdir=/usr/lib64
--libexecdir=/usr/libexec
--localstatedir=/var
--sharedstatedir=/usr/com
--mandir=/usr/share/man
--infodir=/usr/share/info

make

error:

make[1]: Entering directory /root/rpmbuild/BUILD/td-agent-1.1.11/deps/ruby' CC = gcc LD = ld LDSHARED = gcc -shared CFLAGS = -O3 -ggdb -Wextra -Wno-unused-parameter -Wno-parentheses -Wno-long-long -Wno-missing-field-initializers -Wpointer-arith -Wwrite-strings -Wdeclaration-after-statement -Werror-implicit-function-declaration XCFLAGS = -include ruby/config.h -include ruby/missing.h -fvisibility=hidden -DRUBY_EXPORT CPPFLAGS = -I. -I.ext/include/x86_64-linux -I./include -I. DLDFLAGS = SOLIBS = linking miniruby generating encdb.h make[1]: *** [.rbconfig.time] Segmentation fault make[1]: *** Waiting for unfinished jobs.... make[1]: *** [encdb.h] Segmentation fault make[1]: Leaving directory/root/rpmbuild/BUILD/td-agent-1.1.11/deps/ruby'
make: *** [all-recursive] Error 1

in_tail.rb symlinks bugs

in_tail.rbを利用した時にシンボリックリンクに対してtailすると永久ループに入り、同じログを拾ってしまう現象を確認しています。
ハードリンクでtailすると発生しないので、判定で何か問題発生していないでしょうか?

Buffer/Queueが共にFULLになった後の復旧が行われない

td-agent-1.1.9-0にてfluentを使用しております

in_tail.rbのソースを見たところ(失礼ながらログも埋め込ませていただきました)

わざとforward先のFluendをダウンさせ
Buffer/QueueをFULLにしました

その後Fluentdを再開させても@buffer.empty?がfalseのままとなり例外メッセージを吐き続ける状態となりました

ぱっとみた原因ですが最初に303行のrescueにて"BufferQueueLimitError(queue size exceeds limit)"を処理しています
rescue
$log.error $!.to_s
$log.error_backtrace
close
しかし、ここでcloseしている為、これ以降ファイルがclose状態になり、"closed stream"の例外が
発生し続けfluentdを再起動する必要があります

単純に以下のようにしたところ一応例外処理は発生しなくなっております
※単純に別途BufferQueueLimitError例外処理を用意してcloseをしないようにしただけです

  rescue BufferQueueLimitError
    $log.error $!.to_s
    $log.error_backtrace
  rescue => err
    $log.error $!.to_s
    $log.error_backtrace
    close
  end

Add support for IPv6

Hi, I am using IPv6 now. but Fluentd does not support IPv6. Do you have any plans to support IPv6?

IPv6 support in in/out_forward

Currently, in_forward always create a IPv4 UDPSocket.
In this result, socket bind always failed with IPv6 host, e.g. ::1.

When bind parameter is IPv6, in_forward should create a UDPSocket with Socket::AF_INET6.

def start
  # ...
  if ipv4?
    @usock = UDPSocket.new
  else
    @usock = UDPSocket.new(Socket::AF_INET6)
  end
  #...
end

Add in current logfile symlink

Is there any chance we could get a symlink to the current log file in the log directory? Something like access.log_current that always points to the logfile being written to? Scribe has a facility like this, and it makes tailing logs quite simple.

many CLOSE_WAIT's

Hi,

We may have a platform issue in that am seeing a lot of CLOSE_WAIT states at the fluent aggregator side, so I have forwarders sending to two aggregators. It runs for a while and then stops.

The forwarders give:

2012-12-05 15:56:32 +0000: fluent/output.rb:336:rescue in try_flush: temporarily failed to flush the buffer, next retry will be at 2012-12-05 16:30:39 +0000. error="no nodes are available" instance=19480460

The aggregators give have many CLOSE_WAIT states for TCP and I can see many socket "accepts" and few closes.

ubuntu@fluent-worker-01-live:~$ netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c
1067 CLOSE_WAIT
1 ESTABLISHED
3 LISTEN

ubuntu@fluent-worker-01-live:~$ uname -a
Linux fluent-worker-01-live 3.2.0-31-virtual #50-Ubuntu SMP Fri Sep 7 16:36:36 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Running ruby 1.9.3p286 (2012-10-12 revision 37165) [x86_64-linux]

Any ideas ?

Add pixel tracking feature to in_http

Recently we've got many requests of having 'pixel tracking' feature of in_http. This enables Fluentd users to put the data in an easy way (like from HTML pages).

Exec Input Documentation Incomplete

It doesn't mention how to specify the external command. From the tests it appears the parameter is "command". That should probably be mentioned.

Fluent::BufferChunk should support msgpack_map

fluent-plugin-mongo has follwing code:

records = []
chunk.msgpack_each { |record|
  record[@time_key] = Time.at(record[@time_key]) if @include_time_key
  records << record
}

I think this pattern is often-used in some OutputPlugins.

BufferChunk#msgpack_map gets rid of such pattern and
we have the chance to improve speed than msgpack_each.

problem with in_tail of fluentd 0.10.12

I noticed that the "in_tail" plugin shipped with fluentd-0.10.12 consumes 100% CPU time on my box.

OS: Ubuntu 10.04 / RHEL 6
Ruby: 1.9.2p290 (2011-07-09 revision 32553) [x86_64-linux]
fluentd: "gem install --version 0.10.12 fluentd"

It seems the 'io' object passed to IOHandler is always readable, and 'on_readable' is called repeatedly.

Sorry that I have no idea how to fix it.

about the reliability of the out_foward heartbeat

fluentd(0.10.22) td-agent-1.1.6-0. CentOS5.8 SL6.2で発生

apacheログをin_tailで読み込み、おおよそ30台くらいのサーバからout_fowardで送信すると
送信側で以下の様なdetached->recoveredを繰り返すログが出ます。
https://gist.github.com/2848345

受信側は大体全体のログ流入量が200~500message/secの負荷状況でも発生しています。

ただし、全てのサーバで出るのではなく、散発的に出るレベルです。

out_forwardのheartbeatの設定はデフォルトのままで利用しています。

これは、heartbeatをUDPで行っている為に対向側の処理が多いとパケットをdropしてしまう、
受信側のin_forwardでheartbeat処理が遅延してしまっているなどの可能性を考えています。
可能であれば、heartbeatについても信頼性の高いTCP通信があるとUDP特有の問題は
解消出来るのでは無いかと考えています。

一報告として扱って貰えれば助かります。

PositionEntry inode update

in_tailにおいて、 PositionEntryを用いることでinodeの管理をしていますが、log rotationが起こった際にinodeを更新する処理がどこにもないので、log rotationが起こるとPositionEntryのinodeと、実際に読みこみを行っているファイルのinodeが異なる値になってしまいます。

このため、以下のような動きが起こった場合、ログの欠損が発生します。

  • fluentd起動
  • log rotation
  • fluentd再起動
  • pathの場所にあるファイルのinodeと, pos_fileのinodeが異なるため、新しいファイルを読み込もうとしていると誤判定し、ファイルの末尾から読み込み始める <= ログの欠損

Why log level was changed from debug to warn when text does not match the pattern?

I'm implementing a input plugin which extends in_tail plugin.

  • the target log file is written by various processes via syslog
  • the plugin does pattern matching against some of processes in the file

In 317284e log level was changed from debug to warn when text does not mattch the pattern. so I received too many warns like this:

2013-01-18 15:01:49 +0900: pattern not match:  .......
2013-01-18 15:01:49 +0900: pattern not match:  .......

To supress these warns I added '-qq' to TD_AGENT_ARGS in /etc/init.d/td-agent but I noticed that other useful warns also supressed.

Why the log level changed? I'd like to supress 'pattern not match' only not to supress all warns. any ideas?

Fluentd flush_interval issues under load

Under heavy load we are not seeing fluentd flush_interval 10m flush very 10 minutes and it is taking almost a day for the log data to go through the system. Any ideas ?Very basic setup we are using with some simple plugins.

How do i test the fluentd's bottleneck?

I use fluentd to collecte log to MongoDB. i have four log servers, each log server collecte two log file. Now, i need scan eache log file three times, and then forward to three tables in MongoDB. If i need collecte more log, i don't know when is the bottleneck?

unexpected error error="U+00FC from UTF-8 to ASCII-8BIT" while require 'mime/types' in plugin

I'm writing a plugin for fluent, and I meet a error when I require 'mime/types' in the plugin.
Here is the error message:

2012-11-16 22:42:36 +0800: unexpected error error="U+00FC from UTF-8 to ASCII-8BIT"
2012-11-16 22:42:36 +0800: /usr/local/Cellar/rbenv/0.3.0/versions/1.9.3-p286/lib/ruby/gems/1.9.1/gems/mime-types-1.19/lib/mime/types.rb:759:in read' 2012-11-16 22:42:36 +0800: /usr/local/Cellar/rbenv/0.3.0/versions/1.9.3-p286/lib/ruby/gems/1.9.1/gems/mime-types-1.19/lib/mime/types.rb:759:inblock in load_from_file'
2012-11-16 22:42:36 +0800: /usr/local/Cellar/rbenv/0.3.0/versions/1.9.3-p286/lib/ruby/gems/1.9.1/gems/mime-types-1.19/lib/mime/types.rb:759:in `open'
...

It seens to be a encoding problem, I found that both Encoding.default_internal and Encoding.default_external is set to 'ASCII-8BIT' that made this problem.

Finally I found that the code in lib/fluent/engine.rb make this setting.

1851330

So why set default encoding to ASCII-8BIT and how could I fix it.

Failed to "gem install " from source code.

The fluentd 0.10.3 fails to "gem install" when rake it from source code.
The log dump is as follows:

ozawa@ozamba:/workspace/fluentd$ rake
WARNING: no description specified
Successfully built RubyGem
Name: fluentd
Version: 0.10.3
File: fluentd-0.10.3.gem
Generated: fluentd.gemspec
fluentd.gemspec is valid.
ozawa@ozamba:
/workspace/fluentd$ gem install pkg/fluentd-*.gem
ERROR: Error installing pkg/fluentd-0.10.3.gem:
fluentd requires fluentd (>= 0)

out_forward: resolve DNS more intelligently

On the cloud environment such as AWS, the same hostname could be resolved into the different IP after the reboot.

Therefore, it's safer to resolve the hostname each time network transfer happens.

However, this could incur the performance problem. So having the caching is ideal here.

fluentd 0.10.1 cannot parse apache acceess log?

I am testing fluentd 0.10.1.

It cannnot transfer apache access log since 0.10.1 update.

I looked that fluentd is failed parse apache access log.

[transfer image]
transfer server(apache) -- fluentd --> receive server

Following is my environmment and stack trace.

[environment]
-transfer server
  OS:CentOS5.6
  fluentd:0.10.1
  apache:2.2.3

-receive server
  OS:CentOS6.0
  fluentd:0.10.1
  fluent-plugin-mongo:0.4.0

[stack trace]
Error log is output at receive server.

2011-10-18 10:30:26 +0900: plugin/in_stream.rb:110:initialize: accepted fluent socket object_id=19531740
2011-10-18 10:30:26 +0900: fluent/engine.rb:103:rescue in emit_stream: emit transaction faild  error="undefined method `[]=' for nil:NilClass"
  2011-10-18 10:30:26 +0900: fluent/engine.rb:104:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/mixin.rb:150:in `filter_record'
  2011-10-18 10:30:26 +0900: fluent/engine.rb:104:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/mixin.rb:96:in `block in format_stream'
  2011-10-18 10:30:26 +0900: fluent/engine.rb:104:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/event.rb:127:in `feed_each'
  2011-10-18 10:30:26 +0900: fluent/engine.rb:104:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/event.rb:127:in `each'
  2011-10-18 10:30:26 +0900: fluent/engine.rb:104:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/mixin.rb:95:in `format_stream'
  2011-10-18 10:30:26 +0900: fluent/engine.rb:104:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/output.rb:311:in `emit'
  2011-10-18 10:30:26 +0900: fluent/engine.rb:104:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/match.rb:38:in `emit'
  2011-10-18 10:30:26 +0900: fluent/engine.rb:104:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/engine.rb:101:in `emit_stream'
  2011-10-18 10:30:26 +0900: fluent/engine.rb:104:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/plugin/in_stream.rb:81:in `on_message'
  2011-10-18 10:30:26 +0900: fluent/engine.rb:104:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/plugin/in_stream.rb:139:in `feed_each'
  2011-10-18 10:30:26 +0900: fluent/engine.rb:104:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/plugin/in_stream.rb:139:in `on_read_msgpack'
  2011-10-18 10:30:26 +0900: fluent/engine.rb:104:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/plugin/in_stream.rb:131:in `call'
  2011-10-18 10:30:26 +0900: fluent/engine.rb:104:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/plugin/in_stream.rb:131:in `on_read'
  2011-10-18 10:30:26 +0900: fluent/engine.rb:104:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.0.0/lib/cool.io/io.rb:108:in `on_readable'
  2011-10-18 10:30:26 +0900: fluent/engine.rb:104:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.0.0/lib/cool.io/io.rb:170:in `on_readable'
  2011-10-18 10:30:26 +0900: fluent/engine.rb:104:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.0.0/lib/cool.io/loop.rb:96:in `run_once'
  2011-10-18 10:30:26 +0900: fluent/engine.rb:104:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.0.0/lib/cool.io/loop.rb:96:in `run'
  2011-10-18 10:30:26 +0900: fluent/engine.rb:104:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/plugin/in_stream.rb:46:in `run'
2011-10-18 10:30:26 +0900: plugin/in_stream.rb:48:rescue in run: unexpected error error="undefined method `[]=' for nil:NilClass"
  2011-10-18 10:30:26 +0900: plugin/in_stream.rb:49:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/mixin.rb:150:in `filter_record'
  2011-10-18 10:30:26 +0900: plugin/in_stream.rb:49:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/mixin.rb:96:in `block in format_stream'
  2011-10-18 10:30:26 +0900: plugin/in_stream.rb:49:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/event.rb:127:in `feed_each'
  2011-10-18 10:30:26 +0900: plugin/in_stream.rb:49:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/event.rb:127:in `each'
  2011-10-18 10:30:26 +0900: plugin/in_stream.rb:49:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/mixin.rb:95:in `format_stream'
  2011-10-18 10:30:26 +0900: plugin/in_stream.rb:49:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/output.rb:311:in `emit'
  2011-10-18 10:30:26 +0900: plugin/in_stream.rb:49:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/match.rb:38:in `emit'
  2011-10-18 10:30:26 +0900: plugin/in_stream.rb:49:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/engine.rb:101:in `emit_stream'
  2011-10-18 10:30:26 +0900: plugin/in_stream.rb:49:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/plugin/in_stream.rb:81:in `on_message'
  2011-10-18 10:30:26 +0900: plugin/in_stream.rb:49:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/plugin/in_stream.rb:139:in `feed_each'
  2011-10-18 10:30:26 +0900: plugin/in_stream.rb:49:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/plugin/in_stream.rb:139:in `on_read_msgpack'
  2011-10-18 10:30:26 +0900: plugin/in_stream.rb:49:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/plugin/in_stream.rb:131:in `call'
  2011-10-18 10:30:26 +0900: plugin/in_stream.rb:49:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/plugin/in_stream.rb:131:in `on_read'
  2011-10-18 10:30:26 +0900: plugin/in_stream.rb:49:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.0.0/lib/cool.io/io.rb:108:in `on_readable'
  2011-10-18 10:30:26 +0900: plugin/in_stream.rb:49:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.0.0/lib/cool.io/io.rb:170:in `on_readable'
  2011-10-18 10:30:26 +0900: plugin/in_stream.rb:49:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.0.0/lib/cool.io/loop.rb:96:in `run_once'
  2011-10-18 10:30:26 +0900: plugin/in_stream.rb:49:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.0.0/lib/cool.io/loop.rb:96:in `run'
  2011-10-18 10:30:26 +0900: plugin/in_stream.rb:49:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.1/lib/fluent/plugin/in_stream.rb:46:in `run'

Please help me.

Authentication between nodes is required

Fluentd does not provide authentication methods between fluentd nodes.
This makes it impossible to gain security for circumstances like this. Currently, the only way to maintain security is to run all servers within a local network.

As in Fluentd Meetup Japan 2012, it is said that currently there are no authentication methods between fluentd nodes.
Grep "認証" on this page: http://johtani.jugem.jp/?eid=60


Also, see this thread on the ML:
https://groups.google.com/d/msg/fluentd/LSiF1weaKi0/t66wsZZMrh0J

fluentd can't load library

fluentd's "-r" option doesn't work correctly.

$ sudo /usr/sbin/td-agent -rperftools --user td-agent --group td-agent
2011-04-17 12:31:55 +0400: unexpected error error="undefined local variable or method `lib' for main:Object"
2011-04-17 12:31:55 +0400: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.17/lib/fluent/command/fluentd.rb:143:in `block in <top (required)>' 
2011-04-17 12:31:55 +0400: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.17/lib/fluent/command/fluentd.rb:142:in `each'
2011-04-17 12:31:55 +0400: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.17/lib/fluent/command/fluentd.rb:142:in `<top (required)>'  
2011-04-17 12:31:55 +0400: <internal:lib/rubygems/custom_require>:29:in `require'
2011-04-17 12:31:55 +0400: <internal:lib/rubygems/custom_require>:29:in `require'
2011-04-17 12:31:55 +0400: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.17/bin/fluentd:6:in `<top (required)>'
2011-04-17 12:31:55 +0400: /usr/lib/fluent/ruby/bin/fluentd:19:in `load'
2011-04-17 12:31:55 +0400: /usr/lib/fluent/ruby/bin/fluentd:19:in `<top (required)>'
2011-04-17 12:31:55 +0400: /usr/sbin/td-agent:5:in `load'
2011-04-17 12:31:55 +0400: /usr/sbin/td-agent:5:in `<main>'

Not working SIGHUP (used -o/--log option)

  1. fluentd -o/--log option
  2. SIGHUP fluentd process
  3. unexpected error error="undefined method `reopen' for #Fluent::Log:0x00000001406ec8"
 2011-11-10 20:53:36 +0900: unexpected error error="undefined method `reopen' for #<Fluent::Log:0x00000001406ec8>"
  2011-11-10 20:53:36 +0900: /usr/lib/ruby/gems/1.9.1/gems/fluent-0.9.21/lib/fluent/command/fluentd.rb:190:in `block in <top (required)>'
  2011-11-10 20:53:36 +0900: /usr/lib/ruby/gems/1.9.1/gems/cool.io-1.0.0/lib/cool.io/loop.rb:96:in `call'
  2011-11-10 20:53:36 +0900: /usr/lib/ruby/gems/1.9.1/gems/cool.io-1.0.0/lib/cool.io/loop.rb:96:in `run_once'
  2011-11-10 20:53:36 +0900: /usr/lib/ruby/gems/1.9.1/gems/cool.io-1.0.0/lib/cool.io/loop.rb:96:in `run'
  2011-11-10 20:53:36 +0900: /usr/lib/ruby/gems/1.9.1/gems/fluent-0.9.21/lib/fluent/engine.rb:129:in `run'
  2011-11-10 20:53:36 +0900: /usr/lib/ruby/gems/1.9.1/gems/fluent-0.9.21/lib/fluent/command/fluentd.rb:217:in `<top (required)>'
  2011-11-10 20:53:36 +0900: /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
  2011-11-10 20:53:36 +0900: /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
  2011-11-10 20:53:36 +0900: /usr/lib/ruby/gems/1.9.1/gems/fluent-0.9.21/bin/fluentd:6:in `<top (required)>'
  2011-11-10 20:53:36 +0900: /usr/bin/fluentd:19:in `load'
  2011-11-10 20:53:36 +0900: /usr/bin/fluentd:19:in `<main>'

SetTagKeyMixin does not set tag...

In SetTagKeyMixin#filter_record:

record[@time_key] = @timef.format(time)

It's a SetTimeKeyMixin operation.
I think following code is correct.

record[@tag_key] = tag

Please adding "log level description" on fluentd log

※日本語です。

fluentd自体のログは、log.rbの実装を見る限りANSI Colorにもとづいて色分けして出力されるようになってます。ただし"out.tty"が偽の場合、エスケープシーケンスが出力されず、ログレベルを表すメタ情報が欠落します。
端的に言うと、"-o" オプションで出力したログファイル中にログレベルの記述が無く、ログ監視の際にレベルに応じてアラートを送る、という事が行いづらくなっています。

log.rbについて、以下のようにログレベルの記述を追記するように変更可能でしょうか?

※36行目あたり
LEVEL_TRACE = 0
LEVEL_DEBUG = 1
LEVEL_INFO = 2
LEVEL_WARN = 3
LEVEL_ERROR = 4
LEVEL_FATAL = 5
//以下を追加
LOG_TEXT = ['TRACE', 'DEBUG', 'INFO', 'WARN', 'ERROR', 'FATAL']

※280行目あたり
// return time.strftime(@time_format)
// 以下のように変更
return "#{time.strftime(@time_format)} [#{LOG_TEXT[level]}] "

tcp heartbeat

In some environments, Fluentd can't send heartbeat via udp.
After deeply discussion, we will implement tcp heartbeat.

Key point is what forward plugins send/receive empty msgpack list as heartbeat.
When in_forward receives empty msgpack list, its plugin doesn't emit.

In configuration, we introduce heartbeat param to select heartbeat type.
Default is still udp.
If you set tcp, Fluentd uses above tcp heartbeat instead of udp.

The tail plugin may use too much memory

If the log file is very large. The Tail plugin will read all lines ,so may use to much memory.
I think add so code like these, :
class IOHandler
...
def on_notify
lines = []

    while true
      begin
        if @buffer.empty?
          @io.read_nonblock(2048, @buffer)
        else
          @buffer << @io.read_nonblock(2048, @iobuf)
        end
        while line = @buffer.slice!(/.*?\n/m)
          lines << line
          if lines.size >= 1000
            @receive_lines.call(lines)
            @pe.update(@io.stat.ino, @io.pos - @buffer.bytesize)
            lines.clear()
          end
        end
      rescue EOFError
        break
      end
    end

Ruby process down when open more than 3 files

I faced a trouble in bellow condition.

Ruby 1.9.3-p125 on Mac OS X (10.8.0)
fluentd 0.10.28
fluent-plugin-forest 0.10.28

There no trouble on CentOS release 5.8 (Final) 2.6.18-308.el5,
I cannot find that the trouble is derived from my environment only or not.

fluentd.conf

<source>
  type forward
  port 11111
</source>

<match test.a>
  type file
  time_slice_format %Y%m%d%H
  path ./a.log
</match>
<match test.b>
  type file
  time_slice_format %Y%m%d%H
  path ./b.log
</match>
<match test.c>
  type file
  time_slice_format %Y%m%d%H
  path ./c.log
</match>

fluentd start

% fluentd -c fluentd.conf -o system.log 

information

fluentd % fluentd -c fluentd.conf -o system.log
/Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/thread.rb:71: [BUG] pthread_cond_wait: Invalid argument (EINVAL)
ruby 1.9.3p125 (2012-02-16 revision 34643) [x86_64-darwin10.8.0]

-- Control frame information -----------------------------------------------
c:0007 p:---- s:0024 b:0024 l:000023 d:000023 CFUNC  :sleep
c:0006 p:0031 s:0020 b:0020 l:000019 d:000019 METHOD /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/thread.rb:71
c:0005 p:0018 s:0015 b:0015 l:000014 d:000014 METHOD /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/output.rb:155
c:0004 p:0146 s:0011 b:0011 l:000010 d:000010 METHOD /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/output.rb:129
c:0003 p:---- s:0006 b:0006 l:000005 d:000005 FINISH
c:0002 p:---- s:0004 b:0004 l:000190 d:000003 IFUNC 
c:0001 p:---- s:0002 b:0002 l:000001 d:000001 TOP   

-- Ruby level backtrace information ----------------------------------------
/Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/output.rb:129:in `run'
/Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/output.rb:155:in `cond_wait'
/Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/thread.rb:71:in `wait'
/Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/thread.rb:71:in `sleep'

-- C level backtrace information -------------------------------------------

   See Crash Report log file under ~/Library/Logs/CrashReporter or
   /Library/Logs/CrashReporter, for the more detail of.

-- Other runtime information -----------------------------------------------

* Loaded script: /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/bin/fluentd

* Loaded features:

    0 enumerator.so
    1 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/x86_64-darwin10.8.0/enc/encdb.bundle
    2 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/x86_64-darwin10.8.0/enc/trans/transdb.bundle
    3 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/rubygems/defaults.rb
    4 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/x86_64-darwin10.8.0/rbconfig.rb
    5 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/rubygems/deprecate.rb
    6 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/rubygems/exceptions.rb
    7 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/rubygems/custom_require.rb
    8 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/rubygems.rb
    9 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/rubygems/version.rb
   10 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/rubygems/requirement.rb
   11 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/rubygems/platform.rb
   12 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/rubygems/specification.rb
   13 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/rubygems/path_support.rb
   14 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/rubygems/dependency.rb
   15 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/x86_64-darwin10.8.0/etc.bundle
   16 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/fileutils.rb
   17 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/x86_64-darwin10.8.0/pathname.bundle
   18 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/pathname.rb
   19 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/x86_64-darwin10.8.0/psych.bundle
   20 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/x86_64-darwin10.8.0/stringio.bundle
   21 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/nodes/node.rb
   22 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/nodes/stream.rb
   23 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/nodes/document.rb
   24 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/nodes/sequence.rb
   25 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/nodes/scalar.rb
   26 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/nodes/mapping.rb
   27 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/nodes/alias.rb
   28 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/nodes.rb
   29 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/streaming.rb
   30 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/visitors/visitor.rb
   31 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/x86_64-darwin10.8.0/strscan.bundle
   32 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/scalar_scanner.rb
   33 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/visitors/to_ruby.rb
   34 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/visitors/emitter.rb
   35 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/visitors/yaml_tree.rb
   36 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/json/ruby_events.rb
   37 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/visitors/json_tree.rb
   38 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/visitors/depth_first.rb
   39 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/visitors.rb
   40 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/handler.rb
   41 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/tree_builder.rb
   42 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/parser.rb
   43 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/omap.rb
   44 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/set.rb
   45 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/coder.rb
   46 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/core_ext.rb
   47 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/x86_64-darwin10.8.0/date_core.bundle
   48 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/date/format.rb
   49 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/date.rb
   50 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/deprecated.rb
   51 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych/json.rb
   52 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/psych.rb
   53 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/yaml.rb
   54 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/bundler-1.1.5/lib/bundler/psyched_yaml.rb
   55 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/bundler-1.1.5/lib/bundler/gem_helpers.rb
   56 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/bundler-1.1.5/lib/bundler/match_platform.rb
   57 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/bundler-1.1.5/lib/bundler/rubygems_ext.rb
   58 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/rubygems/config_file.rb
   59 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/bundler-1.1.5/lib/bundler/rubygems_integration.rb
   60 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/bundler-1.1.5/lib/bundler/version.rb
   61 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/bundler-1.1.5/lib/bundler.rb
   62 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/rubygems-bundler-1.0.7/lib/rubygems-bundler/noexec.rb
   63 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/optparse.rb
   64 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/log.rb
   65 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/env.rb
   66 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/version.rb
   67 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/supervisor.rb
   68 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/thread.rb
   69 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/x86_64-darwin10.8.0/socket.bundle
   70 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/socket.rb
   71 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/time.rb
   72 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/monitor.rb
   73 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/json-1.7.5/lib/json/version.rb
   74 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/ostruct.rb
   75 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/json-1.7.5/lib/json/generic_object.rb
   76 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/json-1.7.5/lib/json/common.rb
   77 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/x86_64-darwin10.8.0/enc/utf_16be.bundle
   78 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/x86_64-darwin10.8.0/enc/utf_16le.bundle
   79 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/x86_64-darwin10.8.0/enc/utf_32be.bundle
   80 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/x86_64-darwin10.8.0/enc/utf_32le.bundle
   81 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/json-1.7.5/lib/json/ext/parser.bundle
   82 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/json-1.7.5/lib/json/ext/generator.bundle
   83 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/json-1.7.5/lib/json/ext.rb
   84 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/json-1.7.5/lib/json.rb
   85 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/yajl-ruby-1.1.0/lib/yajl/yajl.bundle
   86 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/yajl-ruby-1.1.0/lib/yajl.rb
   87 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/uri/common.rb
   88 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/uri/generic.rb
   89 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/uri/ftp.rb
   90 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/uri/http.rb
   91 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/uri/https.rb
   92 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/uri/ldap.rb
   93 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/uri/ldaps.rb
   94 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/uri/mailto.rb
   95 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/uri.rb
   96 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/msgpack-0.4.7/lib/msgpack.bundle
   97 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/iobuffer-1.1.2/lib/iobuffer/version.rb
   98 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/iobuffer-1.1.2/lib/iobuffer_ext.bundle
   99 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/iobuffer-1.1.2/lib/iobuffer.rb
  100 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/cool.io-1.1.0/lib/cool.io/version.rb
  101 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/cool.io-1.1.0/lib/cool.io_ext.bundle
  102 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/cool.io-1.1.0/lib/cool.io/loop.rb
  103 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/cool.io-1.1.0/lib/cool.io/meta.rb
  104 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/cool.io-1.1.0/lib/cool.io/io.rb
  105 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/cool.io-1.1.0/lib/cool.io/iowatcher.rb
  106 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/cool.io-1.1.0/lib/cool.io/timer_watcher.rb
  107 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/cool.io-1.1.0/lib/cool.io/async_watcher.rb
  108 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/cool.io-1.1.0/lib/cool.io/listener.rb
  109 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/cool.io-1.1.0/lib/cool.io/dns_resolver.rb
  110 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/x86_64-darwin10.8.0/fcntl.bundle
  111 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/timeout.rb
  112 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/x86_64-darwin10.8.0/digest.bundle
  113 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/digest.rb
  114 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/x86_64-darwin10.8.0/openssl.bundle
  115 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/openssl/bn.rb
  116 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/openssl/cipher.rb
  117 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/openssl/config.rb
  118 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/openssl/digest.rb
  119 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/openssl/buffering.rb
  120 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/openssl/ssl-internal.rb
  121 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/openssl/x509-internal.rb
  122 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/openssl.rb
  123 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/securerandom.rb
  124 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/resolv.rb
  125 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/cool.io-1.1.0/lib/cool.io/socket.rb
  126 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/cool.io-1.1.0/lib/cool.io/server.rb
  127 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/cool.io-1.1.0/lib/http11_client.bundle
  128 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/cool.io-1.1.0/lib/cool.io/http_client.rb
  129 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/cool.io-1.1.0/lib/cool.io/dsl.rb
  130 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/cool.io-1.1.0/lib/cool.io.rb
  131 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/cool.io-1.1.0/lib/cool.io/eventmachine.rb
  132 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/status.rb
  133 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/config.rb
  134 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/engine.rb
  135 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/mixin.rb
  136 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/singleton.rb
  137 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/process.rb
  138 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/plugin.rb
  139 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/parser.rb
  140 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/event.rb
  141 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/buffer.rb
  142 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/input.rb
  143 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/output.rb
  144 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/match.rb
  145 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/load.rb
  146 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/plugin/buf_file.rb
  147 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/plugin/buf_memory.rb
  148 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/plugin/buf_zfile.rb
  149 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/plugin/in_debug_agent.rb
  150 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/plugin/in_exec.rb
  151 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/plugin/in_forward.rb
  152 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/plugin/in_gc_stat.rb
  153 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/http_parser.rb-0.5.3/lib/ruby_http_parser.bundle
  154 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/http_parser.rb-0.5.3/lib/http_parser.rb
  155 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/http_parser.rb-0.5.3/lib/http/parser.rb
  156 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/plugin/in_http.rb
  157 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/plugin/in_object_space.rb
  158 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/plugin/in_status.rb
  159 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/plugin/in_stream.rb
  160 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/plugin/in_syslog.rb
  161 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/plugin/in_tail.rb
  162 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/plugin/out_copy.rb
  163 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/plugin/out_exec.rb
  164 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/plugin/out_exec_filter.rb
  165 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/plugin/out_file.rb
  166 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/plugin/out_forward.rb
  167 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/plugin/out_null.rb
  168 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/plugin/out_roundrobin.rb
  169 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/plugin/out_stdout.rb
  170 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/plugin/out_stream.rb
  171 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluentd-0.10.28/lib/fluent/plugin/out_test.rb
  172 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/delegate.rb
  173 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/tmpdir.rb
  174 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/tempfile.rb
  175 /Users/muddydixon/.rvm/gems/ruby-1.9.3-p125@global/gems/fluent-plugin-forest-0.1.2/lib/fluent/plugin/out_forest.rb
  176 /Users/muddydixon/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/x86_64-darwin10.8.0/zlib.bundle

[NOTE]
You may have encountered a bug in the Ruby interpreter or extension libraries.
Bug reports are welcome.
For details: http://www.ruby-lang.org/bugreport.html

multiline messages in in_syslog aren't handled

I don't know if this is the case for all multiline messages but we have observed it using the following plugins:

in_syslog (as the source)
out_rewrite
out_forest
out_s3

What appears to be happening is any log entries that are multiline, such as stack traces, are being output to td-agent.log rather than to the configured 'match'.

I can send more details if required.

disable time formatting

I do no wish my timestamps to be formatted.
This does not appear to be possible, at present it defaults to iso8601.

Perhaps if config had:

disable_time_format true

then we could conditionally format time.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.