fluent / fluent-plugin-scribe Goto Github PK

We deployed fluentd to production using this plugin along with the out_redshift plugin.

Even during our initial benchmarks we saw that working with in_scribe gives far worse results than working with other input methods (like in_forward, which was giving 18kmsg/sec vs. 1kmsg/sec with in_scribe). But when we pushed real production traffic with all the plugins setup (during benchmark we used only in_scribe and out_file) it just couldn't handle the load (we're talking about ~300msg/sec).

It looks like the culprit is that all the message handling is happening on the same thread as the one that receives the Scribe messages and there is no actual use of Cool.io. So very often the processing gets delayed for some reason, the Scribe server will get a timeout and will stop sending data in until the retry period ends. But even then after a minute or so it dies again.

We worked around this issue by having in_scribe enqueue all the messages into a Queue and have another thread that will call Engine.emit on the messages in the queue. But this is sub optimal and far from being "production ready".

We are sending logs to both elasticsearch and scribe. For elasticsearch, we are able to run multiple workers in the same machine. We are getting error when trying to run multiple workers for scribe.

Currently, I can only specify one msg_format in my config. I have entries that both include JSON and don't include JSON. I'd like the ability to parse both.

Ideally it should try to parse the object via JSON, and then fall back to text if it can't.