Git Product home page Git Product logo

fluent-plugin-burrow's Issues

nested message field in message field gets lost when overlaying parsed message

Given a json structured rsyslog event which has a field named message which includes a jsonformatted json string from an application.
And fluent configuration such that the message field from rsyslog is parsed as json and overlayed over the original event.
When an event is emitted by an app which includes message as part of its log,
Then the burrow plugin parses the json, overlays it onto the main record but removes the `message field (which was parsed out of the message.

Rsyslog Configuration

# A lovely template
template(name="json-photon"
  type="list") {
    constant(value="{")
      constant(value="\"@timestamp\":\"")       property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"type\":\"syslog")
      constant(value="\",\"tag\":\"")           property(name="syslogtag" format="json")
      constant(value="\",\"relayhost\":\"")     property(name="fromhost")
      constant(value="\",\"relayip\":\"")       property(name="fromhost-ip")
      constant(value="\",\"logsource\":\"")     property(name="source")
      constant(value="\",\"hostname\":\"")      property(name="hostname" caseconversion="lower")
      constant(value="\",\"program\":\"")       property(name="programname")
      constant(value="\",\"priority\":\"")      property(name="pri")
      constant(value="\",\"severity\":\"")      property(name="syslogseverity")
      constant(value="\",\"facility\":\"")      property(name="syslogfacility")
      constant(value="\",\"severity_label\":\"")   property(name="syslogseverity-text")
      constant(value="\",\"facility_label\":\"")   property(name="syslogfacility-text")
      constant(value="\",\"message\":\"")       property(name="msg" format="json")
    constant(value="\"}\n")
}

# This line sends all lines to defined IP address at port 10034,
# using the "json-photon" format template

*.*                         @fluent:10034;json-photon

FluentD Configuration

<system>
  log_level trace
</system>

<source>
    port 10034
    bind 0.0.0.0
    tag syslog
    format json
    @type udp
    @id syslog-source
</source>


<filter>
     @type stdout
     @id debug-filter
</filter>


<match syslog>
  @type rewrite_tag_filter
  @id syslog-matcher
  <rule>
    key tag
    pattern /^(\w+)/
    tag syslog.$1
  </rule>
</match>

# Attempt to filter all events as if they have json
# and when they do, expand the object and
# overlay to the base of this record
# then reemit without the `syslog` prefix for processing
# like normal things

<match syslog.**>
    @type burrow
    key_name message
    format json
    remove_prefix syslog
    action overlay
</match>

<match **>
     @type stdout
     @id debug-matcher
</match>

Python Test Code

from logging.handlers import SysLogHandler
import logging

logger = logging.getLogger()

#add handler to the logger
handler = logging.handlers.SysLogHandler()


#add formatter to the handler
formatter = logging.Formatter('python: { "loggerName":"%(name)s", "timestamp":"%(asctime)s", "pathName":"%(pathname)s", "logRecordCreationTime":"%(created)f", "functionName":"%(funcName)s", "levelNo":"%(levelno)s", "lineNo":"%(lineno)d", "time":"%(msecs)d", "levelName":"%(levelname)s", "message":"%(message)s"}')

handler.formatter = formatter
logger.addHandler(handler)



#logger.info("Test INFO Message")
#logging.warn("Hello WARNING message")
#logging.error("Hello ERROR message")

logging.critical("Hello CRITICAL message")

Results

Fluent logs show (Formatted json for readability);

fluent_1   | 2019-11-23 12:24:02.203101000 +0000 syslog.python: 
{
  "@timestamp": "2019-11-23T12:24:02.202400+00:00",
  "type": "syslog",
  "tag": "python:",
  "relayhost": "172.18.0.1",
  "relayip": "172.18.0.1",
  "logsource": "172.18.0.1",
  "hostname": "172.18.0.1",
  "program": "python",
  "priority": "10",
  "severity": "2",
  "facility": "1",
  "severity_label": "crit",
  "facility_label": "user",
  "message": " { \"loggerName\":\"root\", \"timestamp\":\"2019-11-23 12:24:02,191\", \"pathName\":\"send_syslog_message.py\", \"logRecordCreationTime\":\"1574511842.191565\", \"functionName\":\"<module>\", \"levelNo\":\"50\", \"lineNo\":\"22\", \"time\":\"191\", \"levelName\":\"CRITICAL\", \"message\":\"Hello CRITICAL message\"}"
}

which includes the message from rsyslog, and the json formatted string.

Once it parses the event we then get

fluent_1   | 1970-01-01 00:03:11.000000000 +0000 python: 
{
  "@timestamp": "2019-11-23T12:24:02.202400+00:00",
  "type": "syslog",
  "tag": "python:",
  "relayhost": "172.18.0.1",
  "relayip": "172.18.0.1",
  "logsource": "172.18.0.1",
  "hostname": "172.18.0.1",
  "program": "python",
  "priority": "10",
  "severity": "2",
  "facility": "1",
  "severity_label": "crit",
  "facility_label": "user",
  "loggerName": "root",
  "timestamp": "2019-11-23 12:24:02,191",
  "pathName": "send_syslog_message.py",
  "logRecordCreationTime": "1574511842.191565",
  "functionName": "<module>",
  "levelNo": "50",
  "lineNo": "22",
  "levelName": "CRITICAL"
}

Which does not include the message key anymore.

Expected

The message key from the origin event is replaced with the message key parsed from the json-string the origin record contained.

can't convert nil into an exact number

@kaecyra I'm running into an issue trying to parse Docker logs using FluentD logging driver.

Error is:

2015-11-25 21:36:01 +0000 [warn]: emit transaction failed: error_class=TypeError error="can't convert nil into an exact number" tag="requestbin"
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin/out_stdout.rb:44:in `at'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin/out_stdout.rb:44:in `block in emit'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/event.rb:54:in `call'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/event.rb:54:in `each'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin/out_stdout.rb:43:in `emit'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/event_router.rb:88:in `emit_stream'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/engine.rb:116:in `emit_stream'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/engine.rb:107:in `emit'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-burrow-1.1/lib/fluent/plugin/out_burrow.rb:143:in `block in emit'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/event.rb:54:in `call'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/event.rb:54:in `each'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-burrow-1.1/lib/fluent/plugin/out_burrow.rb:102:in `emit'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/event_router.rb:88:in `emit_stream'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/event_router.rb:79:in `emit'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin/in_forward.rb:169:in `on_message'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin/in_forward.rb:243:in `call'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin/in_forward.rb:243:in `block in on_read_msgpack'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin/in_forward.rb:242:in `feed_each'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin/in_forward.rb:242:in `on_read_msgpack'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin/in_forward.rb:228:in `call'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin/in_forward.rb:228:in `on_read'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/io.rb:123:in `on_readable'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/io.rb:186:in `on_readable'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run_once'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin/in_forward.rb:91:in `run'
2015-11-25 21:36:01 +0000 [warn]: emit transaction failed: error_class=TypeError error="can't convert nil into an exact number" tag="marathon.requestbin"
  2015-11-25 21:36:01 +0000 [warn]: suppressed same stacktrace
2015-11-25 21:36:01 +0000 [error]: forward error error=#<TypeError: can't convert nil into an exact number> error_class=TypeError
  2015-11-25 21:36:01 +0000 [error]: suppressed same stacktrace
2015-11-25 21:36:01 +0000 [warn]: emit transaction failed: error_class=TypeError error="can't convert nil into an exact number" tag="requestbin"
  2015-11-25 21:36:01 +0000 [warn]: suppressed same stacktrace
2015-11-25 21:36:01 +0000 [warn]: emit transaction failed: error_class=TypeError error="can't convert nil into an exact number" tag="marathon.requestbin"

Logs are emitted with tag marathon.requestbin.

My FluentD conf:

<match marathon.requestbin*>
    type burrow
    key_name log
    action inplace
    remove_prefix marathon
    format json
</match>

<match requestbin*>
  type stdout
</match>

Here is what the log look like on stdout:

2015-11-25 21:38:11 +0000 marathon.requestbin: {"log":"10.42.122.245 - - [25/Nov/2015 21:38:11] \"GET /delay/2 HTTP/1.1\" 200 -","container_id":"467f6f962940172a8c5a21e2dce3354d23604ed73386975c37a533455989f822","container_name":"/mesos-a84767fb-caaf-46cb-921c-e083f441715a-S1.7e7380da-fb88-4429-b26a-8449ad6edd1b","source":"stderr"}
2015-11-25 21:38:21 +0000 marathon.requestbin: {"log":"10.42.122.245 - - [25/Nov/2015 21:38:21] \"GET /delay/2 HTTP/1.1\" 200 -","container_id":"467f6f962940172a8c5a21e2dce3354d23604ed73386975c37a533455989f822","container_name":"/mesos-a84767fb-caaf-46cb-921c-e083f441715a-S1.7e7380da-fb88-4429-b26a-8449ad6edd1b","source":"stderr"}

Any ideas? Thanks in advance for looking at it.

Numeric time stamp causes time_key parsing to fail

If time_stamp is declared for a particular field in the JSON, and the field is an integer e.g. {"ts":1435527344}, and the time_format is declared as %s, the burrow plugin passes the values to the fluent parser but the fluent parser fails:

time_key ts
time_format %s

error_class=Fluent::ParserError error="value must be string: 1435527344"

The integer should be processed as a valid strftime epoch time stamp similar to:
% date -u --date=@1435527344
Sun Jun 28 21:35:44 UTC 2015

burrow may need to change the type in case of integer time stamps before submitting.

Thank you!

Convert to Filter Type Plugin

This is a great plugin but from my limited use it seems that the fact that it's an output plugin creates some unnecessary work (tagging and re-emission). I think it might make more sense to refactor it as a filter plugin.

Release new version

First of all, thank you for nice plugin.
May I ask you to release new version of it to Gems repo?
Currently only old versions are available but #7 is quite valuable.
Also, it will be great to create Github releases which mirror versions of Gems repo and show current master branch state.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.