vanilla / fluent-plugin-burrow Goto Github PK

This plugin for Fluentd allows to extract a single key from an existing record and re-parse it with a supplied format.

License: MIT License

Ruby 100.00%

fluent-plugin-burrow's Introduction

[![](https://img.shields.io/github/license/vanilla/vanilla.svg)](https://github.com/vanilla/vanilla/blob/master/LICENSE) [![CircleCI](https://circleci.com/gh/vanilla/vanilla/tree/master.svg?style=svg)](https://circleci.com/gh/vanilla/vanilla/tree/master)

Howdy, Stranger!

Vanilla was born out of the desire to create flexible, customizable, and downright entertaining community solutions. Vanilla has been used to power tens of thousands of community forums around the world and we couldn't be happier if you've decided to use Vanilla to grow yours.

Forum	Rich Editor	Dashboard

Every community is unique. Vanilla is a finely-crafted platform on which designers and developers can build a custom-tailored environment that meets your community's particular needs.

5 reasons Vanilla is the sweetest forum

We've reimagined traditional forums for mass-appeal.
Our theming flexibility is second-to-none.
Impossibly good integration options with single sign-ons and embedding.
The best tools available for community management.
Curated features with great plugin options, not the kitchen sink.

Installation

The current version of Vanilla requires PHP 7.2+ and MySQL 5.7+. The following PHP extensions are also required: cURL, DOM, Fileinfo, GD, intl, JSON, libxml, PDO

Upgrading from an earlier version of Vanilla? See our upgrade notes.

Changes to Fulltext Indexing

Full-text index support has been disabled by default as of Vanilla 4. To enable full-text index support, add a FullTextIndexing key under the Database section of your site config and set its value to true. Failure to add this config value will result in full-text indexes being removed from Vanilla's database tables.

Contributing

Local Development - Environment, Configuration & Debugging & Build Tools.
Running Unit tests.
Coding Standard - PHP, Typescript, Database Naming
Writing Pull Requests
Contributing Guidelines
Contributing to Translations

Getting Help

Reporting Security Issues

Please disclose security issues responsibly by emailing [email protected] with a full description or join our bug bounty program. We cannot award bounties outside that program.

We'll work on releasing an updated version as quickly as possible. Please do not email non-security issues; use the issue tracker instead.

Building Releases

Vanilla releases are built using Phing to create a pre-built deploy-ready copies of Vanilla. To build these run the following in the root of the repository.

./bin/release

The following dependenies are all required for building a release.

node
yarn
php
composer

License & Legal Stuff

Vanilla is free, open source software distributed under the license gpl-2.0-only. We accept and encourage contributions from our community and sometimes give hugs in return.

Vanilla Forums is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 2 of the License, or (at your option) any later version.

Vanilla Forums is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with Vanilla Forums.

If not, see http://www.gnu.org/licenses/. Contact Vanilla Forums Inc. at support [at] vanillaforums [dot] com

Bonk!

Just kidding, everything's awesome.

fluent-plugin-burrow's People

Contributors

Stargazers

Watchers

Forkers

kozmagabor poison nelsonenzo cpoirier tgrayiwc isabella232

fluent-plugin-burrow's Issues

Release this plugin as a gem

https://rubygems.org/gems/fluent-plugin-burrow

Currently, burrow plugin is not available on rubygems.org.
Releasing plugin as a gem is useful for other users.

Release new version

First of all, thank you for nice plugin.
May I ask you to release new version of it to Gems repo?
Currently only old versions are available but #7 is quite valuable.
Also, it will be great to create Github releases which mirror versions of Gems repo and show current master branch state.

nested message field in message field gets lost when overlaying parsed message

Given a json structured rsyslog event which has a field named message which includes a jsonformatted json string from an application.
And fluent configuration such that the message field from rsyslog is parsed as json and overlayed over the original event.
When an event is emitted by an app which includes message as part of its log,
Then the burrow plugin parses the json, overlays it onto the main record but removes the `message field (which was parsed out of the message.

Rsyslog Configuration

# A lovely template
template(name="json-photon"
  type="list") {
    constant(value="{")
      constant(value="\"@timestamp\":\"")       property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"type\":\"syslog")
      constant(value="\",\"tag\":\"")           property(name="syslogtag" format="json")
      constant(value="\",\"relayhost\":\"")     property(name="fromhost")
      constant(value="\",\"relayip\":\"")       property(name="fromhost-ip")
      constant(value="\",\"logsource\":\"")     property(name="source")
      constant(value="\",\"hostname\":\"")      property(name="hostname" caseconversion="lower")
      constant(value="\",\"program\":\"")       property(name="programname")
      constant(value="\",\"priority\":\"")      property(name="pri")
      constant(value="\",\"severity\":\"")      property(name="syslogseverity")
      constant(value="\",\"facility\":\"")      property(name="syslogfacility")
      constant(value="\",\"severity_label\":\"")   property(name="syslogseverity-text")
      constant(value="\",\"facility_label\":\"")   property(name="syslogfacility-text")
      constant(value="\",\"message\":\"")       property(name="msg" format="json")
    constant(value="\"}\n")
}

# This line sends all lines to defined IP address at port 10034,
# using the "json-photon" format template

*.*                         @fluent:10034;json-photon

FluentD Configuration

<system>
  log_level trace
</system>

<source>
    port 10034
    bind 0.0.0.0
    tag syslog
    format json
    @type udp
    @id syslog-source
</source>


<filter>
     @type stdout
     @id debug-filter
</filter>


<match syslog>
  @type rewrite_tag_filter
  @id syslog-matcher
  <rule>
    key tag
    pattern /^(\w+)/
    tag syslog.$1
  </rule>
</match>

# Attempt to filter all events as if they have json
# and when they do, expand the object and
# overlay to the base of this record
# then reemit without the `syslog` prefix for processing
# like normal things

<match syslog.**>
    @type burrow
    key_name message
    format json
    remove_prefix syslog
    action overlay
</match>

<match **>
     @type stdout
     @id debug-matcher
</match>

Python Test Code

from logging.handlers import SysLogHandler
import logging

logger = logging.getLogger()

#add handler to the logger
handler = logging.handlers.SysLogHandler()


#add formatter to the handler
formatter = logging.Formatter('python: { "loggerName":"%(name)s", "timestamp":"%(asctime)s", "pathName":"%(pathname)s", "logRecordCreationTime":"%(created)f", "functionName":"%(funcName)s", "levelNo":"%(levelno)s", "lineNo":"%(lineno)d", "time":"%(msecs)d", "levelName":"%(levelname)s", "message":"%(message)s"}')

handler.formatter = formatter
logger.addHandler(handler)



#logger.info("Test INFO Message")
#logging.warn("Hello WARNING message")
#logging.error("Hello ERROR message")

logging.critical("Hello CRITICAL message")

Results

Fluent logs show (Formatted json for readability);

fluent_1   | 2019-11-23 12:24:02.203101000 +0000 syslog.python: 
{
  "@timestamp": "2019-11-23T12:24:02.202400+00:00",
  "type": "syslog",
  "tag": "python:",
  "relayhost": "172.18.0.1",
  "relayip": "172.18.0.1",
  "logsource": "172.18.0.1",
  "hostname": "172.18.0.1",
  "program": "python",
  "priority": "10",
  "severity": "2",
  "facility": "1",
  "severity_label": "crit",
  "facility_label": "user",
  "message": " { \"loggerName\":\"root\", \"timestamp\":\"2019-11-23 12:24:02,191\", \"pathName\":\"send_syslog_message.py\", \"logRecordCreationTime\":\"1574511842.191565\", \"functionName\":\"<module>\", \"levelNo\":\"50\", \"lineNo\":\"22\", \"time\":\"191\", \"levelName\":\"CRITICAL\", \"message\":\"Hello CRITICAL message\"}"
}

which includes the message from rsyslog, and the json formatted string.

Once it parses the event we then get

fluent_1   | 1970-01-01 00:03:11.000000000 +0000 python: 
{
  "@timestamp": "2019-11-23T12:24:02.202400+00:00",
  "type": "syslog",
  "tag": "python:",
  "relayhost": "172.18.0.1",
  "relayip": "172.18.0.1",
  "logsource": "172.18.0.1",
  "hostname": "172.18.0.1",
  "program": "python",
  "priority": "10",
  "severity": "2",
  "facility": "1",
  "severity_label": "crit",
  "facility_label": "user",
  "loggerName": "root",
  "timestamp": "2019-11-23 12:24:02,191",
  "pathName": "send_syslog_message.py",
  "logRecordCreationTime": "1574511842.191565",
  "functionName": "<module>",
  "levelNo": "50",
  "lineNo": "22",
  "levelName": "CRITICAL"
}

Which does not include the message key anymore.

Expected

The message key from the origin event is replaced with the message key parsed from the json-string the origin record contained.

can't convert nil into an exact number

@kaecyra I'm running into an issue trying to parse Docker logs using FluentD logging driver.

Error is:

2015-11-25 21:36:01 +0000 [warn]: emit transaction failed: error_class=TypeError error="can't convert nil into an exact number" tag="requestbin"
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin/out_stdout.rb:44:in `at'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin/out_stdout.rb:44:in `block in emit'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/event.rb:54:in `call'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/event.rb:54:in `each'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin/out_stdout.rb:43:in `emit'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/event_router.rb:88:in `emit_stream'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/engine.rb:116:in `emit_stream'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/engine.rb:107:in `emit'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-burrow-1.1/lib/fluent/plugin/out_burrow.rb:143:in `block in emit'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/event.rb:54:in `call'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/event.rb:54:in `each'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-burrow-1.1/lib/fluent/plugin/out_burrow.rb:102:in `emit'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/event_router.rb:88:in `emit_stream'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/event_router.rb:79:in `emit'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin/in_forward.rb:169:in `on_message'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin/in_forward.rb:243:in `call'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin/in_forward.rb:243:in `block in on_read_msgpack'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin/in_forward.rb:242:in `feed_each'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin/in_forward.rb:242:in `on_read_msgpack'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin/in_forward.rb:228:in `call'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin/in_forward.rb:228:in `on_read'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/io.rb:123:in `on_readable'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/io.rb:186:in `on_readable'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run_once'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run'
  2015-11-25 21:36:01 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin/in_forward.rb:91:in `run'
2015-11-25 21:36:01 +0000 [warn]: emit transaction failed: error_class=TypeError error="can't convert nil into an exact number" tag="marathon.requestbin"
  2015-11-25 21:36:01 +0000 [warn]: suppressed same stacktrace
2015-11-25 21:36:01 +0000 [error]: forward error error=#<TypeError: can't convert nil into an exact number> error_class=TypeError
  2015-11-25 21:36:01 +0000 [error]: suppressed same stacktrace
2015-11-25 21:36:01 +0000 [warn]: emit transaction failed: error_class=TypeError error="can't convert nil into an exact number" tag="requestbin"
  2015-11-25 21:36:01 +0000 [warn]: suppressed same stacktrace
2015-11-25 21:36:01 +0000 [warn]: emit transaction failed: error_class=TypeError error="can't convert nil into an exact number" tag="marathon.requestbin"

Logs are emitted with tag marathon.requestbin.

My FluentD conf:

<match marathon.requestbin*>
    type burrow
    key_name log
    action inplace
    remove_prefix marathon
    format json
</match>

<match requestbin*>
  type stdout
</match>

Here is what the log look like on stdout:

2015-11-25 21:38:11 +0000 marathon.requestbin: {"log":"10.42.122.245 - - [25/Nov/2015 21:38:11] \"GET /delay/2 HTTP/1.1\" 200 -","container_id":"467f6f962940172a8c5a21e2dce3354d23604ed73386975c37a533455989f822","container_name":"/mesos-a84767fb-caaf-46cb-921c-e083f441715a-S1.7e7380da-fb88-4429-b26a-8449ad6edd1b","source":"stderr"}
2015-11-25 21:38:21 +0000 marathon.requestbin: {"log":"10.42.122.245 - - [25/Nov/2015 21:38:21] \"GET /delay/2 HTTP/1.1\" 200 -","container_id":"467f6f962940172a8c5a21e2dce3354d23604ed73386975c37a533455989f822","container_name":"/mesos-a84767fb-caaf-46cb-921c-e083f441715a-S1.7e7380da-fb88-4429-b26a-8449ad6edd1b","source":"stderr"}

Any ideas? Thanks in advance for looking at it.

Convert to Filter Type Plugin

This is a great plugin but from my limited use it seems that the fact that it's an output plugin creates some unnecessary work (tagging and re-emission). I think it might make more sense to refactor it as a filter plugin.

Numeric time stamp causes time_key parsing to fail

If time_stamp is declared for a particular field in the JSON, and the field is an integer e.g. {"ts":1435527344}, and the time_format is declared as %s, the burrow plugin passes the values to the fluent parser but the fluent parser fails:

time_key ts
time_format %s

error_class=Fluent::ParserError error="value must be string: 1435527344"

The integer should be processed as a valid strftime epoch time stamp similar to:
% date -u --date=@1435527344
Sun Jun 28 21:35:44 UTC 2015

burrow may need to change the type in case of integer time stamps before submitting.

Thank you!