Fluentd plugin to add or replace fields of a event record
See .travis.yml
Note that fluent-plugin-record-reformer supports both v0.14 API and v0.12 API in one gem.
Use RubyGems:
gem install fluent-plugin-record-reformer
Example:
<match foo.**>
type record_reformer
remove_keys remove_me
renew_record false
enable_ruby false
tag reformed.${tag_prefix[-2]}
<record>
hostname ${hostname}
input_tag ${tag}
last_tag ${tag_parts[-1]}
message ${record['message']}, yay!
</record>
</match>
Assume following input is coming (indented):
foo.bar {
"remove_me":"bar",
"not_remove_me":"bar",
"message":"Hello world!"
}then output becomes as below (indented):
reformed.foo {
"not_remove_me":"bar",
"hostname":"YOUR_HOSTNAME",
"input_tag":"foo.bar",
"last_tag":"bar",
"message":"Hello world!, yay!",
}Example:
<match foo.**>
type record_reformer
remove_keys remove_me
renew_record false
enable_ruby false
tag reformed.${tag_prefix[-2]}
hostname ${hostname}
input_tag ${tag}
last_tag ${tag_parts[-1]}
message ${record['message']}, yay!
</match>
This results in same, but please note that following option parameters are reserved, so can not be used as a record key.
-
output_tag (obsolete)
The output tag name. This option is deprecated. Use
tagoption instead -
tag
The output tag name.
-
remove_keys
Specify record keys to be removed by a string separated by , (comma) like
remove_keys message,foo -
renew_record bool
renew_record truecreates an output record newly without extending (merging) the input record fields. Default isfalse. -
renew_time_key string
renew_time_key foooverwrites the time of events with a value of the record fieldfooif exists. The value offoomust be a unix time. -
keep_keys
You may want to remain some record fields although you specify
renew_record true. Then, specify record keys to be kept by a string separated by , (comma) likekeep_keys message,foo -
enable_ruby bool
Enable to use ruby codes in placeholders. See
Placeholderssection. Default istrue(just for lower version compatibility). -
auto_typecast bool
Automatically cast the field types. Default is false. NOTE: This option is effective only for field values comprised of a single placeholder.
Effective Examples:
foo ${foo}Non-Effective Examples:
foo ${foo}${bar} foo ${foo}bar foo 1Internally, this keeps the type of value if the value text is comprised of a single placeholder, otherwise, values are treated as strings.
When you need to cast field types manually, out_typecast and filter_typecast are available.
Following placeholders are available:
-
${record["key"]} Record value of
keysuch as${record["message"]}in the above example (available from v0.8.0).- Originally, record placeholders were available as
${key}such as${message}. This is still kept for the backward compatibility, but would be removed in the future.
- Originally, record placeholders were available as
-
${hostname} Hostname of the running machine
-
${tag} Input tag
-
${time} Time of the event
-
${tags[N]} (Obsolete. Use tag_parts) Input tag splitted by '.'
-
${tag_parts[N]} Input tag splitted by '.' indexed with N such as
${tag_parts[0]},${tag_parts[-1]}. -
${tag_prefix[N]} Tag parts before and on the index N. For example,
Input tag: prefix.test.tag.suffix ${tag_prefix[0]} => prefix ${tag_prefix[1]} => prefix.test ${tag_prefix[-2]} => prefix.test.tag ${tag_prefix[-1]} => prefix.test.tag.suffix -
${tag_suffix[N]} Tag parts after and on the index N. For example,
Input tag: prefix.test.tag.suffix ${tag_suffix[0]} => prefix.test.tag.suffix ${tag_suffix[1]} => test.tag.suffix ${tag_suffix[-2]} => tag.suffix ${tag_suffix[-1]} => suffix
It is also possible to write a ruby code in placeholders if you set enable_ruby true option, so you may write some codes as
- ${time.strftime('%Y-%m-%dT%H:%M:%S%z')}
- ${tag_parts.last}
but, please note that enabling ruby codes is not encouraged by security reasons and also in terms of the performance.
Following plugins look similar:
- fluent-plugin-record-modifier
- fluent-plugin-format
- fluent-plugin-add
- filter_record_transformer is a Fluentd v0.12 built-in plugin which is based on record-reformer.
See CHANGELOG.md for details.
- Fork it
- Create your feature branch (
git checkout -b my-new-feature) - Commit your changes (
git commit -am 'Add some feature') - Push to the branch (
git push origin my-new-feature) - Create new Pull Request
Copyright (c) 2013 - 2015 Naotoshi Seo. See LICENSE for details.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent