Git Product home page Git Product logo

fluent-plugin-docker-journald-concat's Introduction

fluent-plugin-docker-journald-concat

Gem Version

Fluentd Filter plugin to concatenate partial log messages generated by Docker daemon with Journald logging driver

Requirements

fluent-plugin-docker-journald-concat fluentd ruby
>= 0.1.3 >= v0.14.0 >= 2.1

Installation

Add this line to your application's Gemfile:

gem 'fluent-plugin-docker-journald-concat'

And then execute:

$ bundle

Or install it yourself as:

$ gem install fluent-plugin-docker-journald-concat

Configuration

key

The key to be concatenated for partial events. Default value is "message".

match_key

The key which indicates a partial event. Default value is "container_partial_message".

stream_identity_key

The key to determine which stream an event belongs to. Default value is "container_id".

flush_interval

The number of seconds after which the last received event will be flushed. If specified 0, wait for next line forever. Default value is "60".

timeout_label

The label assigned to timed out events. Default value is "nil".

Usage

Insert this into your Fluentd config to get started.

<filter docker.**>
  @type docker_journald_concat
</filter>

The full Fluentd config example with Systemd Journal input.

<source>
  @type systemd
  tag docker
  path /var/log/journal
  read_from_head true
  filters [{ "_TRANSPORT": "journal", "_COMM": "dockerd" }]
  <storage>
    @type local
    persistent true
    path /tmp/journal.pos
  </storage>
  <entry>
    field_map {
      "_SOURCE_REALTIME_TIMESTAMP": "timestamp",
      "_HOSTNAME": "hostname",
      "PRIORITY": "priority",
      "MESSAGE": "message",
      "CONTAINER_NAME": "container_name",
      "CONTAINER_ID": "container_id",
      "CONTAINER_PARTIAL_MESSAGE": "container_partial_message"
    }
    field_map_strict true
  </entry>
</source>

<filter docker.**>
  @type record_modifier
  <record>
    # {"6" => "stdout", "3" => "stderr"}
    source ${record["priority"] == "6" ? "stdout" : "stderr"}
    timestamp ${record["timestamp"].to_i / 1000}
  </record>
  remove_keys priority
</filter>

<filter docker.**>
  @type docker_journald_concat
</filter>

<match docker.**>
  @type stdout
</match>

You can handle timeout events and remaining buffers on shutdown of this plugin.

<label @ERROR>
  <match docker.log>
    @type file
    path /path/to/error.log
  </match>
</label>

Handle timed out log lines the same as normal logs.

<filter **>
  @type docker_journald_concat
  timeout_label @NORMAL
</filter>

<match **>
  @type relabel
  @label @NORMAL
</match>

<label @NORMAL>
  <match **>
    @type stdout
  </match>
</label>

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

License

The gem is available as open source under the terms of the MIT License.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.