ecwws / fluent-plugin-elasticsearch-timestamp-check Goto Github PK

Since fluentd supports nanosecond timestamp, could we extend this plugin to support epoch nano as well?

We have one source that currently set timestamp field to epoch nano, so at the moment we need to do a transformation to millisecs, would be nice if the plugin supported it out-of-the-box.

how to remove fluent_converted_timestamp or fluent_added_timestamp?
thanks. :)

Hey i added conf in following way but i still don't have value in milliseconds.

<filter **>
  type elasticsearch_timestamp_check
  subsecond_precision 3
</filter>

@timestamp |   | October 1st 2020, 11:48:23.000
fluent_added_timestamp |   | October 1st 2020, 11:48:23.000

Hello, ecwws, thanks for this plugin.

Elasticsearch support:

epoch_millis
A formatter for the number of milliseconds since the epoch. Note, that this timestamp is subject to the limits of a Java Long.MIN_VALUE and Long.MAX_VALUE.
epoch_second
A formatter for the number of seconds since the epoch. Note, that this timestamp is subject to the limits of a Java Long.MIN_VALUE and Long. MAX_VALUE divided by 1000 (the number of milliseconds in a second).

And I checked source code, DateTime.parse only support from datetime string to DateTime, not support timestamp(epoch_millis/epoch_second).

How about add epoch_millis and epoch_second support?

In the case @timestamp (and fluent_added_timestamp) is added from fluentd event time, the timestamp is truncated to seconds.

The cause is the conversion of the time to an integer (Fluent::EventTime - to_int)

(It is the same issue as #15 but it was not described good enough)

Proposed solution

Use to_r method instead (Fluent::EventTime - to_r which converts it in a Time object ready to use with Time.at

I'm curious why not use Time.at(time) in the fallback option instead of Time.now, here

AFAIU, every event that is ingested in fluntd has an internal time key, either extracted from the event record (using formatting/parsing) or by using the time when it was ingested, since the event can travel through the pipeline, by the time it reaches this plugin Time.now doesn't represent the actual ingestion time of the event.

Any thoughts? @ecwws @cosmo0920

I'm observing that some events are not processed correctly due to a log statement issue.

Given an event like:

{
  "timestamp": "10/Aug/2021:22:48:47 +0000",
  "access": "GET / HTTP/1.1 200",
  "type": "access"
}

(that is, not using the "normal timestamp processing" but instead the secondary/backup parsing)

The result is an error such as:

2021-08-10 22:48:52 +0000 [warn]: #0 dump an error event: error_class=NameError error="undefined local variable or method `field' for #<Fluent::Plugin::ElasticsearchTimestampCheckFilter:0x00007f1780cc19a8>\nDid you mean?  yield" location="/usr/local/bundle/gems/fluent-plugin-elasticsearch-timestamp-check-0.3.0/lib/fluent/plugin/filter_elasticsearch_timestamp_check.rb:62:in `rescue in block in filter'" tag="docker.var.lib.docker.containers.83e55d29166ac414218614dda4e51e0409d9844e0629d50128edbb5e3bc6c9d7.83e55d29166ac414218614dda4e51e0409d9844e0629d50128edbb5e3bc6c9d7-json.log" time=2021-08-10 

22:48:52.731255982 +0000 record={"timestamp"=>"10/Aug/2021:22:48:47 +0000", "access"=>"GET / HTTP/1.1 200"}

It seems like the issue is that field is out of scope at this line (but I don't know Ruby..):

error_class=Fluent::ConfigError error=Unknown output plugin 'elasticsearch_timestamp_check'

sudo gem install fluent-plugin-elasticsearch-timestamp-check

<match pre.**>
  type elasticsearch_timestamp_check
</match>

(also tried new syntax @type)