Use Time.at(time) instead of Time.now for fallback about fluent-plugin-elasticsearch-timestamp-check HOT 10 CLOSED

For practical purposes, the difference between event time and the time obtained using Time.now are probably negligible. However, the point you brought up, using time field of the object that was supplied to the plugin does have a certain valid use case: parser plugins that has a setting of time_key will cause the time field to be set using the field from the source data.

Let me test a couple things out, then I'll have it fixed up.

from fluent-plugin-elasticsearch-timestamp-check.

@dannyk81 sorry it took long, 0.2.8 should be using the event time instead of Time.now for added timestamp

from fluent-plugin-elasticsearch-timestamp-check.

@dannyk81 sorry, work suddenly got really really busy, I'm going to try to figure this out tonight

from fluent-plugin-elasticsearch-timestamp-check.

Because msgpack cannot handle Time object.
In record, users have to insert timestamp as String object or something which contains Time information.
For using elasticsearch, milliseconds time resolution(.%L) is enough.

from fluent-plugin-elasticsearch-timestamp-check.

Thanks @cosmo0920, I'm probably missing something, but can't we use this:

Time.at(time).strftime('%Y-%m-%dT%H:%M:%S.%L%z')

Here's the full snippet (lines 48 - 52):

      unless record['fluent_converted_timestamp']
        record['@timestamp'] = record['fluent_added_timestamp'] =
          Time.at(time).strftime('%Y-%m-%dT%H:%M:%S.%L%z')
        $log.debug("Timestamp added: #{record['@timestamp']}")
      end

The filter should have access to tag, time and record ?

from fluent-plugin-elasticsearch-timestamp-check.

The filter should have access to tag, time and record ?

Within #filter, we should access tag, time, and record but we can only modify record.

from fluent-plugin-elasticsearch-timestamp-check.

Yes I uderstand, but I'm not proposing to modify the time object, just to derive the values for keys @timestamp & fluent_added_timestamp in the record from time

record['@timestamp'] = record['fluent_added_timestamp'] = Time.at(time).strftime('%Y-%m-%dT%H:%M:%S.%L%z')

This should work isn't it?

from fluent-plugin-elasticsearch-timestamp-check.

Thanks @ecwws 😄

In our deployment, we are using this plugin on the aggregation fluentd nodes, before sending the events to Elasticsearch. We try to keep our fluentd forwarders configuration as basic/simple as possible.

In this kind of setup, since the forwarders have a flush interval of 5 seconds, using Time.now on the aggregation nodes will always introduce a delay in the value of @timestamp compared to the actual ingestion time of the even.

from fluent-plugin-elasticsearch-timestamp-check.

@ecwws any luck with this?

from fluent-plugin-elasticsearch-timestamp-check.

@ecwws ping 😄

from fluent-plugin-elasticsearch-timestamp-check.