[Improve] epoch_millis and epoch_second support about fluent-plugin-elasticsearch-timestamp-check HOT 6 CLOSED

understood. I'll try to get the update in the next day or two

from fluent-plugin-elasticsearch-timestamp-check.

How about like this:

      if existing
        if existing.is_a? Float or existing.is_a? Integer   # is epoch_second or epoch_millis
          if existing.to_i.to_s.length == 10  # epoch_second
            record['@timestamp'] =
              Time.at(existing).strftime('%Y-%m-%dT%H:%M:%S.%L%z')
            record['fluent_converted_timestamp'] =
              Time.at(existing).strftime('%Y-%m-%dT%H:%M:%S.%L%z')
          elsif existing.to_i.to_s.length == 13  # epoch_millis
            record['@timestamp'] =
              Time.at(existing/1000).strftime('%Y-%m-%dT%H:%M:%S.%L%z')
            record['fluent_converted_timestamp'] =
              Time.at(existing/1000).strftime('%Y-%m-%dT%H:%M:%S.%L%z')
          # else
          #   invalid format, do nothing
        else
          record['@timestamp'] =
            DateTime.parse(existing).strftime('%Y-%m-%dT%H:%M:%S.%L%z')
          record['fluent_converted_timestamp'] =
            DateTime.parse(existing).strftime('%Y-%m-%dT%H:%M:%S.%L%z')
        $log.debug("Timestamp parsed: #{record['@timestamp']}")

from fluent-plugin-elasticsearch-timestamp-check.

Thank you for your input.

I do however have one question about this. This filter is intended for setting the @timestamp field to correct format before it hits elasticsearch so the data can be indexed correctly for tools that utilizes logstash-style index. If I understand your comments correctly, what you are proposing is to give this filter the ability to accept epoc_second/epoc_millis as valid @timestamp field input and convert them to proper iso8601 format. Am I understanding this correctly?

I guess what confused me is your mention of elasticsearch's support of epoch_millis/epoch_second, where if the target elasticsearch is setup to accept epoc_millis/epoc_second (which I believe is what elasticsearch >=2.0 defaults to on Date type), wouldn't we want to just pass the value through untouched (aside from ensuring they're of type integer) so elasticsearch would do the proper handling of the data?

I guess the better solution would be putting in a configuration to selectively enable/disable the epoc_millis/epoc_seconds passthrough rather than always forcing the conversion. What do you think?

from fluent-plugin-elasticsearch-timestamp-check.

@ecwws Thanks for your reply.

I do however have one question about this. This filter is intended for setting the @timestamp field to correct format before it hits elasticsearch so the data can be indexed correctly for tools that utilizes logstash-style index. If I understand your comments correctly, what you are proposing is to give this filter the ability to accept epoc_second/epoc_millis as valid @timestamp field input and convert them to proper iso8601 format. Am I understanding this correctly?

Yes, you are right, I made a mistake, epoc_second/epoc_millis does not need convert them to isoformat.

I guess what confused me is your mention of elasticsearch's support of epoch_millis/epoch_second, where if the target elasticsearch is setup to accept epoc_millis/epoc_second (which I believe is what elasticsearch >=2.0 defaults to on Date type), wouldn't we want to just pass the value through untouched (aside from ensuring they're of type integer) so elasticsearch would do the proper handling of the data?

I guess the better solution would be putting in a configuration to selectively enable/disable the epoc_millis/epoc_seconds passthrough rather than always forcing the conversion. What do you think?

Sound great, it's a good idea add a configuration to passthrough.

One more suggestion, we using elasticsearch dynamic mapping which only accept strict_date_optional_time by default, exclude epoc_second/epoc_millis, and I tried some test like below:

1. isoformat: "@timestamp" : "2017-04-17T07:31:09.323+0000",

   • all works, dynamic type is "date"

2. epoc_second string, int or float: "@timestamp" : 1492481059

   • dynamic type is "string/long/double"
   • setting type date default: ("format" : "strict_date_optional_time||epoch_millis"), post success, but does not be identified as date.
   • setting type date("format" : "strict_date_optional_time||epoch_millis||epoc_second"), all works.

3. epoc_millis string, int or float: "@timestamp" : 1492481059000

   • dynamic type is "string/long/double"
   • setting type date default: ("format" : "strict_date_optional_time||epoch_millis"), post success, but does not recones as date.

So, I think iso format is the best date format for dynamic mapping, and hope add other configuration which will force transfer epoch_millis/epoch_millis to iso format.

from fluent-plugin-elasticsearch-timestamp-check.

sorry took so long, things got busy. I opted as you recommended to force convert to iso format. I'll add an option to passthrough the seconds/millis at a later time. If there's an urgent need for that, please open another issue I'll try to address it asap.

from fluent-plugin-elasticsearch-timestamp-check.

gem version 0.2.4 and above contains epoch seconds/millis support

from fluent-plugin-elasticsearch-timestamp-check.