Interval is needed as config option about logstash-input-elasticsearch HOT 9 OPEN

I would also need this feature to periodically query data from on index, modifying it and writing it to a different index.

In general, I think it would make sense, that all input plugins that are not processing 'unlimited' streams should have unified scheduling options. Current situation is quite different, e.g.:

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html

• does not support scheduling

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-exec.html

• supports configuring an interval in seconds

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html

• supports cron-like scheduling (rufus): schedule => "* * * * *"

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http_poller.html

• supports different cron-like scheduling (rufus) A hash with + key: "cron" | "every" | "in" | "at" + value: string Examples: a) { "every" ⇒ "1h" } b) { "cron" ⇒ "* * * * * UTC" }

from logstash-input-elasticsearch.

+1 vote

from logstash-input-elasticsearch.

+1

from logstash-input-elasticsearch.

@JathinSanghvi @CraigFoote We don't' have a voting system, and a +1 comment doesn't provide enough details about what you are asking -- can you be specific?

If you want to contribute a +1 the best way to do that without sending a few hundred emails (a comment sends such emails) is to use Github's reaction feature (the smiley face at the top of every comment and issue).

from logstash-input-elasticsearch.

In terms of solving this, Elasticsearch already has facilities for searching on date ranges based on 'now' -- Check out these docs: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html#ranges-on-dates

from logstash-input-elasticsearch.

I believe it is similar to what I need:
I need to run query periodically by some interval, or better by cron schedule as in jdbc input plugin.
It might also be useful to have last_run_time as a variable - it can be used in query.

In terms of solving this, Elasticsearch already has facilities for searching on date ranges based on 'now' -- Check out these docs: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html#ranges-on-dates

That doesn't solve the issue of running query periodically.

from logstash-input-elasticsearch.

+1

from logstash-input-elasticsearch.

Some unified scheduling option would be great.
In the meantime I've ported Rufus based scheduling support with PR #81.
Along with logstash-filter-elasticsearch it is enough for me to do near-realtime lookup enrichments .

from logstash-input-elasticsearch.

scheduling by @davidecavestro (pr #81) has been merged into master and released in version 4.2.0 of this plugin

from logstash-input-elasticsearch.