Implement scrollapi about elastico HOT 7 CLOSED

Hi @SMAPPER
The -Form parameter is actually working. The problem is still the 10000 records of #2.
As suggested by the error message you get, I need to add support for the scroll api in the cmdlet.
I will try to have a look at it this weekend.

from elastico.

Have you been able to look into this more? I really appreciate what you are doing with this module.

I originally thought when you added the -From that I could increment and do this:

Search-ElasticV5 -Node es01:9200 -Index logstash-* -Size 10000 -From 0
Search-ElasticV5 -Node es01:9200 -Index logstash-* -Size 10000 -From 10000
Search-ElasticV5 -Node es01:9200 -Index logstash-* -Size 10000 -From 20000

And so on and so forth. Then it would just be might job to create a proper loop based on a count. Right now it looks like it is currently adding the from parameter with the size parameter and searching for more than 10,000 records which causes this to fail.

Example:

Search-ElasticV5 -Node es01:9200 -Index logstash-* -Size 10000 -From 0

This successfully pulls back 10,000 records.

Search-ElasticV5 -Node es01:9200 -Index logstash-* -Size 10000 -From 10000

This fails stating it is trying to pull 20,000 records.

Search-ElasticV5 -Node es01:9200 -Index logstash-* -Size 10000 -From 20000

This fails stating it is trying to pull 30,000 records.

from elastico.

Hi @SMAPPER
In order to do that with the current module you need to change the index.max_result_window index setting, by increasing it for example to 30000 and you won't get the error.

The proper fix would be for the powershell module to implement the scrollapi. I implemented the feature but during my tests, it only works with elasticsearch v1 and v2 but not v5. As I'm currently investigating where exactly the issue comes from, the change is still not published. I will get back here when I manage to find more time to investigate the problem.

from elastico.

from elastico.

Hi,
I published a new version of the module that implements the scroll api.
So forget about the multiple -From -Size.
Just have a first call to the search with the -Scroll switch.
The first call will now return some records and also a ScrollId
You then need to have further calls just specifing the -ScrollId 'id' to fetch more records until the call returns no records.
Hope this is clear... try and let me know how it goes.

from elastico.

Just had time to sit down and play with this. Not sure I'm using it right though. Could you provide an example of using this to rotate through results until all have been pulled out?

from elastico.

I haven't seen any update recently here so considering this closed.

from elastico.