Comments (3)
jordansissel
commented on August 16, 2024
By default Logstash uses a search_type of scan which probably has the same or similar effect. (https://github.com/logstash-plugins/logstash-input-elasticsearch/blob/master/lib/logstash/inputs/elasticsearch.rb#L45-L47 and https://github.com/logstash-plugins/logstash-input-elasticsearch/blob/master/lib/logstash/inputs/elasticsearch.rb#L118)
Have you measured any performance differences with this change? If so, in what configurations?
from logstash-input-elasticsearch.
jimczi
commented on August 16, 2024
The scan search type always uses the sort by _doc optimization but watch out because this is deprecated in ES 2.1.0. You should consider switching to a regular scroll request where the sort by "_doc" is needed to have similar performance as @shaharmor suggested.
from logstash-input-elasticsearch.
untergeek
commented on August 16, 2024
This was handled in #51
from logstash-input-elasticsearch.
Related Issues (20)
- Integration testing on Elasticsearch 8.0.0-SNAPSHOT is broken
- Possible thread leak due to Rufus if not properly closed
- Cannot get new connection from pool when same pipeline read and write to same index HOT 7
- [Doc] Update links to new cloud content in Logstash Reference
- Clearing a scroll occasionally raises a 'too_long_frame_exception' HOT 7
- Allow the use of the size parameter inside a query
- investigate plugin CI failures (against snapshots)
- plugin fails to start with default hosts setting
- Use "search_after" instead of "scroll" HOT 1
- Improve UX of connection+product validation at register
- Test failing with JNR error: getprotobyname_r failed
- regression on Manticore 0.8.0 due port being part of host
- custom user agent no longer set since LS 7.16 HOT 1
- Add `schedule_every` setting HOT 1
- Failure to create an event from the payload can crash the plugin
- Add a "sincedb" type of mecanism HOT 1
- Hello, how can logstash-input-elasticsearch support the ignore_unavailable function of elasticsearch? HOT 1
- Results from all scrolls for each slice are stored in memory leading to OOM HOT 1
- The plugin doesn't work with Elasticsearch on cloud `cloud_id` & `api_key` settings.
- BREAKING: ssl_certificate_verification => true uses deprecated verifier since v4.17.0
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from logstash-input-elasticsearch.