Limit launch new MergeThread count to execute pending merges with max_merge_count about elasticsearch HOT 4 OPEN

Pinging @elastic/es-search (Team:Search)

from elasticsearch.

IO throttling is an interesting problem, in fact, we are working on adjusting the logic there substantially in Lucene: apache/lucene#13293

On modern NVMe drives, the way throttling works now doesn't make sense.

Have you tried turning off auto-throttling all together to see if your performance improves?
index.merge.scheduler.auto_throttle setting this to false.

from elasticsearch.

Thanks for you reply.

My cluster run on mechanical hard drive, 12 drives per node, i think io throttle was very useful for this. but elasticsearch's IndexThrottle is limit indexing when launched MergeThread count more than max_merge_count on below code:
"EngineMergeScheduler.java"

        public synchronized void beforeMerge(OnGoingMerge merge) {
            int maxNumMerges = mergeScheduler.getMaxMergeCount();
            if (numMergesInFlight.incrementAndGet() > maxNumMerges) {
                if (isThrottling.getAndSet(true) == false) {
                    logger.info("now throttling indexing: numMergesInFlight={}, maxNumMerges={}", numMergesInFlight, maxNumMerges);
                    activateThrottling();
                }
            }
        }

I just used "ElasticsearchConcurrentMergeScheduler.maybeStall" to limit to launch MergeThread to avoid IndexThrottle activate, and it was reduced a most of rejections for indexing

@benwtrent

from elasticsearch.

Pinging @elastic/es-distributed (Team:Distributed)

from elasticsearch.