Racecondition on Index Cycle (manual or by retention config) with IndexBlockCheck about graylog2-server HOT 2 OPEN

Optimizations here could be adapted directly for this issue: #18563

from graylog2-server.

Hello Henry, thanks for raising this!

Just to note, flapping leader usually indicates that the HTTP thread pool on the leader is being exceeded.

We observe this occur on clusters that have many hundreds of Sidecars that phone home frequently on top of other performance issues (can reduce impact by increasing the phone home interval of sidecars), clusters that have MongoDB using low IOPS storage (less than 3000 IOPS), clusters with alerts set to run very frequently (every x seconds) and on clusters where in the server.conf the sum of values for the config keys below is too high:

inputbuffer_processors
processbuffer_processors
outputbuffer_processors

from graylog2-server.