Write requests may stall too long when the write buffer is full about greptimedb HOT 1 CLOSED

After adding some logs, I probably found the cause. Currently, a region only notifies the worker it belongs to after flushing memtables.

Suppose we have 3 workers A, B, and C. All workers are writing:

• Workers A and B are flushing more than 1/2 of the write buffer
• Worker C checks WriteBufferManagerImpl::should_flush_engine(). We already flush more than 1/2 of the write buffer so worker C won't flush
• Worker C checks WriteBufferManagerImpl::should_stall() and stalls requests
• Workers A, B, and C continue writing and stall more requests
• Workers A and B finish flushing but they can't notify Worker C to handle stalled requests
• All clients waiting for worker C will be blocked forever

To fix this issue, I think we could notify all workers after a flush job is finished.

from greptimedb.