Clickhouse flushing algo about dnsmonster HOT 2 CLOSED

I agree it's a bit confusing but it's there for a reason. I used to have the algorithm you're mentioning. meaning it did batching but no matter if the batch is full or not, flush would happen in intervals. but the performance of that hit a ceiling after 200k request per second.

At the moment my assumption is when you set up dnsmonster you know if you're in a low-volume env (flush interval) or a high-volume env (batch count).

from dnsmonster.

I find it a bit strange you get this performance issue. It is probably that when you are uploading to clickhouse you are not processing anything from the channel. Running multiple clickhouse consumer processes, I'd probably rather recommend spawning a new goroutine to handle the upload of the data at each interval so that the consumer loop continues running all the time. I'd guess this may be your bottleneck.

from dnsmonster.