Document the working of QueueCapacity/BatchSize/BatchTimeout about kafka-go HOT 2 CLOSED

@dgregoire are you able to share which version of Kafka you had done these tests against, as well as the number of brokers and partitions in the topic? The writer definitely shouldn't block until the deadline is reached, because we have tests that verify its behavior.

If it timed out due to reaching the context deadline you should be getting en error returned from the (*Writer).WriteMessages call, is this what you observed? (seems like it from your screenshot but I just wanted to confirm)

A batch size of 2 is quite small, I would expect this to put significant load on the Kafka brokers, which means some requests could timeout if there is concurrency involved and you hit hardware limits on the underlying hardware (like disk bandwidth, IOPS, CPU, etc...).

from kafka-go.

@achille-roussel Kafka 1.0.1 with 3 brokers, 12 partitions / topic.

The specific use case is:

• default batch size of 5/2
• context with timeout in 100/200/500ms
• WriteMessages() with 1 kafka.Message

In all combinaisons I tested, the WriteMessages function locks until the context is deadline exceeded and the function returns that error, unless the configured batch size is 1.

I suspected this might happen when using send messages with a number of message != batch size but never proved that theory, what do you think?

from kafka-go.