QUESTION: open requests buffer is full about kafka-proxy HOT 8 CLOSED

I am not sure if this will be helpful.

• proxy-request-buffer-size and proxy-response-buffer-size are bytes buffers used in the go to write / read from socket
• proxy-listener-read-buffer-size and kafka-connection-read-buffer-size set SO_RCVBUF
• proxy-listener-write-buffer-size and kafka-connection-write-buffer-size set SO_SNDBUF

Default values in Kafka 2.2 for SO_RCVBUF / SO_SNDBUF are 102400.
In Java Drivers OS defaults are used.

For system changes you should probably search for "TCP Tuning" e.g. https://cromwell-intl.com/open-source/performance-tuning/tcp.html

from kafka-proxy.

I missed the error message.
"open requests buffer is full" error comes from the kafka proxy itself and it has nothing to do with request / socket buffers. It signifies that channel used during initial request (ApiVersions message) is full.
Messages from the channel are consumed when the Kafka initial response is received. This channel buffer size is 256 (param kafka-max-open-requests). So this should never happen when the kafka responses are received and when the clients and the proxy are configured properly.

Questions:

• When do you get this failure ?
• Do you received any traffic back from Kafka (firewall problem) ?
• How is your proxy (proxies) configured ?
• How the clients are connected to proxy ?
• Do you use ELB ?

from kafka-proxy.

I have tested that with the following setup:

• AWS MSK with Kafka 2.1.0
• EC2 instance with kafka-proxy in the same VPC as MSK (all open between it and MSK, open ports 32500,32501,32502 to producer)
• Producer outside MSK VPC

I get this failure when producer has published around 500 msgs.

I can also consume Kafka messages from the instance where producer is - I hope that this qualifies as 'receiving data back from Kafka'.

I run proxy with the following configuration:

--bootstrap-server-mapping "<kafka_bootstrap_ip_1>:9092,0.0.0.0:32500,<ip_of_proxy-instance>:32500"
--bootstrap-server-mapping "<kafka_bootstrap_ip_2>:9092,0.0.0.0:32501,<ip_of_proxy-instance>:32501"
--bootstrap-server-mapping "<kafka_bootstrap_ip_2>:9092,0.0.0.0:32502,<ip_of_proxy-instance>:32502"

Producer connects to <ip_of_proxy-instance> on all 3 ports.

ELB is not used.

from kafka-proxy.

I cannot reproduce the behavior you described.

I created a MKS and an EC2 instance where a proxy runs
https://gist.github.com/everesio/262e11c6e5cebf56f1d5111c8cd7da3f

After ssh to ec2 with kafka-proxy, starting the proxy, I could run kafka-producer-perf-test.sh from my laptop using kafka-proxy public IP without any problems ( throughput -1 and throughput 10)

I would suggest that you check your Security Groups / Firewall Rules.

from kafka-proxy.

Thanks for your help.
Your example works for me as well. The problem was with using non default producer, which apparently messes things. I used producer that comes with NiFi (tool I am using for data processing).

from kafka-proxy.

I missed the error message.
"open requests buffer is full" error comes from the kafka proxy itself and it has nothing to do with request / socket buffers. It signifies that channel used during initial request (ApiVersions message) is full.
Messages from the channel are consumed when the Kafka initial response is received. This channel buffer size is 256 (param kafka-max-open-requests). So this should never happen when the kafka responses are received and when the clients and the proxy are configured properly.

Questions:

* When do you get this failure ?

* Do you received any traffic back from Kafka (firewall problem) ?

* How is your proxy (proxies) configured ?

* How the clients are connected to proxy ?

* Do you use ELB ?

Hi @everesio. I am currently investigating a similar problem. The initial indicator here is, when a producer is configured with acks=0, I see this error quite often in my logs. With large message throughput, when this error appears, I see a significant drop in performance.

from kafka-proxy.

Hi @radekg,
thanks for narrowing down the problem. Kafka proxy expects a response from server even for ack=0.
I will create a bug issue.

RequiredAcks:
"This field indicates how many acknowledgements the servers should receive before responding to the request. If it is 0 the server will not send any response (this is the only case where the server will not reply to a request). If it is 1, the server will wait the data is written to the local log before sending a response. If it is -1 the server will block until the message is committed by all in sync replicas before sending a response."

from kafka-proxy.

@everesio thank you for this quick reply. I could help coming up with a fix, if I could receive some pointers to the place in the code where this can be handled?

from kafka-proxy.