Multiple proxies with different client certificates about kafka-proxy HOT 7 CLOSED

FTR the clients have access to different topics, for example Client A has ACLs for topic A but not topic B and Client B has ACLs for topic B but not topic A

from kafka-proxy.

So far the proxy can load only one client certificate. This could be extended by providing a list of client certs to load https://github.com/grepplabs/kafka-proxy/blob/master/proxy/tls.go#L187

Each proxy works independently, so you can configure multiple proxy / proxy groups. The clients and proxies should be configured in that way that the traffic from different groups is not mixed.

There is no real need to use StatefulSet, you can use Deployment as well.
Provided snippet sets up 1 proxy per broker. In this setup proxy must know mapping of other proxies (list of external-server-mappings ), so STS is a ways to provided consistent naming.

from kafka-proxy.

Thanks for quick response @everesio !

Actually I don't need more than one client certificate per proxy group. I prefer each client to have separate proxy group, this is multi-tenant scenario so having each tenant proxy in different pod enables proxy resource limits (CPU, Memory, Open connections) on tenant level. Client connects to single proxy (tenant proxy) with proper client certificate (tenant certificate) so traffic is not mixed. So all good here.

Thanks for explaining the reason for StatefulSet, I agree it simplifies setup. Given my Kafka knowledge I am bit afraid about particular scenario:

1. Kafka with 3 brokers and 3 STS proxies, each proxy maps one broker as bootstrap and 2 others as external as in the example
2. Because of network problem one stateful set goes into unknown state
   In this scenario K8s will not automatically replace the pod (pod is not reachable) and will not add new one. From Kafka perspective all 3 brokers are healthy so Kafka will continue advertise healthy broker with unhealthy proxy, the other two proxies will also continue mapping to the unhealthy proxy. Manual intervention for unhealthy STS will be needed
   Feel free to clarify if I misunderstood anything

We use AWS so I am thinking about following alternative:

1. Setup EC2 with ENI for each proxy (ENI provides static private IP)
2. In case when EC2 is unhealthy launch new EC2 and reassign network interface to it
   This is just an idea, not sure how it works out (seems to me this is the approach that AWS MSK uses for replacing unhealthy nodes)

I also plan to try with K8s. However, as you mentioned without stateful set I will not have static network interface so it would be necessary to restart two other proxies in order to change external server mappings for replaced pod. Need to think more about it, maybe DNS or some K8s resources will address it.

Anyways things look promising. This project is really unique offering in Kafka proxies space right now :) Thank you for maintaining it

from kafka-proxy.

For more HA, you can always increase number replicas of each group (STS).
Each replica from the same STS would represent the same broker node.
Each K8S service would load balance ( for new connections) between the replicas of one STS.

from kafka-proxy.

@everesio good to know that I can have more than one instance of proxy behind loadbalancer port. I wonder if this works in case of TLS and client certificates. We use AWS NLB with SSL passthrough so that client negotiates certificates with proxy and NLB (which is in the middle) just forwards the requests back and forth. AWS NLB does not have sticky sessions so may forward randomly, not sure if it is going to stick cert negotiation process with one proxy server and not fallback to another. We can test it though so thanks for the tip.

from kafka-proxy.

client will see only names of N load balancers (each LB can represent a Kafka broker).
it is transparent if behind one LB there are M kafka proxies pods connecting to one kafka broker (together N*M proxy pods).
TLS cert are checked only during connection creation and connections are persistent.

from kafka-proxy.

@everesio thank you for detailed explanation. This resolves my concerns.

from kafka-proxy.