Comments (4)
rajk0007
commented on June 3, 2024
1
@everesio My client is outside of kubernetes cluster.
I was finally able to figure out the issue. I was using nlb with the k8s service. In AWS, when nlb is created, it creates a target group which comes with tcp health check for each port exposed in service. kafka proxy didn;t understand this health check and failed with the message above. Solution was to use elb instead of nlb and expose port 9080(default healthcheck port) Doing this, there is no backend target group creation and healthcheck is only for first port listed in your service. I put port 9080 as first port(which maps to http healthcheck). This way there is no tcp check on broker listener port(in my case port 443)
Here is deployment and service yaml. Couple of things I also in deployment below is external dns integration and cert for proxy listener. This has nothing to do with the issue though.
---
apiVersion: v1
kind: Service
metadata:
labels:
app: kafka-proxy-1
name: kafka-proxy-1
annotations:
external-dns.alpha.kubernetes.io/hostname: broker1.mydomain.net
spec:
ports:
- name: health
port: 9080
protocol: TCP
targetPort: 9080
- name: kafka-proxy-1
port: 443
protocol: TCP
targetPort: 443
selector:
app: kafka-proxy-1
sessionAffinity: None
type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
labels:
app: kafka-proxy-2
name: kafka-proxy-2
annotations:
external-dns.alpha.kubernetes.io/hostname: broker2.mydomain.net
spec:
ports:
- name: health
port: 9080
protocol: TCP
targetPort: 9080
- name: kafka-proxy-2
port: 443
protocol: TCP
targetPort: 443
selector:
app: kafka-proxy-2
sessionAffinity: None
type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
labels:
app: kafka-proxy-3
name: kafka-proxy-3
annotations:
external-dns.alpha.kubernetes.io/hostname: broker3.mydomain.net
spec:
ports:
- name: health
port: 9080
protocol: TCP
targetPort: 9080
- name: kafka-proxy-3
port: 443
protocol: TCP
targetPort: 443
selector:
app: kafka-proxy-3
sessionAffinity: None
type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: kafka-proxy-1
spec:
replicas: 1
selector:
matchLabels:
app: kafka-proxy-1
template:
metadata:
labels:
app: kafka-proxy-1
spec:
hostNetwork: true
containers:
- name: kafka-proxy
securityContext:
capabilities:
add: ["NET_ADMIN", "SYS_TIME","NET_BIND_SERVICE"]
image: grepplabs/kafka-proxy:latest
args:
- 'server'
- '--log-format=json'
- '--bootstrap-server-mapping=b-1.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:443,broker1.mydomain.net:443'
- '--external-server-mapping=b-1.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,broker1.mydomain.net:443'
- '--external-server-mapping=b-2.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,broker2.mydomain.net:443'
- '--external-server-mapping=b-3.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,broker3.mydomain.net:443'
- '--tls-enable'
- '--log-level=debug'
- '--dynamic-listeners-disable'
- '--tls-insecure-skip-verify'
- '--proxy-request-buffer-size=32768'
- '--proxy-response-buffer-size=32768'
- '--proxy-listener-read-buffer-size=32768'
- '--proxy-listener-write-buffer-size=131072'
- '--kafka-connection-read-buffer-size=131072'
- '--kafka-connection-write-buffer-size=32768'
- '--proxy-listener-key-file=/opt/tls/tls.key'
- '--proxy-listener-cert-file=/opt/tls/tls.crt'
- '--proxy-listener-tls-enable'
ports:
- name: kafka-port1
containerPort: 443
- name: health
containerPort: 9080
volumeMounts:
- mountPath: /opt/tls
name: kafka-wildcard-cert
readOnly: true
volumes:
- name: kafka-wildcard-cert
secret:
defaultMode: 420
secretName: kafka-wildcard-cert
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: kafka-proxy-2
spec:
replicas: 1
selector:
matchLabels:
app: kafka-proxy-2
template:
metadata:
labels:
app: kafka-proxy-2
spec:
hostNetwork: true
containers:
- name: kafka-proxy
securityContext:
capabilities:
add: ["NET_ADMIN", "SYS_TIME","NET_BIND_SERVICE"]
image: grepplabs/kafka-proxy:latest
args:
- 'server'
- '--log-format=json'
- '--bootstrap-server-mapping=b-2.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:443,broker2.mydomain.net:443'
- '--external-server-mapping=b-1.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,broker1.mydomain.net:443'
- '--external-server-mapping=b-2.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,broker2.mydomain.net:443'
- '--external-server-mapping=b-3.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,broker3.mydomain.net:443'
- '--tls-enable'
- '--log-level=debug'
- '--dynamic-listeners-disable'
- '--tls-insecure-skip-verify'
- '--proxy-request-buffer-size=32768'
- '--proxy-response-buffer-size=32768'
- '--proxy-listener-read-buffer-size=32768'
- '--proxy-listener-write-buffer-size=131072'
- '--kafka-connection-read-buffer-size=131072'
- '--kafka-connection-write-buffer-size=32768'
- '--proxy-listener-key-file=/opt/tls/tls.key'
- '--proxy-listener-cert-file=/opt/tls/tls.crt'
- '--proxy-listener-tls-enable'
ports:
- name: kafka-port2
containerPort: 443
- name: health
containerPort: 9080
volumeMounts:
- mountPath: /opt/tls
name: kafka-wildcard-cert
readOnly: true
volumes:
- name: kafka-wildcard-cert
secret:
defaultMode: 420
secretName: kafka-wildcard-cert
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: kafka-proxy-3
spec:
replicas: 1
selector:
matchLabels:
app: kafka-proxy-3
template:
metadata:
labels:
app: kafka-proxy-3
spec:
hostNetwork: true
containers:
- name: kafka-proxy
securityContext:
capabilities:
add: ["NET_ADMIN", "SYS_TIME","NET_BIND_SERVICE"]
image: grepplabs/kafka-proxy:latest
args:
- 'server'
- '--log-format=json'
- '--bootstrap-server-mapping=b-3.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:443,broker3.mydomain.net:443'
- '--external-server-mapping=b-1.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,broker1.mydomain.net:443'
- '--external-server-mapping=b-2.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,broker2.mydomain.net:443'
- '--external-server-mapping=b-3.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,broker3.mydomain.net:443'
#- '--dial-address-mapping=b-3.pdp-kafka.mc3d5e.c6.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:443'
- '--tls-enable'
- '--log-level=debug'
- '--dynamic-listeners-disable'
- '--tls-insecure-skip-verify'
- '--proxy-request-buffer-size=32768'
- '--proxy-response-buffer-size=32768'
- '--proxy-listener-read-buffer-size=32768'
- '--proxy-listener-write-buffer-size=131072'
- '--kafka-connection-read-buffer-size=131072'
- '--kafka-connection-write-buffer-size=32768'
- '--proxy-listener-key-file=/opt/tls/tls.key'
- '--proxy-listener-cert-file=/opt/tls/tls.crt'
- '--proxy-listener-tls-enable'
ports:
- name: kafka-port3
containerPort: 443
- name: health
containerPort: 9080
volumeMounts:
- mountPath: /opt/tls
name: kafka-wildcard-cert
readOnly: true
volumes:
- name: kafka-wildcard-cert
secret:
defaultMode: 420
secretName: kafka-wildcard-cert
from kafka-proxy.
rajk0007
commented on June 3, 2024
Here are my pods and nodes
➜ kafka-proxy kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kafka-proxy-1-7845cb57cc-hgf46 1/1 Running 0 2m30s 20.10.3.191 ip-20-10-3-252.ec2.internal <none> <none>
kafka-proxy-2-f9c556868-4kvl7 1/1 Running 0 2m30s 20.10.2.84 ip-20-10-2-185.ec2.internal <none> <none>
kafka-proxy-3-cc5cb5c6b-g4r6v 1/1 Running 0 2m30s 20.10.1.210 ip-20-10-1-197.ec2.internal <none> <none>
➜ kafka-proxy kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kafka-proxy-1-7845cb57cc-hgf46 1/1 Running 0 2m47s 20.10.3.191 ip-20-10-3-252.ec2.internal <none> <none>
kafka-proxy-2-f9c556868-4kvl7 1/1 Running 0 2m47s 20.10.2.84 ip-20-10-2-185.ec2.internal <none> <none>
kafka-proxy-3-cc5cb5c6b-g4r6v 1/1 Running 0 2m47s 20.10.1.210 ip-20-10-1-197.ec2.internal <none> <none>
from kafka-proxy.
rajk0007
commented on June 3, 2024
@everesio I tried your example here. But that did not work .
from kafka-proxy.
everesio
commented on June 3, 2024
Could you provide your client configuration ?
How do you want to access the proxy , from a client running in or outside kubernetes ?
from kafka-proxy.
Related Issues (20)
- IAM auth passthrough to MSK with encryption in transit?
- MSK TLS Errors HOT 1
- TLS extract CN and lookup on LDAP for SASL
- cannot connect to aws kafka from on-prem server with SASL_SSL enabled HOT 2
- Simple use case to connect docker compose container to remote vpn kafka cluster over socks5a proxy via ssh HOT 1
- Pod startup issue after version 0.3.3-all HOT 3
- CVE-2023-37788 - github.com/elazarl/goproxy HOT 3
- AWS MSK Serverless - had error: api key -13567 is invalid HOT 4
- updating to tag 0.3.7-all from 0.3.3-all getting error auth-local-command HOT 1
- [Question] Can I attach 3 bootstrap server endpoints to a single port? HOT 1
- [Question] If my Kafka brokers are running version 2.8.1, should I be using kafka-proxy version 0.2.9? HOT 1
- "Metadata" request (ApiKey=3 and ApiVersion=5) in the Kafka Proxy is not following the protocol structure defined by Kafka protocol guide HOT 1
- [Question] is there a plan to release a Java implementation of Kafka Proxy ? HOT 1
- [Need Help] Sending Custom METADATA response through Kafka Proxy
- [Question] is there a plan to support HTTPS proxy ?
- will there be an update to resovle 7 vulnerabilitys
- tls: failed to parse private key AWS MSK HOT 6
- bad performance when executing kafka-producer-perf-test.sh HOT 3
- Can not use grepplabs/kafka-proxy ARM image as base image HOT 1
- one port mapping to 6 broker HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kafka-proxy.