No Route to Host on StatefulSet Update about kubernetes-kafka HOT 6 CLOSED

What kind of k8s cluster is it you're using? Do you mean the kind of rolling update that was new to StatefulSet in 1.7?

Have you tried execing into the failing brokers to try ping/curl, or do they die too fast for that?

from kubernetes-kafka.

@solsson This is a private topology AWS cluster. And yes, I mean the rolling update mechanism introduced for StatefulSets in 1.7. That said, I was also seeing this behaviour when I was previously manually deleting pods.

I have execd into the pods, and they are unable to reach any external addresses (including intra-cluster) which is very odd to me. I also checked that other pods on the same node could still reach the network, and they could. This is also not isolated to a single machine which makes me think there is either a bug with something this Kafka image is doing or with the way in which Kubernetes is handing the associated StatefulSets.

from kubernetes-kafka.

I have no clue but I can keep guessing :) Can they ping cluster IPs, external IPs? I've seen issues with Kubernetes networking, in particular DNS resolution, on Alpine based images (mentioned in #46 and solsson/dockerfiles#5). What kind of Kafka image are you using?

from kubernetes-kafka.

Sorry, I think this might actually be related to the CNI provider I'm using (Calico). I found this chain of related issues/PRs: kubernetes/kops#2538. I'll test with the updated version of Calico and report back.

Thanks!

from kubernetes-kafka.

Got this figured out. As those issues mention, there was a race condition in the Calico policy controller that would add and then remove new pod endpoints (when the opposite was supposed to happen). I updated to calico/node:v2.6.1 and calico/cni:v1.8.3 in the calico-node daemonset and to calico/kube-policy-controller:master in calico-policy-controller deployment. Since then I've had no issues with running rolling restarts of Kafka and Zookeeper StatefulSets.

from kubernetes-kafka.

I am using weave and i got the same problem
ports are open 2888 and 3888
i am trying usin a hs service, and the same problem

from kubernetes-kafka.