CNI/IPAM plugin on Ubuntu 17.10 breaks DNS about azure-container-networking HOT 18 CLOSED

@tamilmani1989 @cpressland would #345 fix this?

from azure-container-networking.

Worse, tried on Ubuntu 16.10 and CNI/IPAM knocks the VM completely off the network.

from azure-container-networking.

@seanknox is this a self managed k8s cluster, or setup via acs-engine? If you are using acs-engine, can you please share cluster config?

Also, is it kube-dns that breaks, or default azure dns, or are you specifying dns in your Vnet config?

from azure-container-networking.

@seanknox if you are working on self manages cluster, please ensure you have ebtables installed on nodes before using cni. Otherwise, layer 2 operations fails and vm can loose connectivity.

from azure-container-networking.

@sharmasushant it's a self-managed cluster (built using terraform: http://github.com/seanknox/acstack). layer 3 and layer 4 continues to work fine: I can ping and connect to sockets on the internet. the issue with Ubuntu 17.10 is DNS resolution fails once CNI kicks in on the node.

from azure-container-networking.

Also, can confirm that conntrack and ebtables are already installed.

from azure-container-networking.

@sharadagarwal do you need any additional information to investigate?

Edit, oops, meant @sharmasushant.

from azure-container-networking.

@seanknox did you mean to ask someone else?

from azure-container-networking.

@saiyan86 can you take a look?
@seanknox can you please try v1.0.2 to unblock while we look at 1.0.3
v1.0.2 has been working well with ace-engine for some time now.

from azure-container-networking.

@sharmasushant just tried v1.0.2 and same problem. Note that acs-engine uses Ubuntu 16.04-LTS, which is older.

from azure-container-networking.

@sharmasushant I just saw this. Will take a look.

from azure-container-networking.

I don't think this is the case, as the bug refers to adding the local caching nameserver to a list of existing nameservers in resolv.conf. In my case, I'm only using the default nameserver that comes with Ubuntu 17.10 (nameserver 127.0.0.53).

from azure-container-networking.

@seanknox - Can you change nameserver to 168.63.129.16 in resolv.conf and check if its working? Also please share /var/log/azure-vnet.log and /var/log/azure-vnet-ipam.log

from azure-container-networking.

@seanknox can you please share an update if your issue is still there?

from azure-container-networking.

Closing stale issue

from azure-container-networking.

Could we reopen this? In my investigation of implementing Azure-CNI into our own cluster #344 we've run into the same issue. I worked-around it with this:

apt -y install resolvconf

cat <<EOF | tee /etc/resolvconf/resolv.conf.d/head
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolv
conf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual namese
rvers.
nameserver 168.63.129.16
EOF

systemctl restart resolvconf.service

While the above works, it still feels like a hack.

from azure-container-networking.

Turned out it was actually much easier to solve than the above. The Kubelet has an option specifically for this and we've set it as follows in the Kubelet config YAML: resolvConf: "/run/systemd/resolve/resolv.conf"

from azure-container-networking.

@cpressland #345 should fix this.

from azure-container-networking.