Comments (8)
The reason for the this reported behaviour is that the interface attached to the bridge (azure0) is not well configured. Meaning a default setting for the hairpin mode is used. The man for bridge states the following
hairpin on or hairpin off
Controls whether traffic may be send back out of the port on which it was received.
By default, this flag is turned off and the bridge will not forward traffic back
out of the receiving port.
As a workaround one can perform the following changes to identify the right interface
Howto find out the interface for a given POD to alter the hairpin flag?
First identify the node on which the POD is running
kubectl get pod nginx-reachability-testing -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-reachability-testing 1/1 Running 0 1h 10.240.0.6 k8s-agentpool1-21713571-0
Second get the interface detail of the POD/Container
kubectl exec -it nginx-reachability-testing bash
root@nginx-reachability-testing:/# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
22: eth0@if21: <BROADCAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP qlen 1000
link/ether ce:b7:4d:b5:a8:cb brd ff:ff:ff:ff:ff:ff
inet 10.240.0.6/12 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::ccb7:4dff:feb5:a8cb/64 scope link
valid_lft forever preferred_lft forever
Note down the detail:
--> 22: eth0@if21
Third, get to the agent on which the POD/Container is running
As seen previously the network interface has the number 22 and is connected to interface 21. The reason is a veth pair which is in use.
Go to the agent and search for interface the docker container is connected to. In our case it is interface 22--> ip add |grep @if22
ip add | grep @if22
21: azvethc9e7b2c@if22: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master azure0 state UP group default qlen 1000
The interface name we get returned is the one which is connected on the bridge (azure0) This is the name we have to use to get its hairpin mode changed
Howto change the hairpin mode?
The hairpin mode for a given interface can easily be altered by performing this step
--> sudo bridge link set dev azvethc9e7b2c hairpin on
Verification
Get the external IP of the POD
kubectl get service nginx-reachability-testing
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-reachability-testing LoadBalancer 10.0.157.109 13.74.43.81 80:32594/TCP 1h
connect to your POD:
kubectl exec -it nginx-reachability-testing bash
Connect from inside the POD against the LB
root@nginx-reachability-testing:/# curl 13.74.43.81
<title>Welcome to nginx!</title> <style> body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style>Welcome to nginx!
If you see this page, the nginx web server is successfully installed and working. Further configuration is required.
For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.Thank you for using nginx.
Change the hairpin mode to off again
sudo bridge link set dev azvethc9e7b2c hairpin off
and try to connect against the LB. This will show that the connection is not established again.
So revert the modification
sudo bridge link set dev azvethc9e7b2c hairpin on
from azure-container-networking.
To allow some kind of automation I have created this simple script which can be use as a starting base: https://gist.github.com/malachma/8603d1d6daedceb4320d20b70527f0ac
It uses the kubernetes python-client. So install it first. Get also sure that the nodes are able to establish
ssh connections via pub-key only. To listen for a ceratin POD which get added to the cluster replace the variable "nameOfPod"
from azure-container-networking.
@carlpett Is it still happening for you? Can you please share your cluster config. We can try to repro this.
from azure-container-networking.
@sharmasushant Yes, still happens. After some more investigation, it does not only happen for ingresses, but all services with type LoadBalancer. Here is a repro case:
apiVersion: v1
kind: Pod
metadata:
name: nginx-reachability-testing
labels:
app: nginx-reachability-testing
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx-reachability-testing
spec:
ports:
- port: 80
name: http
type: LoadBalancer
selector:
app: nginx-reachability-testing
The public IP is reachable from anywhere, except from within the pod that is backing the service.
I reported this while waiting for a response on an Azure support ticket, which has now gotten started, so there is an ongoing case there (118011617470683)
from azure-container-networking.
Found the following issue reports which are related to the hairpin mode
kubernetes/kubernetes#53269
containernetworking/cni#476
The main context is this one
kubernetes/kubernetes#45790
from azure-container-networking.
We have the exact same use case and we experience the same issue. The workaround from @malachma worked for us.
from azure-container-networking.
Any updates on this issue being fixed natively in the CNI plugin? While we could use the script above and automate it, this seems like something the plugin should do natively and is probably a simple change. I'm happy to look at creating a PR if needed.
Thanks.
from azure-container-networking.
This issue is fixed #248
from azure-container-networking.
Related Issues (20)
- Test
- question iptables vs firewalld / non(AKS) Kubernetes HOT 3
- NPM constantly output telemetry errors HOT 2
- Critical CVEs HOT 2
- Add support for port ranges in NetworkPolicy with "endPort" field HOT 3
- On k8s 1.28 (public preview), Windows NPM crash HOT 1
- CNI Installer migration
- Change flow of using Azure Virtual Network Subnet in the Azure Kubernetes Service HOT 4
- Invalid CIDR address when creating Docker network using CNM plugin HOT 1
- Authenticate with Managed Identity from a Docker container connected to an Azure VNet using CNM plugin
- Azure CNI dynamic ip allocation batch size does not starts with 16 HOT 1
- Remove assests from release which are not supported
- Remove CNMS Code HOT 1
- Windows LoopbackDSR policy missing for IPv6 HOT 9
- Remove CNM Code HOT 9
- Unable to send packets from pods on certain nodes to certain pods in the rest of the cluster. HOT 7
- Failed to initialize key-value store of network plugin: error Acquiring store lock: timed out locking store HOT 2
- Telemetry leaks sockets on error HOT 4
- 10-azure.conflist missing from AKS AzureLinux/CBLMariner node image HOT 1
- AKS: edge case in Azure NPM policy enforcement HOT 9
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from azure-container-networking.