What happened? Network policy Egress rules are not behaving as ex

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clip

Forgot the logs <div class="snippet-clipboard-content notranslate position-relativ

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

Are you adding --service-cluster-ip-range (see <a hre

NetworkPolicy's Egress is not working,about cloudnativelabs/kube-router

Comments (12)

aauren commented on June 27, 2024

@vladimirtiukhtin

Can you share the output to the following commands?

kubectl get namespace kube-system -o yaml

kubectl get namespace test -o yaml

kubectl get namespaces -l kubernetes.io/metadata.name=kube-system

kubectl get pods -n kube-system -l app.kubernetes.io/name=coredns

from kube-router.

vladimirtiukhtin commented on June 27, 2024

apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: "2023-03-20T22:03:13Z"
  labels:
    kubernetes.io/metadata.name: kube-system
  name: kube-system
  resourceVersion: "12"
  uid: 65127612-c087-42aa-8413-a63087946f5f
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: "2023-03-23T15:09:12Z"
  labels:
    kubernetes.io/metadata.name: test
  name: test
  resourceVersion: "462381"
  uid: df73085a-eff1-468e-8e9b-c0be0e64837d
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

# kubectl get namespaces -l kubernetes.io/metadata.name=kube-system
NAME          STATUS   AGE
kube-system   Active   364d

# kubectl get pods -n kube-system -l app.kubernetes.io/name=coredns
NAME                       READY   STATUS    RESTARTS   AGE
coredns-6d9f7b5ccb-4d5h7   1/1     Running   0          215d
coredns-6d9f7b5ccb-mv9lm   1/1     Running   0          96d

from kube-router.

aauren commented on June 27, 2024

Thanks for the output information.

I'm having a hard time reproducing this issue. Are you able to give the following:

The Kubernetes manifests you're using
Step by step instructions for how you're testing this
Increase kube-router's logging level to 2 and a provide a tgz of the log output during the reproduction

from kube-router.

vladimirtiukhtin commented on June 27, 2024

So I made some more tests this evening and here are the results:

Apply the manifest below:

apiVersion: v1
kind: Namespace
metadata:
  labels:
    kubernetes.io/metadata.name: test
  name: test
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all
  namespace: test
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: test
  namespace: test
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: test
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: test
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          app.kubernetes.io/name: coredns  
    ports:
      - port: 53
        protocol: UDP
      - port: 53
        protocol: TCP
---
apiVersion: v1
kind: Pod
metadata:
  name: server
  namespace: test
  labels:
    app: server
spec:
  containers:
  - command:
    - nc
    args:
    - -lkv
    - -p
    - "8080"
    - -e
    - "true"
    image: alpine
    imagePullPolicy: Always
    name: nc
---
apiVersion: v1
kind: Pod
metadata:
  name: client
  namespace: test
spec:
  containers:
  - command:
    - tail
    args:
    - -f
    - /dev/null
    image: alpine
    imagePullPolicy: Always
    name: nc
---
apiVersion: v1
kind: Service
metadata:
  name: server
  namespace: test
spec:
  ports:
  - name: 80-8080
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: server
  sessionAffinity: None
  type: ClusterIP

Run kubectl -n test get pods -o wide to grab the IP of the server. Do kubectl -n test get svc to grab the IP of the svc
Log in to the client kubectl -n test exec -ti client -- sh
Run nc ${server_ip}:8080 -w 5 -v. You will see 10.0.10.105:8080 (10.0.10.105:8080) open. Server pod logs connection accordingly, so all good
Run nc ${svc_ip}:80 -w 5 -v, nc exits immediately without logging anything. But you can see with tcpdump that it gets reject from kernel serving server pod
Try nslookup google.com. You will get nslookup: write to '172.16.0.53': Connection refused

from kube-router.

vladimirtiukhtin commented on June 27, 2024

Forgot the logs

Apr 12 21:14:22 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:22.014629 3280670 service_endpoints_sync.go:454] Setting up policy routing required for Direct Server Return functionality.
Apr 12 21:14:22 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:22.014681 3280670 policy.go:82] Attained ipset mutex lock, continuing...
Apr 12 21:14:22 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:22.016295 3280670 service_endpoints_sync.go:459] Custom routing table kube-router-dsr required for Direct Server Return is setup as expected.
Apr 12 21:14:22 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:22.016311 3280670 service_endpoints_sync.go:462] Setting up custom route table required to add routes for external IP's.
Apr 12 21:14:22 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:22.019084 3280670 service_endpoints_sync.go:470] Custom routing table required for Direct Server Return (external_ip) is setup as expected.
Apr 12 21:14:22 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:22.019103 3280670 service_endpoints_sync.go:86] IPVS servers and services are synced to desired state
Apr 12 21:14:22 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:22.019110 3280670 service_endpoints_sync.go:31] sync ipvs services took 320.804882ms
Apr 12 21:14:22 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:22.170664 3280670 policy.go:142] Iptables chains in the filter table are synchronized with the network policies.
Apr 12 21:14:22 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:22.170828 3280670 policy.go:85] Returned ipset mutex lock
Apr 12 21:14:22 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:22.170903 3280670 policy.go:77] Syncing network policy chains took 335.422382ms
Apr 12 21:14:22 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:22.183439 3280670 network_policy_controller.go:551] Attempting to attain ipset mutex lock
Apr 12 21:14:22 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:22.183463 3280670 network_policy_controller.go:553] Attained ipset mutex lock, continuing...
Apr 12 21:14:22 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:22.185137 3280670 network_policy_controller.go:556] Returned ipset mutex lock
Apr 12 21:14:22 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:22.185154 3280670 network_policy_controller.go:233] sync iptables took 458.052327ms
Apr 12 21:14:27 general-74aab0e3d49bdfda kube-router[3280670]: time="2024-04-12T21:14:27Z" level=info msg="skipped asn negotiation: peer-as: 64512, peer-type: internal" Key=10.103.0.9 State=BGP_FSM_OPENSENT Topic=Peer
Apr 12 21:14:27 general-74aab0e3d49bdfda kube-router[3280670]: time="2024-04-12T21:14:27Z" level=info msg="Peer Up" Key=10.103.0.9 State=BGP_FSM_OPENCONFIRM Topic=Peer
Apr 12 21:14:27 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:27.684476 3280670 network_routes_controller.go:451] Processing bgp route advertisement from peer: 10.103.0.9
Apr 12 21:14:27 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:27.685159 3280670 network_routes_controller.go:545] injectRoute Path Looks Like: nlri:<type_url:"type.googleapis.com/gobgpapi.IPAddressPrefix" value:"\010\030\022\01010.0.5.0" > pattrs:<type_url:"type.googleapis.com/gobgpapi.OriginAttribute" > pattrs:<type_url:"type.googleapis.com/gobgpapi.AsPathAttribute" > pattrs:<type_url:"type.googleapis.com/gobgpapi.NextHopAttribute" value:"\n\n10.103.0.9" > pattrs:<type_url:"type.googleapis.com/gobgpapi.LocalPrefAttribute" value:"\010d" > age:<seconds:1712956467 > validation:<> family:<afi:AFI_IP safi:SAFI_UNICAST > source_asn:64512 source_id:"10.103.0.9" neighbor_ip:"10.103.0.9" local_identifier:1
Apr 12 21:14:27 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:27.685195 3280670 network_routes_controller.go:661] Cleaning up old routes for 10.0.5.0/24 if there are any
Apr 12 21:14:27 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:27.685368 3280670 network_routes_controller.go:666] Cleaning up any lingering tunnel interfaces named: tun-1010309
Apr 12 21:14:28 general-74aab0e3d49bdfda kube-router[3280670]: time="2024-04-12T21:14:28Z" level=info msg="skipped asn negotiation: peer-as: 64512, peer-type: internal" Key=10.103.0.2 State=BGP_FSM_OPENSENT Topic=Peer
Apr 12 21:14:28 general-74aab0e3d49bdfda kube-router[3280670]: time="2024-04-12T21:14:28Z" level=info msg="Peer Up" Key=10.103.0.2 State=BGP_FSM_OPENCONFIRM Topic=Peer
Apr 12 21:14:28 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:28.691787 3280670 network_routes_controller.go:451] Processing bgp route advertisement from peer: 10.103.0.2
Apr 12 21:14:28 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:28.691899 3280670 network_routes_controller.go:545] injectRoute Path Looks Like: nlri:<type_url:"type.googleapis.com/gobgpapi.IPAddressPrefix" value:"\010\030\022\01010.0.2.0" > pattrs:<type_url:"type.googleapis.com/gobgpapi.OriginAttribute" > pattrs:<type_url:"type.googleapis.com/gobgpapi.AsPathAttribute" > pattrs:<type_url:"type.googleapis.com/gobgpapi.NextHopAttribute" value:"\n\n10.103.0.2" > pattrs:<type_url:"type.googleapis.com/gobgpapi.LocalPrefAttribute" value:"\010d" > age:<seconds:1712956468 > validation:<> family:<afi:AFI_IP safi:SAFI_UNICAST > source_asn:64512 source_id:"10.103.0.2" neighbor_ip:"10.103.0.2" local_identifier:1
Apr 12 21:14:28 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:28.691948 3280670 network_routes_controller.go:661] Cleaning up old routes for 10.0.2.0/24 if there are any
Apr 12 21:14:28 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:28.692193 3280670 network_routes_controller.go:666] Cleaning up any lingering tunnel interfaces named: tun-1010302
Apr 12 21:14:30 general-74aab0e3d49bdfda kube-router[3280670]: time="2024-04-12T21:14:30Z" level=info msg="skipped asn negotiation: peer-as: 64512, peer-type: internal" Key=10.103.0.14 State=BGP_FSM_OPENSENT Topic=Peer
Apr 12 21:14:30 general-74aab0e3d49bdfda kube-router[3280670]: time="2024-04-12T21:14:30Z" level=info msg="Peer Up" Key=10.103.0.14 State=BGP_FSM_OPENCONFIRM Topic=Peer
Apr 12 21:14:30 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:30.684370 3280670 network_routes_controller.go:451] Processing bgp route advertisement from peer: 10.103.0.14
Apr 12 21:14:30 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:30.684444 3280670 network_routes_controller.go:545] injectRoute Path Looks Like: nlri:<type_url:"type.googleapis.com/gobgpapi.IPAddressPrefix" value:"\010\030\022\t10.0.12.0" > pattrs:<type_url:"type.googleapis.com/gobgpapi.OriginAttribute" > pattrs:<type_url:"type.googleapis.com/gobgpapi.AsPathAttribute" > pattrs:<type_url:"type.googleapis.com/gobgpapi.NextHopAttribute" value:"\n\01310.103.0.14" > pattrs:<type_url:"type.googleapis.com/gobgpapi.LocalPrefAttribute" value:"\010d" > age:<seconds:1712956470 > validation:<> family:<afi:AFI_IP safi:SAFI_UNICAST > source_asn:64512 source_id:"10.103.0.14" neighbor_ip:"10.103.0.14" local_identifier:1
Apr 12 21:14:30 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:30.684478 3280670 network_routes_controller.go:661] Cleaning up old routes for 10.0.12.0/24 if there are any
Apr 12 21:14:30 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:14:30.684650 3280670 network_routes_controller.go:666] Cleaning up any lingering tunnel interfaces named: tun-10103014
Apr 12 21:15:02 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:02.873825 3280670 namespace.go:40] Received update for namespace: test
Apr 12 21:15:02 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:02.873865 3280670 network_policy_controller.go:236] Starting sync of iptables with version: 1712956502873860110
Apr 12 21:15:02 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:02.904340 3280670 policy.go:43] Received update for network policy: test/deny-all
Apr 12 21:15:02 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:02.906885 3280670 policy.go:80] Attempting to attain ipset mutex lock
Apr 12 21:15:02 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:02.906907 3280670 policy.go:82] Attained ipset mutex lock, continuing...
Apr 12 21:15:02 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:02.925576 3280670 policy.go:43] Received update for network policy: test/test
Apr 12 21:15:02 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:02.925604 3280670 network_policy_controller.go:213] Full sync request queue was full, skipping...
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.034737 3280670 policy.go:142] Iptables chains in the filter table are synchronized with the network policies.
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.034764 3280670 policy.go:85] Returned ipset mutex lock
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.034773 3280670 policy.go:77] Syncing network policy chains took 127.889761ms
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.039541 3280670 pod.go:53] Received update to pod: test/server
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.039563 3280670 network_policy_controller.go:213] Full sync request queue was full, skipping...
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.045194 3280670 network_services_controller.go:962] Received update to service: test/server from watch API
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.045229 3280670 network_services_controller.go:1106] Skipping service name:elasticsearch-0 namespace:some-other-namespace as there is no cluster IP
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.045235 3280670 ecmp_vip.go:184] Received new service: test/server from watch API
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.045247 3280670 network_services_controller.go:1106] Skipping service name:elasticsearch-2 namespace:some-other-namespace as there is no cluster IP
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.045405 3280670 network_services_controller.go:1106] Skipping service name:elasticsearch-1 namespace:some-other-namespace as there is no cluster IP
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.045414 3280670 network_services_controller.go:1106] Skipping service name:redis-replica namespace:some-other-namespace as there is no cluster IP
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.045420 3280670 network_services_controller.go:1106] Skipping service name:redis-replica namespace:some-other-other-namespace as there is no cluster IP
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.045430 3280670 network_services_controller.go:1106] Skipping service name:elasticsearch-4 namespace:some-other-namespace as there is no cluster IP
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.045438 3280670 network_services_controller.go:1106] Skipping service name:elasticsearch namespace:some-other-other-namespace as there is no cluster IP
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.045444 3280670 network_services_controller.go:1106] Skipping service name:elasticsearch namespace:some-other-namespace as there is no cluster IP
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.045449 3280670 network_services_controller.go:1106] Skipping service name:mariadb namespace:some-other-namespace as there is no cluster IP
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.045455 3280670 network_services_controller.go:1106] Skipping service name:elasticsearch-3 namespace:some-other-namespace as there is no cluster IP
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.045473 3280670 network_services_controller.go:1106] Skipping service name:mariadb namespace:some-other-other-namespace as there is no cluster IP
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.045523 3280670 network_services_controller.go:986] Syncing IPVS services sync on update to service: test/server
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.045539 3280670 network_services_controller.go:454] Performing requested sync of ipvs services
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.047483 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.225.14:80 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.047564 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.168:9090 (Weight: 1) already exists in the ipvs service tcp:172.16.225.14:80 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.048079 3280670 network_policy_controller.go:551] Attempting to attain ipset mutex lock
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.048093 3280670 network_policy_controller.go:553] Attained ipset mutex lock, continuing...
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.049029 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.162.153:6379 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.049086 3280670 network_services_controller.go:1805] ipvs destination 10.0.12.195:6379 (Weight: 1) already exists in the ipvs service tcp:172.16.162.153:6379 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.049593 3280670 network_policy_controller.go:556] Returned ipset mutex lock
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.049607 3280670 network_policy_controller.go:233] sync iptables took 175.74433ms
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.049623 3280670 network_policy_controller.go:236] Starting sync of iptables with version: 1712956503049619258
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.050302 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.186.221:80 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.050362 3280670 network_services_controller.go:1805] ipvs destination 10.0.12.83:8080 (Weight: 1) already exists in the ipvs service tcp:172.16.186.221:80 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.050396 3280670 network_services_controller.go:1805] ipvs destination 10.0.10.72:8080 (Weight: 1) already exists in the ipvs service tcp:172.16.186.221:80 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.051590 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.0.1:443 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.051656 3280670 network_services_controller.go:1805] ipvs destination 10.103.0.2:6443 (Weight: 1) already exists in the ipvs service tcp:172.16.0.1:443 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.052941 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.97.9:80 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.052992 3280670 network_services_controller.go:1805] ipvs destination 10.0.12.194:8765 (Weight: 1) already exists in the ipvs service tcp:172.16.97.9:80 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.054401 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.207.197:80 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.055895 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.194.199:443 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.055945 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.123:10250 (Weight: 1) already exists in the ipvs service tcp:172.16.194.199:443 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.057396 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.179.173:80 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.057443 3280670 network_services_controller.go:1805] ipvs destination 10.0.12.94:8001 (Weight: 1) already exists in the ipvs service tcp:172.16.179.173:80 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.058706 3280670 network_services_controller.go:1723] Successfully added service: tcp:172.16.54.104:80 (Flags: )
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.059915 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.214.142:80 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.061117 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.47.177:80 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.061172 3280670 network_services_controller.go:1805] ipvs destination 10.0.10.100:8080 (Weight: 1) already exists in the ipvs service tcp:172.16.47.177:80 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.062705 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.108.25:80 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.062757 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.62:8765 (Weight: 1) already exists in the ipvs service tcp:172.16.108.25:80 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.063948 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.225.132:6379 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.064015 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.50:6379 (Weight: 1) already exists in the ipvs service tcp:172.16.225.132:6379 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.064640 3280670 pod.go:53] Received update to pod: test/client
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.065439 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.41.221:9189 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.065524 3280670 network_services_controller.go:1805] ipvs destination 10.0.10.4:9189 (Weight: 1) already exists in the ipvs service tcp:172.16.41.221:9189 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.065575 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.2:9189 (Weight: 1) already exists in the ipvs service tcp:172.16.41.221:9189 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.065625 3280670 network_services_controller.go:1805] ipvs destination 10.0.12.3:9189 (Weight: 1) already exists in the ipvs service tcp:172.16.41.221:9189 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.065673 3280670 network_services_controller.go:1805] ipvs destination 10.0.2.11:9189 (Weight: 1) already exists in the ipvs service tcp:172.16.41.221:9189 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.067102 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.174.230:8080 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.067148 3280670 network_services_controller.go:1805] ipvs destination 10.0.12.196:8080 (Weight: 1) already exists in the ipvs service tcp:172.16.174.230:8080 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.068402 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.206.215:80 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.068517 3280670 network_services_controller.go:1805] ipvs destination 10.0.10.41:8000 (Weight: 1) already exists in the ipvs service tcp:172.16.206.215:80 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.069864 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.0.53:53 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.069918 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.32:53 (Weight: 1) already exists in the ipvs service tcp:172.16.0.53:53 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.069952 3280670 network_services_controller.go:1805] ipvs destination 10.0.2.15:53 (Weight: 1) already exists in the ipvs service tcp:172.16.0.53:53 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.073427 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.35.84:80 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.075267 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.11.196:8080 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.075343 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.12:8080 (Weight: 1) already exists in the ipvs service tcp:172.16.11.196:8080 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.077806 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.238.109:9402 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.077962 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.6:9402 (Weight: 1) already exists in the ipvs service tcp:172.16.238.109:9402 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.080772 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.18.187:80 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.080848 3280670 network_services_controller.go:1805] ipvs destination 10.0.10.142:8080 (Weight: 1) already exists in the ipvs service tcp:172.16.18.187:80 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.080895 3280670 network_services_controller.go:1805] ipvs destination 10.0.12.63:8080 (Weight: 1) already exists in the ipvs service tcp:172.16.18.187:80 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.082539 3280670 ecmp_vip.go:325] Received update to endpoint: test/server from watch API
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.082563 3280670 ecmp_vip.go:184] Updating service test/server triggered by endpoint update event
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.082632 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.168.244:8080 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.082678 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.47:8080 (Weight: 1) already exists in the ipvs service tcp:172.16.168.244:8080 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.083981 3280670 network_services_controller.go:1703] ipvs service udp:172.16.0.53:53 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.084032 3280670 network_services_controller.go:1805] ipvs destination 10.0.2.15:53 (Weight: 1) already exists in the ipvs service udp:172.16.0.53:53 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.084057 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.32:53 (Weight: 1) already exists in the ipvs service udp:172.16.0.53:53 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.084578 3280670 policy.go:80] Attempting to attain ipset mutex lock
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.084595 3280670 policy.go:82] Attained ipset mutex lock, continuing...
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.085248 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.148.198:443 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.085300 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.10:6443 (Weight: 1) already exists in the ipvs service tcp:172.16.148.198:443 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.086509 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.98.50:9093 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.086564 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.163:9093 (Weight: 1) already exists in the ipvs service tcp:172.16.98.50:9093 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.087942 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.11.196:8081 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.087987 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.12:8081 (Weight: 1) already exists in the ipvs service tcp:172.16.11.196:8081 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.089177 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.220.60:9189 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.089228 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.27:9189 (Weight: 1) already exists in the ipvs service tcp:172.16.220.60:9189 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.090373 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.206.215:443 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.090424 3280670 network_services_controller.go:1805] ipvs destination 10.0.10.41:8443 (Weight: 1) already exists in the ipvs service tcp:172.16.206.215:443 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.090564 3280670 network_services_controller.go:1703] ipvs service tcp:10.103.0.10:30386 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.090598 3280670 network_services_controller.go:1805] ipvs destination 10.0.10.41:8443 (Weight: 1) already exists in the ipvs service tcp:10.103.0.10:30386 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.090609 3280670 network_services_controller.go:1703] ipvs service tcp:10.103.0.10:31529 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.090634 3280670 network_services_controller.go:1805] ipvs destination 10.0.10.41:8000 (Weight: 1) already exists in the ipvs service tcp:10.103.0.10:31529 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.091864 3280670 network_services_controller.go:1703] ipvs service tcp:142.132.245.229:443 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.091879 3280670 service_endpoints_sync.go:335] no FW mark found for service, nothing to cleanup
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.091915 3280670 network_services_controller.go:1805] ipvs destination 10.0.10.41:8443 (Weight: 1) already exists in the ipvs service tcp:142.132.245.229:443 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: E0412 21:15:03.093193 3280670 network_services_controller.go:195] Failed to replace route to service VIP 2a01:4f8:c011:eef::1 configured on kube-dummy-if. Error: exit status 1, Output: Error: inet6 address is expected rather than "10.103.0.10".
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.093259 3280670 network_services_controller.go:1723] Successfully added service: tcp:2a01:4f8:c011:eef::1:443 (Flags: )
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.093274 3280670 service_endpoints_sync.go:335] no FW mark found for service, nothing to cleanup
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.093310 3280670 network_services_controller.go:1794] Successfully added destination 10.0.10.41:8443 (Weight: 1) to the service tcp:2a01:4f8:c011:eef::1:443 (Flags: )
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.094786 3280670 network_services_controller.go:1703] ipvs service tcp:142.132.245.229:80 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.094802 3280670 service_endpoints_sync.go:335] no FW mark found for service, nothing to cleanup
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.094870 3280670 network_services_controller.go:1805] ipvs destination 10.0.10.41:8000 (Weight: 1) already exists in the ipvs service tcp:142.132.245.229:80 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: E0412 21:15:03.096434 3280670 network_services_controller.go:195] Failed to replace route to service VIP 2a01:4f8:c011:eef::1 configured on kube-dummy-if. Error: exit status 1, Output: Error: inet6 address is expected rather than "10.103.0.10".
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.096483 3280670 network_services_controller.go:1723] Successfully added service: tcp:2a01:4f8:c011:eef::1:80 (Flags: )
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.096492 3280670 service_endpoints_sync.go:335] no FW mark found for service, nothing to cleanup
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.096518 3280670 network_services_controller.go:1794] Successfully added destination 10.0.10.41:8000 (Weight: 1) to the service tcp:2a01:4f8:c011:eef::1:80 (Flags: )
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.096526 3280670 service_endpoints_sync.go:477] Cleaning up if any, old service IPs on dummy interface
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.096793 3280670 service_endpoints_sync.go:521] Cleaning up if any, old ipvs service and servers which are no longer needed
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.096916 3280670 service_endpoints_sync.go:564] Found a IPVS service tcp:42.1.4.248:443 (Flags: [hashed entry]) which is no longer needed so cleaning up
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.097239 3280670 service_endpoints_sync.go:564] Found a IPVS service tcp:42.1.4.248:80 (Flags: [hashed entry]) which is no longer needed so cleaning up
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.097397 3280670 network_services_controller.go:751] Attempting to attain ipset mutex lock
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.430673 3280670 policy.go:142] Iptables chains in the filter table are synchronized with the network policies.
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.430695 3280670 policy.go:85] Returned ipset mutex lock
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.430705 3280670 policy.go:77] Syncing network policy chains took 346.126531ms
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.430782 3280670 network_services_controller.go:753] Attained ipset mutex lock, continuing...
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.532307 3280670 network_policy_controller.go:551] Attempting to attain ipset mutex lock
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.630529 3280670 network_services_controller.go:756] Returned ipset mutex lock
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.630561 3280670 service_endpoints_sync.go:454] Setting up policy routing required for Direct Server Return functionality.
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.630605 3280670 network_policy_controller.go:553] Attained ipset mutex lock, continuing...
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.632044 3280670 service_endpoints_sync.go:459] Custom routing table kube-router-dsr required for Direct Server Return is setup as expected.
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.632069 3280670 service_endpoints_sync.go:462] Setting up custom route table required to add routes for external IP's.
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.632299 3280670 network_policy_controller.go:556] Returned ipset mutex lock
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.632314 3280670 network_policy_controller.go:233] sync iptables took 582.691865ms
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.632332 3280670 network_policy_controller.go:236] Starting sync of iptables with version: 1712956503632327333
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.634742 3280670 service_endpoints_sync.go:470] Custom routing table required for Direct Server Return (external_ip) is setup as expected.
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.634799 3280670 service_endpoints_sync.go:86] IPVS servers and services are synced to desired state
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.634815 3280670 service_endpoints_sync.go:31] sync ipvs services took 589.268632ms
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.634830 3280670 network_services_controller.go:1383] No hairpin-mode enabled services found -- no hairpin rules created
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.637608 3280670 network_services_controller.go:914] Received update to endpoint: test/server from watch API
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.637633 3280670 network_services_controller.go:1106] Skipping service name:redis-replica namespace:some-other-namespace as there is no cluster IP
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.637639 3280670 network_services_controller.go:1106] Skipping service name:redis-replica namespace:some-other-other-namespace as there is no cluster IP
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.637659 3280670 network_services_controller.go:1106] Skipping service name:elasticsearch-4 namespace:some-other-namespace as there is no cluster IP
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.637667 3280670 network_services_controller.go:1106] Skipping service name:elasticsearch namespace:some-other-other-namespace as there is no cluster IP
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.637674 3280670 network_services_controller.go:1106] Skipping service name:elasticsearch namespace:some-other-namespace as there is no cluster IP
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.637681 3280670 network_services_controller.go:1106] Skipping service name:elasticsearch-3 namespace:some-other-namespace as there is no cluster IP
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.637691 3280670 network_services_controller.go:1106] Skipping service name:mariadb namespace:some-other-namespace as there is no cluster IP
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.637695 3280670 network_services_controller.go:1106] Skipping service name:mariadb namespace:some-other-other-namespace as there is no cluster IP
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.637702 3280670 network_services_controller.go:1106] Skipping service name:elasticsearch-0 namespace:some-other-namespace as there is no cluster IP
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.637711 3280670 network_services_controller.go:1106] Skipping service name:elasticsearch-2 namespace:some-other-namespace as there is no cluster IP
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.637718 3280670 network_services_controller.go:1106] Skipping service name:elasticsearch-1 namespace:some-other-namespace as there is no cluster IP
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.637770 3280670 network_services_controller.go:951] Skipping IPVS services sync on endpoint: test/server update as nothing changed
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.659294 3280670 policy.go:80] Attempting to attain ipset mutex lock
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.659311 3280670 policy.go:82] Attained ipset mutex lock, continuing...
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.902612 3280670 policy.go:142] Iptables chains in the filter table are synchronized with the network policies.
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.902638 3280670 policy.go:85] Returned ipset mutex lock
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.902646 3280670 policy.go:77] Syncing network policy chains took 243.355181ms
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.915357 3280670 network_policy_controller.go:551] Attempting to attain ipset mutex lock
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.915375 3280670 network_policy_controller.go:553] Attained ipset mutex lock, continuing...
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.916859 3280670 network_policy_controller.go:556] Returned ipset mutex lock
Apr 12 21:15:03 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:03.916876 3280670 network_policy_controller.go:233] sync iptables took 284.545477ms
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.806408 3280670 pod.go:53] Received update to pod: test/server
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.806437 3280670 network_policy_controller.go:236] Starting sync of iptables with version: 1712956504806431585
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.819606 3280670 network_services_controller.go:914] Received update to endpoint: test/server from watch API
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.819622 3280670 ecmp_vip.go:325] Received update to endpoint: test/server from watch API
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.819641 3280670 ecmp_vip.go:184] Updating service test/server triggered by endpoint update event
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.819650 3280670 network_services_controller.go:1106] Skipping service name:elasticsearch namespace:some-other-namespace as there is no cluster IP
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.819674 3280670 network_services_controller.go:1106] Skipping service name:mariadb namespace:some-other-namespace as there is no cluster IP
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.819686 3280670 network_services_controller.go:1106] Skipping service name:elasticsearch-3 namespace:some-other-namespace as there is no cluster IP
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.819694 3280670 network_services_controller.go:1106] Skipping service name:mariadb namespace:some-other-other-namespace as there is no cluster IP
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.819718 3280670 network_services_controller.go:1106] Skipping service name:elasticsearch-0 namespace:some-other-namespace as there is no cluster IP
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.819731 3280670 network_services_controller.go:1106] Skipping service name:elasticsearch-1 namespace:some-other-namespace as there is no cluster IP
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.819740 3280670 network_services_controller.go:1106] Skipping service name:elasticsearch-2 namespace:some-other-namespace as there is no cluster IP
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.819752 3280670 network_services_controller.go:1106] Skipping service name:redis-replica namespace:some-other-namespace as there is no cluster IP
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.819759 3280670 network_services_controller.go:1106] Skipping service name:redis-replica namespace:some-other-other-namespace as there is no cluster IP
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.819771 3280670 network_services_controller.go:1106] Skipping service name:elasticsearch namespace:some-other-other-namespace as there is no cluster IP
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.819779 3280670 network_services_controller.go:1106] Skipping service name:elasticsearch-4 namespace:some-other-namespace as there is no cluster IP
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.819849 3280670 network_services_controller.go:948] Syncing IPVS services sync for update to endpoint: test/server
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.819886 3280670 network_services_controller.go:454] Performing requested sync of ipvs services
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.821580 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.206.215:80 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.821662 3280670 network_services_controller.go:1805] ipvs destination 10.0.10.41:8000 (Weight: 1) already exists in the ipvs service tcp:172.16.206.215:80 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.822949 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.41.221:9189 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.823004 3280670 network_services_controller.go:1805] ipvs destination 10.0.2.11:9189 (Weight: 1) already exists in the ipvs service tcp:172.16.41.221:9189 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.823031 3280670 network_services_controller.go:1805] ipvs destination 10.0.10.4:9189 (Weight: 1) already exists in the ipvs service tcp:172.16.41.221:9189 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.823055 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.2:9189 (Weight: 1) already exists in the ipvs service tcp:172.16.41.221:9189 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.823081 3280670 network_services_controller.go:1805] ipvs destination 10.0.12.3:9189 (Weight: 1) already exists in the ipvs service tcp:172.16.41.221:9189 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.824315 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.54.104:80 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.824362 3280670 network_services_controller.go:1794] Successfully added destination 10.0.10.107:8080 (Weight: 1) to the service tcp:172.16.54.104:80 (Flags: [hashed entry])
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.825820 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.108.25:80 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.825907 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.62:8765 (Weight: 1) already exists in the ipvs service tcp:172.16.108.25:80 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.827188 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.206.215:443 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.827240 3280670 network_services_controller.go:1805] ipvs destination 10.0.10.41:8443 (Weight: 1) already exists in the ipvs service tcp:172.16.206.215:443 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.828458 3280670 network_services_controller.go:1703] ipvs service udp:172.16.0.53:53 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.828524 3280670 network_services_controller.go:1805] ipvs destination 10.0.2.15:53 (Weight: 1) already exists in the ipvs service udp:172.16.0.53:53 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.828555 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.32:53 (Weight: 1) already exists in the ipvs service udp:172.16.0.53:53 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.830132 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.0.1:443 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.830182 3280670 network_services_controller.go:1805] ipvs destination 10.103.0.2:6443 (Weight: 1) already exists in the ipvs service tcp:172.16.0.1:443 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.831658 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.35.84:80 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.833005 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.207.197:80 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.834492 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.11.196:8081 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.834584 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.12:8081 (Weight: 1) already exists in the ipvs service tcp:172.16.11.196:8081 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.836373 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.179.173:80 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.836423 3280670 network_services_controller.go:1805] ipvs destination 10.0.12.94:8001 (Weight: 1) already exists in the ipvs service tcp:172.16.179.173:80 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.837747 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.225.132:6379 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.837806 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.50:6379 (Weight: 1) already exists in the ipvs service tcp:172.16.225.132:6379 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.838883 3280670 policy.go:80] Attempting to attain ipset mutex lock
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.838901 3280670 policy.go:82] Attained ipset mutex lock, continuing...
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.839130 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.225.14:80 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.839202 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.168:9090 (Weight: 1) already exists in the ipvs service tcp:172.16.225.14:80 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.840448 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.186.221:80 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.840524 3280670 network_services_controller.go:1805] ipvs destination 10.0.10.72:8080 (Weight: 1) already exists in the ipvs service tcp:172.16.186.221:80 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.840569 3280670 network_services_controller.go:1805] ipvs destination 10.0.12.83:8080 (Weight: 1) already exists in the ipvs service tcp:172.16.186.221:80 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.842036 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.0.53:53 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.842108 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.32:53 (Weight: 1) already exists in the ipvs service tcp:172.16.0.53:53 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.842147 3280670 network_services_controller.go:1805] ipvs destination 10.0.2.15:53 (Weight: 1) already exists in the ipvs service tcp:172.16.0.53:53 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.843363 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.238.109:9402 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.843412 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.6:9402 (Weight: 1) already exists in the ipvs service tcp:172.16.238.109:9402 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.845195 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.220.60:9189 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.845246 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.27:9189 (Weight: 1) already exists in the ipvs service tcp:172.16.220.60:9189 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.846629 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.194.199:443 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.846678 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.123:10250 (Weight: 1) already exists in the ipvs service tcp:172.16.194.199:443 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.848447 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.162.153:6379 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.848498 3280670 network_services_controller.go:1805] ipvs destination 10.0.12.195:6379 (Weight: 1) already exists in the ipvs service tcp:172.16.162.153:6379 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.849747 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.168.244:8080 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.849858 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.47:8080 (Weight: 1) already exists in the ipvs service tcp:172.16.168.244:8080 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.851336 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.47.177:80 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.851413 3280670 network_services_controller.go:1805] ipvs destination 10.0.10.100:8080 (Weight: 1) already exists in the ipvs service tcp:172.16.47.177:80 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.852653 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.97.9:80 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.852736 3280670 network_services_controller.go:1805] ipvs destination 10.0.12.194:8765 (Weight: 1) already exists in the ipvs service tcp:172.16.97.9:80 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.853957 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.174.230:8080 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.854010 3280670 network_services_controller.go:1805] ipvs destination 10.0.12.196:8080 (Weight: 1) already exists in the ipvs service tcp:172.16.174.230:8080 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.855196 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.11.196:8080 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.855243 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.12:8080 (Weight: 1) already exists in the ipvs service tcp:172.16.11.196:8080 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.857067 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.98.50:9093 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.857125 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.163:9093 (Weight: 1) already exists in the ipvs service tcp:172.16.98.50:9093 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.858528 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.148.198:443 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.858589 3280670 network_services_controller.go:1805] ipvs destination 10.0.5.10:6443 (Weight: 1) already exists in the ipvs service tcp:172.16.148.198:443 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.859922 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.18.187:80 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.859997 3280670 network_services_controller.go:1805] ipvs destination 10.0.10.142:8080 (Weight: 1) already exists in the ipvs service tcp:172.16.18.187:80 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.860044 3280670 network_services_controller.go:1805] ipvs destination 10.0.12.63:8080 (Weight: 1) already exists in the ipvs service tcp:172.16.18.187:80 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.861285 3280670 network_services_controller.go:1703] ipvs service tcp:172.16.214.142:80 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.861461 3280670 network_services_controller.go:1703] ipvs service tcp:10.103.0.10:30386 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.861502 3280670 network_services_controller.go:1805] ipvs destination 10.0.10.41:8443 (Weight: 1) already exists in the ipvs service tcp:10.103.0.10:30386 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.861516 3280670 network_services_controller.go:1703] ipvs service tcp:10.103.0.10:31529 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.861542 3280670 network_services_controller.go:1805] ipvs destination 10.0.10.41:8000 (Weight: 1) already exists in the ipvs service tcp:10.103.0.10:31529 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.862904 3280670 network_services_controller.go:1703] ipvs service tcp:142.132.245.229:443 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.862923 3280670 service_endpoints_sync.go:335] no FW mark found for service, nothing to cleanup
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.862969 3280670 network_services_controller.go:1805] ipvs destination 10.0.10.41:8443 (Weight: 1) already exists in the ipvs service tcp:142.132.245.229:443 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: E0412 21:15:04.864550 3280670 network_services_controller.go:195] Failed to replace route to service VIP 2a01:4f8:c011:eef::1 configured on kube-dummy-if. Error: exit status 1, Output: Error: inet6 address is expected rather than "10.103.0.10".
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.864597 3280670 network_services_controller.go:1723] Successfully added service: tcp:2a01:4f8:c011:eef::1:443 (Flags: )
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.864604 3280670 service_endpoints_sync.go:335] no FW mark found for service, nothing to cleanup
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.864626 3280670 network_services_controller.go:1794] Successfully added destination 10.0.10.41:8443 (Weight: 1) to the service tcp:2a01:4f8:c011:eef::1:443 (Flags: )
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.865938 3280670 network_services_controller.go:1703] ipvs service tcp:142.132.245.229:80 (Flags: [hashed entry]) already exists so returning
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.865954 3280670 service_endpoints_sync.go:335] no FW mark found for service, nothing to cleanup
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.865992 3280670 network_services_controller.go:1805] ipvs destination 10.0.10.41:8000 (Weight: 1) already exists in the ipvs service tcp:142.132.245.229:80 (Flags: [hashed entry]) so not adding destination
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: E0412 21:15:04.867335 3280670 network_services_controller.go:195] Failed to replace route to service VIP 2a01:4f8:c011:eef::1 configured on kube-dummy-if. Error: exit status 1, Output: Error: inet6 address is expected rather than "10.103.0.10".
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.867386 3280670 network_services_controller.go:1723] Successfully added service: tcp:2a01:4f8:c011:eef::1:80 (Flags: )
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.867392 3280670 service_endpoints_sync.go:335] no FW mark found for service, nothing to cleanup
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.867413 3280670 network_services_controller.go:1794] Successfully added destination 10.0.10.41:8000 (Weight: 1) to the service tcp:2a01:4f8:c011:eef::1:80 (Flags: )
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.867422 3280670 service_endpoints_sync.go:477] Cleaning up if any, old service IPs on dummy interface
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.867654 3280670 service_endpoints_sync.go:521] Cleaning up if any, old ipvs service and servers which are no longer needed
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.867771 3280670 service_endpoints_sync.go:564] Found a IPVS service tcp:42.1.4.248:443 (Flags: [hashed entry]) which is no longer needed so cleaning up
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.868069 3280670 service_endpoints_sync.go:564] Found a IPVS service tcp:42.1.4.248:80 (Flags: [hashed entry]) which is no longer needed so cleaning up
Apr 12 21:15:04 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:04.868231 3280670 network_services_controller.go:751] Attempting to attain ipset mutex lock
Apr 12 21:15:05 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:05.126655 3280670 policy.go:142] Iptables chains in the filter table are synchronized with the network policies.
Apr 12 21:15:05 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:05.126682 3280670 policy.go:85] Returned ipset mutex lock
Apr 12 21:15:05 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:05.126693 3280670 policy.go:77] Syncing network policy chains took 287.812882ms
Apr 12 21:15:05 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:05.126714 3280670 network_services_controller.go:753] Attained ipset mutex lock, continuing...
Apr 12 21:15:05 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:05.224429 3280670 network_policy_controller.go:551] Attempting to attain ipset mutex lock
Apr 12 21:15:05 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:05.318585 3280670 network_services_controller.go:756] Returned ipset mutex lock
Apr 12 21:15:05 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:05.318609 3280670 service_endpoints_sync.go:454] Setting up policy routing required for Direct Server Return functionality.
Apr 12 21:15:05 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:05.318631 3280670 network_policy_controller.go:553] Attained ipset mutex lock, continuing...
Apr 12 21:15:05 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:05.320460 3280670 network_policy_controller.go:556] Returned ipset mutex lock
Apr 12 21:15:05 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:05.320583 3280670 network_policy_controller.go:233] sync iptables took 514.14649ms
Apr 12 21:15:05 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:05.320500 3280670 service_endpoints_sync.go:459] Custom routing table kube-router-dsr required for Direct Server Return is setup as expected.
Apr 12 21:15:05 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:05.320604 3280670 service_endpoints_sync.go:462] Setting up custom route table required to add routes for external IP's.
Apr 12 21:15:05 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:05.323624 3280670 service_endpoints_sync.go:470] Custom routing table required for Direct Server Return (external_ip) is setup as expected.
Apr 12 21:15:05 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:05.323654 3280670 service_endpoints_sync.go:86] IPVS servers and services are synced to desired state
Apr 12 21:15:05 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:05.323662 3280670 service_endpoints_sync.go:31] sync ipvs services took 503.767778ms
Apr 12 21:15:05 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:05.323675 3280670 network_services_controller.go:1383] No hairpin-mode enabled services found -- no hairpin rules created
Apr 12 21:15:05 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:05.804962 3280670 pod.go:53] Received update to pod: test/client
Apr 12 21:15:05 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:05.804998 3280670 network_policy_controller.go:236] Starting sync of iptables with version: 1712956505804992805
Apr 12 21:15:05 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:05.839824 3280670 policy.go:80] Attempting to attain ipset mutex lock
Apr 12 21:15:05 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:05.839855 3280670 policy.go:82] Attained ipset mutex lock, continuing...
Apr 12 21:15:06 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:06.102744 3280670 policy.go:142] Iptables chains in the filter table are synchronized with the network policies.
Apr 12 21:15:06 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:06.102771 3280670 policy.go:85] Returned ipset mutex lock
Apr 12 21:15:06 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:06.102779 3280670 policy.go:77] Syncing network policy chains took 262.962173ms
Apr 12 21:15:06 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:06.120549 3280670 network_policy_controller.go:551] Attempting to attain ipset mutex lock
Apr 12 21:15:06 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:06.120577 3280670 network_policy_controller.go:553] Attained ipset mutex lock, continuing...
Apr 12 21:15:06 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:06.122647 3280670 network_policy_controller.go:556] Returned ipset mutex lock
Apr 12 21:15:06 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:06.122669 3280670 network_policy_controller.go:233] sync iptables took 317.672239ms
Apr 12 21:15:21 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:15:21.503362 3280670 route_sync.go:41] Running local route table synchronization
Apr 12 21:16:21 general-74aab0e3d49bdfda kube-router[3280670]: I0412 21:16:21.503940 3280670 route_sync.go:41] Running local route table synchronization

from kube-router.

vladimirtiukhtin commented on June 27, 2024

I upgraded to 2.1.0, result is the same - reject on service IPs. Direct requests to pods work

from kube-router.

aauren commented on June 27, 2024

@vladimirtiukhtin - Sorry for the run around on this one. I always forget this routing scenario as it is a bit obscure. But I have a good understanding of what you're doing now and I can try to help explain what's happening.

Essentially, what happens is that your traffic flow looks like the following:

Traffic egresses the application towards the ClusterIP
ClusterIP is not within the Pod's Network Namespace
Traffic enters the client node's host network namespace
Traffic is accepted by the kube-dummy-if interface
Traffic is ingressed into the client node via IPVS
Traffic is load balanced and sent from the Node to the IPVS endpoint (Server Pod IP Address) - This is the important step because at this point, the traffic has been SNAT'd to the node's IP address
Traffic routes to the Server Pod IP Address, but the source is now the node IP address of the client
Traffic arrives at the Server Node, but iptables denies it because the client node's IP address isn't in the network policy, only the pod is

The solution for this is to allow list all of your node IPs in your network policy. I know that this is a hassle, but there essentially isn't anything that CNI's can do about it.

Basically, the problem is that network policies were created in terms of pod communication and are service agnostic. This means that they don't have good semantics for this type of traffic flow.

Here are some additional resources that talk about this topic a bit (although not all the details are 1:1 to your scenario since many of them are talking about traffic flowing from external nodes, but its essentially the same issue):

from kube-router.

vladimirtiukhtin commented on June 27, 2024

Hi @aauren. Thank for the response. Your description is correct till

Traffic is load balanced and sent from the Node to the IPVS endpoint (Server Pod IP Address) - This is the important step because at this point, the traffic has been SNAT'd to the node's IP address

By allowing all traffic I get

# kubectl -n test get pods -o wide
NAME     READY   STATUS    RESTARTS   AGE   IP          NODE                       NOMINATED NODE   READINESS GATES
client   1/1     Running   0          43h   10.0.4.16   general-6037108a7c52f8a1   <none>           <none>
server   1/1     Running   0          43h   10.0.6.9    general-3779a47f918eebf3   <none>           <none>
# kubectl -n test get svc
NAME     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
server   ClusterIP   172.16.88.116   <none>        80/TCP    43h

Then exec

# kubectl -n test exec -ti client -- sh
/ # nc -v 172.16.88.116:80
172.16.88.116:80 (172.16.88.116:80) open

On the server pod node

# tcpdump -i any -nnn host 10.0.6.9
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes

08:53:38.278507 enp7s0 In  IP 10.0.4.16.44723 > 10.0.6.9.8080: Flags [S], seq 3767745513, win 64860, options [mss 1410,sackOK,TS val 764536396 ecr 0,nop,wscale 7], length 0
08:53:38.278583 kube-bridge Out IP 10.0.4.16.44723 > 10.0.6.9.8080: Flags [S], seq 3767745513, win 64860, options [mss 1410,sackOK,TS val 764536396 ecr 0,nop,wscale 7], length 0
08:53:38.278591 veth037bd9ef Out IP 10.0.4.16.44723 > 10.0.6.9.8080: Flags [S], seq 3767745513, win 64860, options [mss 1410,sackOK,TS val 764536396 ecr 0,nop,wscale 7], length 0
08:53:38.278623 veth037bd9ef P   IP 10.0.6.9.8080 > 10.0.4.16.44723: Flags [S.], seq 1718754541, ack 3767745514, win 64308, options [mss 1410,sackOK,TS val 2712653961 ecr 764536396,nop,wscale 7], length 0
08:53:38.278630 kube-bridge In  IP 10.0.6.9.8080 > 10.0.4.16.44723: Flags [S.], seq 1718754541, ack 3767745514, win 64308, options [mss 1410,sackOK,TS val 2712653961 ecr 764536396,nop,wscale 7], length 0
08:53:38.278637 enp7s0 Out IP 10.0.6.9.8080 > 10.0.4.16.44723: Flags [S.], seq 1718754541, ack 3767745514, win 64308, options [mss 1410,sackOK,TS val 2712653961 ecr 764536396,nop,wscale 7], length 0
08:53:38.279064 enp7s0 In  IP 10.0.4.16.44723 > 10.0.6.9.8080: Flags [.], ack 1, win 507, options [nop,nop,TS val 764536397 ecr 2712653961], length 0
08:53:38.279072 kube-bridge Out IP 10.0.4.16.44723 > 10.0.6.9.8080: Flags [.], ack 1, win 507, options [nop,nop,TS val 764536397 ecr 2712653961], length 0
08:53:38.279073 veth037bd9ef Out IP 10.0.4.16.44723 > 10.0.6.9.8080: Flags [.], ack 1, win 507, options [nop,nop,TS val 764536397 ecr 2712653961], length 0
08:53:38.279442 veth037bd9ef P   IP 10.0.6.9.36111 > 172.16.0.53.53: 16041+ PTR? 16.4.0.10.in-addr.arpa. (40)
08:53:38.279450 kube-bridge In  IP 10.0.6.9.36111 > 172.16.0.53.53: 16041+ PTR? 16.4.0.10.in-addr.arpa. (40)
08:53:38.279502 enp7s0 Out IP 10.0.6.9.36111 > 10.0.2.30.53: 16041+ PTR? 16.4.0.10.in-addr.arpa. (40)
08:53:38.282743 enp7s0 In  IP 10.0.2.30.53 > 10.0.6.9.36111: 16041 NXDomain* 0/1/0 (114)
08:53:38.282764 kube-bridge Out IP 172.16.0.53.53 > 10.0.6.9.36111: 16041 NXDomain* 0/1/0 (114)
08:53:38.282768 veth037bd9ef Out IP 172.16.0.53.53 > 10.0.6.9.36111: 16041 NXDomain* 0/1/0 (114)
08:53:38.283570 veth037bd9ef P   IP 10.0.6.9.8080 > 10.0.4.16.44723: Flags [F.], seq 1, ack 1, win 503, options [nop,nop,TS val 2712653966 ecr 764536397], length 0
08:53:38.283577 kube-bridge In  IP 10.0.6.9.8080 > 10.0.4.16.44723: Flags [F.], seq 1, ack 1, win 503, options [nop,nop,TS val 2712653966 ecr 764536397], length 0
08:53:38.283588 enp7s0 Out IP 10.0.6.9.8080 > 10.0.4.16.44723: Flags [F.], seq 1, ack 1, win 503, options [nop,nop,TS val 2712653966 ecr 764536397], length 0
08:53:38.286029 enp7s0 In  IP 10.0.4.16.44723 > 10.0.6.9.8080: Flags [.], ack 2, win 507, options [nop,nop,TS val 764536404 ecr 2712653966], length 0
08:53:38.286047 kube-bridge Out IP 10.0.4.16.44723 > 10.0.6.9.8080: Flags [.], ack 2, win 507, options [nop,nop,TS val 764536404 ecr 2712653966], length 0
08:53:38.286051 veth037bd9ef Out IP 10.0.4.16.44723 > 10.0.6.9.8080: Flags [.], ack 2, win 507, options [nop,nop,TS val 764536404 ecr 2712653966], length 0
08:53:43.467307 kube-bridge Out ARP, Request who-has 10.0.6.9 tell 10.0.6.1, length 28
08:53:43.467312 veth037bd9ef Out ARP, Request who-has 10.0.6.9 tell 10.0.6.1, length 28
08:53:43.467353 veth037bd9ef P   ARP, Request who-has 10.0.6.1 tell 10.0.6.9, length 28
08:53:43.467357 kube-bridge In  ARP, Request who-has 10.0.6.1 tell 10.0.6.9, length 28
08:53:43.467366 kube-bridge Out ARP, Reply 10.0.6.1 is-at 42:f8:ca:17:c5:9b, length 28
08:53:43.467368 veth037bd9ef Out ARP, Reply 10.0.6.1 is-at 42:f8:ca:17:c5:9b, length 28
08:53:43.467376 veth037bd9ef P   ARP, Reply 10.0.6.9 is-at f6:c7:37:53:ee:2c, length 28
08:53:43.467377 kube-bridge In  ARP, Reply 10.0.6.9 is-at f6:c7:37:53:ee:2c, length 28

On client pod node

# tcpdump -i any -nnn '(host 10.0.4.16 or host 10.0.6.9 or host 172.16.88.116)'
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes

08:59:42.786951 vethf3083409 P   IP 10.0.4.16.44723 > 172.16.88.116.80: Flags [F.], seq 3767745514, ack 1718754543, win 507, options [nop,nop,TS val 764900904 ecr 2712653966], length 0
08:59:42.786983 kube-bridge In  IP 10.0.4.16.44723 > 172.16.88.116.80: Flags [F.], seq 0, ack 1, win 507, options [nop,nop,TS val 764900904 ecr 2712653966], length 0
08:59:42.994375 vethf3083409 P   IP 10.0.4.16.44723 > 172.16.88.116.80: Flags [F.], seq 0, ack 1, win 507, options [nop,nop,TS val 764901112 ecr 2712653966], length 0
08:59:42.994410 kube-bridge In  IP 10.0.4.16.44723 > 172.16.88.116.80: Flags [F.], seq 0, ack 1, win 507, options [nop,nop,TS val 764901112 ecr 2712653966], length 0
08:59:43.202305 vethf3083409 P   IP 10.0.4.16.44723 > 172.16.88.116.80: Flags [F.], seq 0, ack 1, win 507, options [nop,nop,TS val 764901320 ecr 2712653966], length 0
08:59:43.202338 kube-bridge In  IP 10.0.4.16.44723 > 172.16.88.116.80: Flags [F.], seq 0, ack 1, win 507, options [nop,nop,TS val 764901320 ecr 2712653966], length 0
08:59:43.642292 vethf3083409 P   IP 10.0.4.16.44723 > 172.16.88.116.80: Flags [F.], seq 0, ack 1, win 507, options [nop,nop,TS val 764901760 ecr 2712653966], length 0
08:59:43.642303 kube-bridge In  IP 10.0.4.16.44723 > 172.16.88.116.80: Flags [F.], seq 0, ack 1, win 507, options [nop,nop,TS val 764901760 ecr 2712653966], length 0
08:59:44.445047 vethf3083409 P   IP 10.0.4.16.37509 > 172.16.88.116.80: Flags [S], seq 2878928928, win 64860, options [mss 1410,sackOK,TS val 764902562 ecr 0,nop,wscale 7], length 0
08:59:44.445061 kube-bridge In  IP 10.0.4.16.37509 > 172.16.88.116.80: Flags [S], seq 2878928928, win 64860, options [mss 1410,sackOK,TS val 764902562 ecr 0,nop,wscale 7], length 0
08:59:44.445135 enp7s0 Out IP 10.0.4.16.37509 > 10.0.6.9.8080: Flags [S], seq 2878928928, win 64860, options [mss 1410,sackOK,TS val 764902562 ecr 0,nop,wscale 7], length 0
08:59:44.445693 enp7s0 In  IP 10.0.6.9.8080 > 10.0.4.16.37509: Flags [S.], seq 2246986623, ack 2878928929, win 64308, options [mss 1410,sackOK,TS val 2713020127 ecr 764902562,nop,wscale 7], length 0
08:59:44.445734 kube-bridge Out IP 172.16.88.116.80 > 10.0.4.16.37509: Flags [S.], seq 2246986623, ack 2878928929, win 64308, options [mss 1410,sackOK,TS val 2713020127 ecr 764902562,nop,wscale 7], length 0
08:59:44.445740 vethf3083409 Out IP 172.16.88.116.80 > 10.0.4.16.37509: Flags [S.], seq 2246986623, ack 2878928929, win 64308, options [mss 1410,sackOK,TS val 2713020127 ecr 764902562,nop,wscale 7], length 0
08:59:44.445785 vethf3083409 P   IP 10.0.4.16.37509 > 172.16.88.116.80: Flags [.], ack 1, win 507, options [nop,nop,TS val 764902563 ecr 2713020127], length 0
08:59:44.445791 kube-bridge In  IP 10.0.4.16.37509 > 172.16.88.116.80: Flags [.], ack 1, win 507, options [nop,nop,TS val 764902563 ecr 2713020127], length 0
08:59:44.445812 enp7s0 Out IP 10.0.4.16.37509 > 10.0.6.9.8080: Flags [.], ack 1, win 507, options [nop,nop,TS val 764902563 ecr 2713020127], length 0
08:59:44.450806 enp7s0 In  IP 10.0.6.9.8080 > 10.0.4.16.37509: Flags [F.], seq 1, ack 1, win 503, options [nop,nop,TS val 2713020132 ecr 764902563], length 0
08:59:44.450850 kube-bridge Out IP 172.16.88.116.80 > 10.0.4.16.37509: Flags [F.], seq 1, ack 1, win 503, options [nop,nop,TS val 2713020132 ecr 764902563], length 0
08:59:44.450859 vethf3083409 Out IP 172.16.88.116.80 > 10.0.4.16.37509: Flags [F.], seq 1, ack 1, win 503, options [nop,nop,TS val 2713020132 ecr 764902563], length 0
08:59:44.454266 vethf3083409 P   IP 10.0.4.16.37509 > 172.16.88.116.80: Flags [.], ack 2, win 507, options [nop,nop,TS val 764902572 ecr 2713020132], length 0
08:59:44.454273 kube-bridge In  IP 10.0.4.16.37509 > 172.16.88.116.80: Flags [.], ack 2, win 507, options [nop,nop,TS val 764902572 ecr 2713020132], length 0
08:59:44.454295 enp7s0 Out IP 10.0.4.16.37509 > 10.0.6.9.8080: Flags [.], ack 2, win 507, options [nop,nop,TS val 764902572 ecr 2713020132], length 0
08:59:44.474298 vethf3083409 P   IP 10.0.4.16.44723 > 172.16.88.116.80: Flags [F.], seq 0, ack 1, win 507, options [nop,nop,TS val 764902592 ecr 2712653966], length 0
08:59:44.474323 kube-bridge In  IP 10.0.4.16.44723 > 172.16.88.116.80: Flags [F.], seq 0, ack 1, win 507, options [nop,nop,TS val 764902592 ecr 2712653966], length 0
08:59:46.138273 vethf3083409 P   IP 10.0.4.16.44723 > 172.16.88.116.80: Flags [F.], seq 0, ack 1, win 507, options [nop,nop,TS val 764904256 ecr 2712653966], length 0
08:59:46.138289 kube-bridge In  IP 10.0.4.16.44723 > 172.16.88.116.80: Flags [F.], seq 0, ack 1, win 507, options [nop,nop,TS val 764904256 ecr 2712653966], length 0
08:59:49.626234 kube-bridge Out ARP, Request who-has 10.0.4.16 tell 10.0.4.1, length 28
08:59:49.626245 vethf3083409 Out ARP, Request who-has 10.0.4.16 tell 10.0.4.1, length 28
08:59:49.626282 vethf3083409 P   ARP, Reply 10.0.4.16 is-at be:7a:6d:62:7d:38, length 28
08:59:49.626288 kube-bridge In  ARP, Reply 10.0.4.16 is-at be:7a:6d:62:7d:38, length 28
08:59:49.626388 vethf3083409 P   IP 10.0.4.16.44723 > 172.16.88.116.80: Flags [F.], seq 0, ack 1, win 507, options [nop,nop,TS val 764907744 ecr 2712653966], length 0
08:59:49.626422 kube-bridge In  IP 10.0.4.16.44723 > 172.16.88.116.80: Flags [F.], seq 0, ack 1, win 507, options [nop,nop,TS val 764907744 ecr 2712653966], length 0

As you see, no SNAT happening. I also checked, if I add clusterIP as the ipblock to egress it works. But I expect kube-router to resolve clusterIP on its own

from kube-router.

aauren commented on June 27, 2024

Are you adding --service-cluster-ip-range (see https://www.kube-router.io/docs/user-guide/) as a parameter to kube-router?

This is the only way that kube-router is aware of your IP ranges since they can vary by Kubernetes orchestration method.

If not, if you add that, does this fix the problem you're seeing?

from kube-router.

vladimirtiukhtin commented on June 27, 2024

No, but by adding this flag issue disappears. Thanks a lot, this was not obvious. BTW when I was doing that I ran --cleanup-config and got an error. Unrelated to the subj, but might be worth to look at

# kube-router --cleanup-config
I0419 08:38:35.287507  288326 network_policy_controller.go:713] Cleaning up NetworkPolicyController configurations...
I0419 08:38:35.287552  288326 network_policy_controller.go:746] Successfully cleaned the NetworkPolicyController configurations done by kube-router
I0419 08:38:35.287575  288326 network_services_controller.go:1755] Cleaning up NetworkServiceController configurations...
I0419 08:38:35.289694  288326 network_services_controller.go:1762] ipvs definitions don't have names associated with them for checking, during cleanup we assume that we own all of them and delete all ipvs definitions
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x98 pc=0x180e7d2]

goroutine 1 [running]:
github.com/cloudnativelabs/kube-router/v2/pkg/controllers/proxy.(*NetworkServicesController).deleteHairpinIptablesRules(0xc000628010?, {0x2262d11?, 0x0?})
	/home/runner/work/kube-router/kube-router/pkg/controllers/proxy/network_services_controller.go:1436 +0x52
github.com/cloudnativelabs/kube-router/v2/pkg/controllers/proxy.(*NetworkServicesController).Cleanup(0xc00061f8b0?)
	/home/runner/work/kube-router/kube-router/pkg/controllers/proxy/network_services_controller.go:1779 +0x32b
github.com/cloudnativelabs/kube-router/v2/pkg/cmd.CleanupConfigAndExit()
	/home/runner/work/kube-router/kube-router/pkg/cmd/kube-router.go:71 +0x85
main.Main()
	/home/runner/work/kube-router/kube-router/cmd/kube-router/kube-router.go:65 +0x406
main.main()
	/home/runner/work/kube-router/kube-router/cmd/kube-router/kube-router.go:21 +0x13

from kube-router.

aauren commented on June 27, 2024

Thanks for reporting --cleanup-config. The error that you ran into makes immediate sense after looking at the code. We'll fix that up.

from kube-router.

aauren commented on June 27, 2024

Tracking the cleanup-config issue in #1649 so I'm going to close this one now

I'm glad that adding the --service-cluster-ip-range option fixed the issues you were experiencing. If you have any thoughts on where we could add information about that option and its sibling ip-range options, so that users have a better understanding about the impact of these options, let us know.

Or, if you have a time, a PR for additional docs is always welcome!

from kube-router.

NetworkPolicy's Egress is not working about kube-router HOT 12 CLOSED

Comments (12)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent