macvtap-cni to connect on layer 2 with other cnis about macvtap-cni HOT 8 CLOSED

We have noticed that @kvaps already did exactly that.

kubevirt/kubevirt#7648
kubevirt/community#186

We will try to get this to work to solve our issue.

from macvtap-cni.

I have tried the following configuration.
The virt-launcher starts successfully, and I the interface is liked correctly to ovn. However, the VM stays in the Scheduled state and doesn't start. I have not been able to find any reason for this. Do you have an idea why ?

kind: Vpc
apiVersion: kubeovn.io/v1
metadata:
  name: virt-lab1-vpc
spec:
---
kind: Subnet
apiVersion: kubeovn.io/v1
metadata:
  name: virt-lab1
spec:
  vpc: virt-lab1-vpc
  provider: net1.default.ovn
  cidrBlock: 10.0.1.0/24
  protocol: IPv4
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: net1
  namespace: default
spec:
  config: '{
      "cniVersion": "0.3.0",
      "type": "kube-ovn",
      "server_socket": "/run/openvswitch/kube-ovn-daemon.sock",
      "provider": "net1.default.ovn",
      "promiscMode": true
    }'
---
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
  name: vm-ubuntu-1
spec:
  running: true
  template:
    metadata:
      labels:
        special: vmi-macvtap
    spec:
      nodeSelector:
        kubernetes.io/hostname: node1
      domain:
        devices:
          disks:
            - name: containerdisk
              disk:
                bus: virtio
            - name: cloudinitdisk
              disk:
                bus: virtio
          interfaces:
          - name: default
            masquerade: {}
          - name: l2-network
            macvtap: {}
        machine:
          type: ""
        resources:
          requests:
            memory: 1024M
      networks:
      - name: default
        pod: {}
      - name: l2-network
        multus: # Secondary multus network
          networkName: net1

      terminationGracePeriodSeconds: 0
      volumes:
        - name: containerdisk
          containerDisk:
            image: quay.io/containerdisks/ubuntu:22.04
        - name: cloudinitdisk
          cloudInitNoCloud:
            networkData: |
              version: 2
              ethernets:
                enp1s0:
                  dhcp4: true
                enp2s0:
                  addresses:
                    - 10.0.1.2/24
            userData: |-
              #cloud-config
              password: ubuntu
              chpasswd: { expire: False }
              ssh_authorized_keys:
                - ssh-rsa 
              packages: 
                - qemu-guest-agent
                - lldpd
                - nmap
              runcmd:
                - [ systemctl, start, qemu-guest-agent]
---
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
  name: vm-ubuntu-2
spec:
  running: true
  template:
    metadata:
      labels:
        special: vmi-macvtap
    spec:
      nodeSelector:
        kubernetes.io/hostname: node2
      domain:
        devices:
          disks:
            - name: containerdisk
              disk:
                bus: virtio
            - name: cloudinitdisk
              disk:
                bus: virtio
          interfaces:
          - name: default
            masquerade: {}
          - name: l2-network
            macvtap: {}
        machine:
          type: ""
        resources:
          requests:
            memory: 1024M
      networks:
      - name: default
        pod: {}
      - name: l2-network
        multus: # Secondary multus network
          networkName: net1

      terminationGracePeriodSeconds: 0
      volumes:
        - name: containerdisk
          containerDisk:
            image: quay.io/containerdisks/ubuntu:22.04
        - name: cloudinitdisk
          cloudInitNoCloud:
            networkData: |
              version: 2
              ethernets:
                enp1s0:
                  dhcp4: true
                enp2s0:
                  addresses:
                    - 10.0.1.3/24
            userData: |-
              #cloud-config
              password: ubuntu
              chpasswd: { expire: False }
              ssh_authorized_keys:
                - ssh-rsa 
              packages: 
                - qemu-guest-agent
                - lldpd
                - nmap
              runcmd:
                - [ systemctl, start, qemu-guest-agent]

Interfaces:

110: 2bfd2aa6_net1_h@if109: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000
    link/ether ce:5d:a9:7b:a8:d7 brd ff:ff:ff:ff:ff:ff link-netns cni-930cbdbf-0ee2-10f0-7021-beb7912911a4

netns: cni-930cbdbf-0ee2-10f0-7021-beb7912911a4
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: k6t-eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1400 qdisc noqueue state DOWN mode DEFAULT group default
    link/ether 02:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
3: tap0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1400 qdisc fq_codel master k6t-eth0 state DOWN mode DEFAULT group default qlen 1000
    link/ether f2:cd:72:d5:b8:27 brd ff:ff:ff:ff:ff:ff
107: eth0@if108: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP mode DEFAULT group default
    link/ether 00:00:00:61:cc:37 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    alias 2bfd2aa6e0de_c
109: net1@if110: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP mode DEFAULT group default
    link/ether 00:00:00:85:77:b9 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    alias 2bfd2aa6_net1_c

from macvtap-cni.

I have tried the following configuration. The virt-launcher starts successfully, and I the interface is liked correctly to ovn. However, the VM stays in the Scheduled state and doesn't start. I have not been able to find any reason for this. Do you have an idea why ?

kind: Vpc
apiVersion: kubeovn.io/v1
metadata:
  name: virt-lab1-vpc
spec:
---
kind: Subnet
apiVersion: kubeovn.io/v1
metadata:
  name: virt-lab1
spec:
  vpc: virt-lab1-vpc
  provider: net1.default.ovn
  cidrBlock: 10.0.1.0/24
  protocol: IPv4
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: net1
  namespace: default
spec:
  config: '{
      "cniVersion": "0.3.0",
      "type": "kube-ovn",
      "server_socket": "/run/openvswitch/kube-ovn-daemon.sock",
      "provider": "net1.default.ovn",
      "promiscMode": true
    }'
---
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
  name: vm-ubuntu-1
spec:
  running: true
  template:
    metadata:
      labels:
        special: vmi-macvtap
    spec:
      nodeSelector:
        kubernetes.io/hostname: node1
      domain:
        devices:
          disks:
            - name: containerdisk
              disk:
                bus: virtio
            - name: cloudinitdisk
              disk:
                bus: virtio
          interfaces:
          - name: default
            masquerade: {}
          - name: l2-network
            macvtap: {}
        machine:
          type: ""
        resources:
          requests:
            memory: 1024M
      networks:
      - name: default
        pod: {}
      - name: l2-network
        multus: # Secondary multus network
          networkName: net1

      terminationGracePeriodSeconds: 0
      volumes:
        - name: containerdisk
          containerDisk:
            image: quay.io/containerdisks/ubuntu:22.04
        - name: cloudinitdisk
          cloudInitNoCloud:
            networkData: |
              version: 2
              ethernets:
                enp1s0:
                  dhcp4: true
                enp2s0:
                  addresses:
                    - 10.0.1.2/24
            userData: |-
              #cloud-config
              password: ubuntu
              chpasswd: { expire: False }
              ssh_authorized_keys:
                - ssh-rsa 
              packages: 
                - qemu-guest-agent
                - lldpd
                - nmap
              runcmd:
                - [ systemctl, start, qemu-guest-agent]
---
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
  name: vm-ubuntu-2
spec:
  running: true
  template:
    metadata:
      labels:
        special: vmi-macvtap
    spec:
      nodeSelector:
        kubernetes.io/hostname: node2
      domain:
        devices:
          disks:
            - name: containerdisk
              disk:
                bus: virtio
            - name: cloudinitdisk
              disk:
                bus: virtio
          interfaces:
          - name: default
            masquerade: {}
          - name: l2-network
            macvtap: {}
        machine:
          type: ""
        resources:
          requests:
            memory: 1024M
      networks:
      - name: default
        pod: {}
      - name: l2-network
        multus: # Secondary multus network
          networkName: net1

      terminationGracePeriodSeconds: 0
      volumes:
        - name: containerdisk
          containerDisk:
            image: quay.io/containerdisks/ubuntu:22.04
        - name: cloudinitdisk
          cloudInitNoCloud:
            networkData: |
              version: 2
              ethernets:
                enp1s0:
                  dhcp4: true
                enp2s0:
                  addresses:
                    - 10.0.1.3/24
            userData: |-
              #cloud-config
              password: ubuntu
              chpasswd: { expire: False }
              ssh_authorized_keys:
                - ssh-rsa 
              packages: 
                - qemu-guest-agent
                - lldpd
                - nmap
              runcmd:
                - [ systemctl, start, qemu-guest-agent]

Interfaces:

110: 2bfd2aa6_net1_h@if109: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000
    link/ether ce:5d:a9:7b:a8:d7 brd ff:ff:ff:ff:ff:ff link-netns cni-930cbdbf-0ee2-10f0-7021-beb7912911a4

netns: cni-930cbdbf-0ee2-10f0-7021-beb7912911a4
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: k6t-eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1400 qdisc noqueue state DOWN mode DEFAULT group default
    link/ether 02:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
3: tap0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1400 qdisc fq_codel master k6t-eth0 state DOWN mode DEFAULT group default qlen 1000
    link/ether f2:cd:72:d5:b8:27 brd ff:ff:ff:ff:ff:ff
107: eth0@if108: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP mode DEFAULT group default
    link/ether 00:00:00:61:cc:37 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    alias 2bfd2aa6e0de_c
109: net1@if110: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP mode DEFAULT group default
    link/ether 00:00:00:85:77:b9 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    alias 2bfd2aa6_net1_c

AFAIU, you're not using a macvtap interface: your net-attach-def requests a secondary interface of type kube-ovn. This means the macvtap-cni will not even be invoked.

from macvtap-cni.

IIUC, you are requesting an entirely new feature: you want to use as macvtap's lower device an interface that exists in the pod network namespace.

This project was created to ensure we could connect to host networks in a simple way; the solution was then to use a device plugin + CNI combo, aided by multus to get the character device available in the pod.

I don't think what we have currently is a good fit for what you're requesting, but let me think this through a little bit.

/cc @kvaps this sounds like something you would be interested on.

from macvtap-cni.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

from macvtap-cni.

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

from macvtap-cni.

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

from macvtap-cni.

@kubevirt-bot: Closing this issue.

from macvtap-cni.