ligato / vpp-agent Goto Github PK

View Code? Open in Web Editor NEW

249.0 26.0 128.0 174.8 MB

⚡️ Control plane management agent for FD.io's VPP

Home Page: https://docs.ligato.io/

License: Apache License 2.0

Makefile 0.24% Go 82.17% Shell 0.98% RobotFramework 15.03% Python 1.51% Dockerfile 0.08%

vnf cloud-native control-plane management-agent vpp

vpp-agent's Introduction

VPP Agent

The VPP Agent is a Go implementation of a control/management plane for VPP based cloud-native Virtual Network Functions (VNFs). The VPP Agent is built on top of CN Infra, a framework for developing cloud-native VNFs (CNFs).

The VPP Agent can be used as-is as a management/control agent for VNFs based on off-the-shelf VPP (e.g. a VPP-based vswitch), or as a framework for developing management agents for VPP-based CNFs. An example of a custom VPP-based CNF is the Contiv-VPP vswitch.

Please note that the content of this repository is currently WORK IN PROGRESS!

Status

Releases

Release	Release Date	Info
		latest release

Have a look at the release notes for a complete list of changes.

Branches

Branch	Info	Last Commit
	has switched to v3 ⚠️
	has been DEPRECATED
	provides legacy v2

All development is done against master branch.

Images

Image	Image Size/Layers		Info
			with minimal footprint
			prepared for developers

Quickstart

For a quick start with the VPP Agent, you can use the pre-built Docker images on DockerHub that contain the VPP Agent and VPP: ligato/vpp-agent (or for ARM64: ligato/vpp-agent-arm64).

Start ETCD on your host (e.g. in Docker as described here).

Note: for ARM64 see the information for etcd.
Run VPP + VPP Agent in a Docker container:

docker run -it --rm --name agent1 --privileged ligato/vpp-agent

Manage VPP agent using agentctl:

docker exec -it agent1 agentctl --help
docker exec -it agent1 agentctl status

Check the configuration (via agentctl or in VPP console):

docker exec -it agent1 agentctl dump all
docker exec -it agent1 vppctl show interface

Next Steps

See README of development docker image for more details.

Documentation

Extensive documentation for the VPP Agent can be found at docs.ligato.io.

Architecture

The VPP Agent is basically a set of VPP-specific plugins that use the CN-Infra framework to interact with other services/microservices in the cloud (e.g. a KV data store, messaging, log warehouse, etc.). The VPP Agent exposes VPP functionality to client apps via a higher-level model-driven API. Clients that consume this API may be either external (connecting to the VPP Agent via REST, gRPC API, Etcd or message bus transport), or local Apps and/or Extension plugins running on the same CN-Infra framework in the same Linux process.

The VNF Agent architecture is shown in the following figure:

Each (northbound) VPP API - L2, L3, ACL, ... - is implemented by a specific VNF Agent plugin, which translates northbound API calls/operations into (southbound) low level VPP Binary API calls. Northbound APIs are defined using protobufs, which allow for the same functionality to be accessible over multiple transport protocols (HTTP, gRPC, Etcd, ...). Plugins use the GoVPP library to interact with the VPP.

The following figure shows the VPP Agent in context of a cloud-native VNF, where the VNF's data plane is implemented using VPP/DPDK and its management/control planes are implemented using the VNF agent:

Contributing

If you are interested in contributing, please see the contribution guidelines.

License

Modified 3rd party tools included

protoc-gen-jsonschema (code location in this repository)

vpp-agent's People

Contributors

Stargazers

Watchers

Forkers

lukasmacko jozef-slezak vladolavor milanlenco martinfrk kolichs donghaima rastislavs xerothermic yzhg1983 amarcine batky cgasparini ondrej-fabry mpundlik dasasimkova fgschwan jgloncak jmedved xunleer jeremypogue swatisehgal aalzugha unclejack everything321 mmiklus samelias brecode deathbeam pkotucek andrejkilvady mestery zurrehma stanislav-chlebec olympusho msestak77 gokulchandrap napoleon0621 jhnbrns matrohon kozemcak rewenset miroslavkovacpantheon ironpillow hollyhock13 drenfongwong artem-belov chrismetz09 denis-tingaikin jgallo542 turbock loeliger emnify matus-mrekaj starrysec linkous8 liuchenshuo aloknnikhil ondrajz sdbrain jaredhancock31 stone-to-sea rraghuvaran tonychenyoyo atimeless leehui85 fswang9659 glazychev-art wxxxxxz mohneeshjhapuresoftware spadalkar notsofancyname ross1503 joye1503 odidev elauriault wenchao5211 smilepleasee ari23 lxlee1102 juliusmilan mhalaj1 skmkt yasics moon-shiny ekmixon 1nce-gmbh timya chenz1231 anooppatwork dreamflychen jqk6 xujunjie-cover hecg119 fanjg123 hcg191132cug bingkunxu samuel-dudik lanzhongwen zw281074593

vpp-agent's Issues

Enable IPv6 support when creating Linux interfaces

In plugins/linux/ifplugin/descriptor/interface.go method Create the ip addresses of the injected interfaces are set in this loop:

	for _, ipAddress := range ipAddresses {
		err = d.ifHandler.AddInterfaceIP(hostName, ipAddress)
		// an attempt to add already assign IP is not considered as error
		if err != nil && syscall.EEXIST != err {
			err = errors.Errorf("failed to add IP address %v to linux interface %s: %v",
				ipAddress, linuxIf.Name, err)
			d.log.Error(err)
			return nil, err
		}
	}

However, due to containernetworking/cni#531, it is impossible to set IPv6 addresses. The problem was solved for CNI in containernetworking/plugins#113 for the IPAM CNI plugin. The essential code is here (https://github.com/containernetworking/plugins/blob/master/pkg/ipam/ipam_linux.go#L61):

		// Make sure sysctl "disable_ipv6" is 0 if we are about to add
		// an IPv6 address to the interface
		if !has_enabled_ipv6 && ipc.Version == "6" {
			// Enabled IPv6 for loopback "lo" and the interface
			// being configured
			for _, iface := range [2]string{"lo", ifName} {
				ipv6SysctlValueName := fmt.Sprintf(DisableIPv6SysctlTemplate, iface)

				// Read current sysctl value
				value, err := sysctl.Sysctl(ipv6SysctlValueName)
				if err != nil || value == "0" {
					// FIXME: log warning if unable to read sysctl value
					continue
				}

				// Write sysctl to enable IPv6
				_, err = sysctl.Sysctl(ipv6SysctlValueName, "0")
				if err != nil {
					return fmt.Errorf("failed to enable IPv6 for interface %q (%s=%s): %v", iface, ipv6SysctlValueName, value, err)
				}
			}
			has_enabled_ipv6 = true
		}

The code is pretty straightforward and can be used as an inspiration to solve the problem with VPP-agent.

API for GRE (including ERSPAN)

We need to support ERSPAN ( see #1386 for SPAN issue) as well as GRE tunneling for remote mechanisms for NSM.

https://github.com/FDio/vpp/blob/b7b929931a/src/vnet/gre/gre.api#L18

Issue creating host-interfaces in vpp-agent

I'm trying to play around with vpp-agent in a simple docker topology (see bottom). VPP and the agent come up fine and each container has 3 interfaces eth0 .. eth3. If I use vppctl i can add the interfaces to VPP using "create host-interface name eth0" etc. but trying the same with vpp-agent does not seem to work. Any help or pointers here would be greatly appreciated even if just to debug this further.

vpp-agent-ctl /opt/vpp-agent/dev/etcd.conf -put /vnf-agent/node1/vpp/config/v1/interface/afpacket1 foo.json

I took the JSON from the example:

{
    "afpacket": {
        "host_if_name": "lo"
    },
    "enabled": true,
    "mtu": 1500,
    "ip_addresses": [
        "fdcd:f7fb:995c::/48"
    ],
    "name": "afpacket1",
    "phys_address": "b4:e6:1c:a1:0d:31",
    "type": 4
}

In fact actually it does work for the "lo" interface in host_if_name but not for any of the eth0 .. eth3 veths of the container.

I don't see any errors in the log (following shows afpacket1 with lo working and afpacket2 for eth0 not working but also not giving any errors).

time="2018-09-05 13:25:46.40518" level=info msg="Configuring new interface afpacket1" loc="ifplugin/interface_config.go(215)" logger=vpp-plugin-if-conf

2018-09-05 13:25:46,605 DEBG 'agent' stdout output:
time="2018-09-05 13:25:46.60557" level=info msg="Interface configuration done" ifIdx=1 ifName=afpacket1 loc="ifplugin/interface_config.go(344)" logger=vpp-plugin-if-conf
time="2018-09-05 13:25:46.60571" level=info msg="Assigning new interface afpacket1 to bridge domain" loc="l2plugin/bd_config.go(322)" logger=vpp-plugin-l2-bd-conf

2018-09-05 13:25:46,606 DEBG 'agent' stdout output:
time="2018-09-05 13:25:46.60577" level=info msg="FIB configurator: resolving registered interface afpacket1" loc="l2plugin/fib_config.go(243)" logger=vpp-plugin-l2-fib-conf
time="2018-09-05 13:25:46.60582" level=info msg="FIB: resolution of created interface afpacket1 is done" loc="l2plugin/fib_config.go(249)" logger=vpp-plugin-l2-fib-conf
time="2018-09-05 13:25:46.60592" level=info msg="Linux IF configurator: resolve created vpp interface name:\"afpacket1\" type:AF_PACKET_INTERFACE enabled:true phys_address:\"b4:e6:1c:a1:0d:31\" mtu:1500 ip_addresses:\"fdcd:f7fb:995c::/48\" afpacket:<host_if_name:\"lo\" > " loc="ifplugin/interface_config.go(978)" logger=linux-plugin-if-conf

2018-09-05 13:27:12,642 DEBG 'agent' stdout output:
time="2018-09-05 13:27:12.64265" level=info msg="Modifying Interface afpacket2" loc="ifplugin/interface_config.go(509)" logger=vpp-plugin-if-conf

2018-09-05 13:27:12,643 DEBG 'agent' stdout output:
time="2018-09-05 13:27:12.64274" level=info msg="Configuring new interface afpacket2" loc="ifplugin/interface_config.go(215)" logger=vpp-plugin-if-conf

In addition I'm not certain if the "name"-field actually does anything, it seems the host interface (at least for lo) is just named host-lo. Same goes for the key of the interface in etcd, not sure what is the significance here.

Finally for reference the docker-compose.yaml I'm using to create the topology:

version: '3'
services:
  etcd:
    image: quay.io/coreos/etcd
    command: ["/usr/local/bin/etcd", "-advertise-client-urls", "http://0.0.0.0:2379", "-listen-client-urls", "http://0.0.0.0:2379"]
    ports: ["2379:2379"]

  kafka:
    image: spotify/kafka
    ports: ["9092:9092"]
    environment:
      ADVERTISED_HOST: 172.17.0.1
      ADVERTISED_PORT: 9092

  node1:
    image: ligato/vpp-agent:v1.6
    depends_on: ["kafka", "etcd"]
    privileged: true
    environment:
      MICROSERVICE_LABEL: node1
    networks:
      - default
      - node1
      - node1_node2
      - node1_node3

  node2:
    image: ligato/vpp-agent:v1.6
    depends_on: ["kafka", "etcd"]
    privileged: true
    environment:
      MICROSERVICE_LABEL: node2
    networks:
      - default
      - node2
      - node1_node2
      - node2_node3

  node3:
    image: ligato/vpp-agent:v1.6
    depends_on: ["kafka", "etcd"]
    privileged: true
    environment:
      MICROSERVICE_LABEL: node3
    networks:
      - default
      - node3
      - node1_node3
      - node2_node3

networks:
  default:
    
  node1:
  node1_node2:
  node1_node3:

  node2:
  node2_node3:

  node3:

Thanks in advance

Walkthough of using the agent

For the purposes of introducing people to both cn-infra and the vpp-agent, we need a training document - kept within the repository and current with the code version - that explains how you would write an agent with intelligence in it and the benefits of using Ligato components to do so. This needs to be at the level of explaining to someone who understands SDN, and has found the VPP API documentation at fd.io, how they might create a simple agent that controls VPP for SDN purposes.

API for SPAN

I need vpp-agent to explose a grpc API for SPAN/RSPAN/ERSPAN.

No ABF in ConfigData

Although ABF is added in https://github.com/ligato/vpp-agent/blob/master/api/models/vpp/abf/abf.proto, there is no way to configure it through https://github.com/ligato/vpp-agent/blob/master/api/models/vpp/vpp.proto.

TAP_TO_VPP interface is not stable after updating

Current Behavior

ping doesn't pass through the connection

Failure Information

after updating cross connect ping doesn't pass. Arp entry is incomplete.

/ # arp
? (10.30.2.2) at <incomplete>  on if2_src
? (10.32.0.1) at 8a:94:19:3b:5f:ed [ether]  on eth0

Reverting commit 8c071c0 resolves problem

Related to networkservicemesh#1128

Log is attached

Context

ligato/vpp-agent:v2.1.0

Failure Logs

nsmd-dataplane.log

manually generated abf FibPath gives error?

@ondrej-fabry i have generated the abf.ba.go file and it have this struct:

type FibPath struct {
SwIfIndex uint32
TableID uint32
Weight uint8
Preference uint8
IsLocal uint8
IsDrop uint8
IsUDPEncap uint8
IsUnreach uint8
IsProhibit uint8
IsResolveHost uint8
IsResolveAttached uint8
IsDvr uint8
IsSourceLookup uint8
Afi uint8
NextHop []byte struc:"[16]byte"
NextHopID uint32
RpfID uint32
ViaLabel uint32
NLabels uint8
LabelStack []FibMplsLabel struc:"[16]FibMplsLabel"

}
the label stack field gives this error:
error struc: field LabelStack is a slice with no length or sizeof field
is this is the solution here

Allowing agent to process change requests without requiring initial resync from NB

vppagent should resync from vpp independently at startup independently of whether it receives a Resync request from the outside world.

Failing to do so means that you guarantee bugs if a simple request comes in and precludes certain valid use patterns.

Remove VAT console usage from robot tests

KVScheduler stats does not show dropped transactions

The stats provided by KVScheduler currently do not provide metrics about transactions which did not get into the transaction queue.

The following log contains several such errors:

$ kubectl logs vswitch-6666f464b4-5rvn2 -p | grep 'level=error'
time="2019-05-24 07:57:22.36538" level=error msg="Transaction #0 finished with 4 errors" loc="orchestrator/dispatcher.go(119)" logger=orchestrator.dispatcher
time="2019-05-24 07:57:22.36559" level=error msg="getting resync data failed: resync returned error: KeyErrors: [config/vpp/v2/interfaces/GigabitEthernetd/0/0 (CREATE): field name is invalid: DPDK interface with given name does not exists, config/vpp/v2/interfaces/GigabitEthernetd/0/1 (CREATE): field name is invalid: DPDK interface with given name does not exists, config/vpp/v2/interfaces/TenGigabitEthernet6/0/0 (CREATE): field name is invalid: DPDK interface with given name does not exists, config/vpp/v2/interfaces/TenGigabitEthernet6/0/1 (CREATE): field name is invalid: DPDK interface with given name does not exists]" loc="kvdbsync/watch_impl.go(101)" logger=defaultLogger
time="2019-05-24 07:57:22.37015" level=error msg="failed to send notifications to KVScheduler: transaction queue is full" loc="descriptor/link_state.go(149)" logger=vpp-ifplugin.interface-link-state
time="2019-05-24 07:57:22.37023" level=error msg="failed to send notifications to KVScheduler: transaction queue is full" loc="descriptor/link_state.go(149)" logger=vpp-ifplugin.interface-link-state
time="2019-05-24 07:57:22.37028" level=error msg="failed to send notifications to KVScheduler: transaction queue is full" loc="descriptor/link_state.go(149)" logger=vpp-ifplugin.interface-link-state
time="2019-05-24 07:57:22.37033" level=error msg="failed to send notifications to KVScheduler: transaction queue is full" loc="descriptor/link_state.go(149)" logger=vpp-ifplugin.interface-link-state
time="2019-05-24 07:57:22.37039" level=error msg="failed to send notifications to KVScheduler: transaction queue is full" loc="descriptor/link_state.go(149)" logger=vpp-ifplugin.interface-link-state
time="2019-05-24 07:57:22.37044" level=error msg="failed to send notifications to KVScheduler: transaction queue is full" loc="descriptor/link_state.go(149)" logger=vpp-ifplugin.interface-link-state
time="2019-05-24 07:57:22.37050" level=error msg="failed to send notifications to KVScheduler: transaction queue is full" loc="descriptor/link_state.go(149)" logger=vpp-ifplugin.interface-link-state
time="2019-05-24 07:57:22.37055" level=error msg="failed to send notifications to KVScheduler: transaction queue is full" loc="descriptor/link_state.go(149)" logger=vpp-ifplugin.interface-link-state
time="2019-05-24 07:57:22.37060" level=error msg="failed to send notifications to KVScheduler: transaction queue is full" loc="descriptor/link_state.go(149)" logger=vpp-ifplugin.interface-link-state
time="2019-05-24 07:57:22.37066" level=error msg="failed to send notifications to KVScheduler: transaction queue is full" loc="descriptor/link_state.go(149)" logger=vpp-ifplugin.interface-link-state
time="2019-05-24 07:57:22.37071" level=error msg="failed to send notifications to KVScheduler: transaction queue is full" loc="descriptor/link_state.go(149)" logger=vpp-ifplugin.interface-link-state
time="2019-05-24 07:57:22.37077" level=error msg="failed to send notifications to KVScheduler: transaction queue is full" loc="descriptor/link_state.go(149)" logger=vpp-ifplugin.interface-link-state
time="2019-05-24 07:57:22.37083" level=error msg="failed to send notifications to KVScheduler: transaction queue is full" loc="descriptor/link_state.go(149)" logger=vpp-ifplugin.interface-link-state
time="2019-05-24 07:57:22.37088" level=error msg="failed to send notifications to KVScheduler: transaction queue is full" loc="descriptor/link_state.go(149)" logger=vpp-ifplugin.interface-link-state
time="2019-05-24 07:57:22.37094" level=error msg="failed to send notifications to KVScheduler: transaction queue is full" loc="descriptor/link_state.go(149)" logger=vpp-ifplugin.interface-link-state
time="2019-05-24 07:57:22.37100" level=error msg="failed to send notifications to KVScheduler: transaction queue is full" loc="descriptor/link_state.go(149)" logger=vpp-ifplugin.interface-link-state
time="2019-05-24 08:04:33.44595" level=error msg="VPPApiError: VLAN subif already exists (-56)" loc="descriptor/interface_crud.go(122)" logger=vpp-ifplugin.if-descriptor
time="2019-05-24 08:04:33.74587" level=error msg="Transaction #956 finished with 1 errors" loc="orchestrator/dispatcher.go(119)" logger=orchestrator.dispatcher
time="2019-05-24 08:04:33.74596" level=error msg="KeyErrors: [config/vpp/v2/interfaces/sub103101 (CREATE): VPPApiError: VLAN subif already exists (-56)]" loc="syncbase/done.go(40)" logger=defaultLogger
time="2019-05-24 08:04:34.74978" level=error msg="VPPApiError: VLAN subif already exists (-56)" loc="descriptor/interface_crud.go(122)" logger=vpp-ifplugin.if-descriptor
time="2019-05-24 08:04:35.10796" level=error msg="KeyErrors: [config/vpp/v2/interfaces/sub103101 (CREATE): VPPApiError: VLAN subif already exists (-56)]" loc="kvscheduler/txn_process.go(403)" logger=kvscheduler
time="2019-05-24 08:04:37.11340" level=error msg="VPPApiError: VLAN subif already exists (-56)" loc="descriptor/interface_crud.go(122)" logger=vpp-ifplugin.if-descriptor
time="2019-05-24 08:04:37.65665" level=error msg="KeyErrors: [config/vpp/v2/interfaces/sub103101 (CREATE): VPPApiError: VLAN subif already exists (-56)]" loc="kvscheduler/txn_process.go(403)" logger=kvscheduler
time="2019-05-24 08:04:41.66086" level=error msg="VPPApiError: VLAN subif already exists (-56)" loc="descriptor/interface_crud.go(122)" logger=vpp-ifplugin.if-descriptor
time="2019-05-24 08:04:42.07278" level=error msg="KeyErrors: [config/vpp/v2/interfaces/sub103101 (CREATE): VPPApiError: VLAN subif already exists (-56)]" loc="kvscheduler/txn_process.go(403)" logger=kvscheduler

and the KVScheduler metrics (retrieved from http://localhost:9191/scheduler/stats) contain:

{
  "TxnStats": {
    "TotalProcessed": 1988,
    "OperationCount": {
      "CREATE": 3892,
      "DELETE": 294,
      "UPDATE": 0
    },
    "ValueStateCount": {
      "CONFIGURED": 2381,
      "DISCOVERED": 0,
      "FAILED": 1,
      "INVALID": 4,
      "MISSING": 0,
      "NONEXISTENT": 0,
      "OBTAINED": 1010,
      "PENDING": 493,
      "REMOVED": 294,
      "RETRYING": 3,
      "UNIMPLEMENTED": 0
    },
    "ErrorCount": 8,
    "Methods": [ ... ],
}

Inconsistent Linux interface state after x-connection update

Description

We are observing inconsistent behavior of vpp-agent when trying to create and update cross-connection between two pods. We create and then update two x-connected kernel interfaces (src and dst) via grpc interface. Most of the times only one interface gets updated. The issue is intermittent.

Steps to reproduce

We have 3 pods: a dataplane (ligato/vpp-agent 1.8.1) and two alpine pods as peers (source-pod and destination-pod).

Steps

Create a cross-connection between two pods (agent logs)
Ping dest pod from source pod and vice-versa
Update a cross-connection: change interface name and ip address (agent logs)
Ping dest pod from source pod and vice-versa

Expected

At step 4 we expect source-pod and destination-pod to have if2_src and if2_dst interfaces respectively configured as specified in the update request.

Actual

Destination-pod has if1_dst interface with old ip address (as if step 3 was not executed at all).

Details

Here is some details collected after step 3:
Source-pod ip addr output.
Destination-pod ip addr output
Dataplane-pod ip addr output
Dataplane-pod vppctl sh int output

All requests are made via grpc DataChangeService.Put method
Vpp-agent version is 1.8.1

API for VLXAN-GPE

Network Service Mesh needs support for VXLAN-GPE in order to support pure L3 payloads.

VXLAN-GPE, in contrast to VXLAN, has a 'Next Protocol Field' allowing it to encap more than just an Ethernet frame, specifically it can encap and IP packet.

VXLAN-GPE CLI docs - note specifically the

[next-ip4][next-ip6][next-ethernet][next-nsh]

parameters that select the next protocol field value.

This corresponds to the API field protocol

VPP received SIGBUS while running k8s/perf-demo/ scenarios in VM

Hi,

I am using VMWare VM to bring up the scenario4 of VPP agent.
I followed all the steps that are given in README.md, but I am getting below error:

2018-05-18 17:24:01,048 DEBG 'vpp' stdout output:
load_one_plugin:184: Loaded plugin: kubeproxy_plugin.so (kube-proxy data plane)

2018-05-18 17:24:01,048 DEBG 'agent' stdout output:
time="2018-05-18 17:24:01.04781" level=info msg="plugin PrometheusRPC: Init took 219.266µs" durationInNs=219266 loc="core/agent_core.go(295)" logger=agentcore
time="2018-05-18 17:24:01.04792" level=info msg="plugin GRPC: Init took 77.503µs" durationInNs=77503 loc="core/agent_core.go(295)" logger=agentcore
time="2018-05-18 17:24:01.04800" level=info msg="plugin IfStatePub: Init took 401ns" durationInNs=401 loc="core/agent_core.go(295)" logger=agentcore

2018-05-18 17:24:01,048 DEBG 'vpp' stdout output:
load_one_plugin:184: Loaded plugin: l2e_plugin.so (L2 Emulation)
load_one_plugin:184: Loaded plugin: lb_plugin.so (Load Balancer)

2018-05-18 17:24:01,048 DEBG 'vpp' stdout output:
load_one_plugin:184: Loaded plugin: libsixrd_plugin.so (IPv6 Rapid Deployment on IPv4 Infrastructure (RFC5969))
load_one_plugin:184: Loaded plugin: memif_plugin.so (Packet Memory Interface (experimetal))

2018-05-18 17:24:01,049 DEBG 'vpp' stdout output:
load_one_plugin:184: Loaded plugin: nat_plugin.so (Network Address Translation)

2018-05-18 17:24:01,049 DEBG 'vpp' stdout output:
load_one_plugin:184: Loaded plugin: pppoe_plugin.so (PPPoE)

2018-05-18 17:24:01,050 DEBG 'vpp' stdout output:
load_one_plugin:184: Loaded plugin: stn_plugin.so (VPP Steals the NIC for Container integration)

2018-05-18 17:24:01,068 DEBG 'vpp' stdout output:
/usr/bin/vpp[12]: unix_physmem_region_alloc:231: physmem page for region 'buffers' allocated on the wrong numa node (requested 0 actual 4294967294)

2018-05-18 17:24:01,068 DEBG 'vpp' stdout output:
/usr/bin/vpp[12]: received signal SIGBUS, PC 0x7f9ad03b7a5e

2018-05-18 17:24:01,241 DEBG fd 9 closed, stopped monitoring <POutputDispatcher at 139955335379352 for <Subprocess at 139955336328472 with name vpp in state STARTING> (stdout)>
2018-05-18 17:24:01,241 INFO exited: vpp (terminated by SIGABRT (core dumped); not expected)
2018-05-18 17:24:01,241 DEBG received SIGCLD indicating a child quit
2018-05-18 17:24:02,243 INFO success: vpp_or_agent_not_running entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-05-18 17:24:02,245 INFO spawned: 'vpp' with pid 23
2018-05-18 17:24:02,245 INFO success: agent entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-05-18 17:24:02,250 DEBG 'vpp' stdout output:
vlib_plugin_early_init:356: plugin path /usr/lib/vpp_plugins

2018-05-18 17:24:02,251 DEBG 'vpp' stdout output:
load_one_plugin:184: Loaded plugin: acl_plugin.so (Access Control Lists)

Is there any other step(s), that I need to follow?

vpp crash when using vppagent 1.8

We are seeing a vpp crash in the logs as we try to scale test NSM.

Could you offer any insight or suggestions as to other information to gather (and how) so we can debug this issue?

We are seeing this at a scale of fewer than 10 'L2/L3 Connections', so the number of tapv2 interfaces is fairly small.

Fix build for VPP 19.01

This is related to GoVPP's wrapper package vppapiclient that uses name_vector field directly. Needs to be fixed in GoVPP first.

fix in GoVPP
update GoVPP in vpp-agent

plugins/vpp/ifplugin/interface_config.go:LogError obscures all line number origins of errors

Because:

https://github.com/ligato/vpp-agent/blob/v1.7/plugins/vpp/ifplugin/interface_config.go#L1024

Does the logging in a central function (LogError) without correcting for the fact that the line number we care about is the point of origin of the error rather than the place we log it, it obscures the source of non go-errors errors.

A good solution would be to ensure we wrap all non go-errors errors in a go-error at their point of origin, rather than maskign the info.

Deprecate old tools

Following tools (under cmd/) should be deprecated:

remove agentctl (v1)
rename agentctl2 to agentctl
remove vpp-agent-ctl

Problem with syncing between vpp and vpp agent

I created an interface in VPP through VPPCTL with the following command

vpp# create host-interface name host1out
vpp# sh int
Name                  Idx    State  MTU (L3/IP4/IP6/MPLS)     Counter          Count
host-host1out      1       down         9000/0/0/0
local0                   0      down          0/0/0/0

When I tried to dump the interface through REST, it only showed the interface local0 but not host-host1out.

test@2030023305:~/go/src/github.com/ligato/vpp-agent$ curl -X GET http://0.0.0.0:9191/dump/vpp/v2/interfaces 
{ 
 "0": { "interface": { "type": "SOFTWARE_LOOPBACK" }, 
 "interface_meta": { "sw_if_index": 0, "sub_sw_if_index": 0, "l2_address": "", "internal_name": "local0", "admin_state": 0, "link_state": 0, "link_duplex": 0, "link_mtu": 0, "link_speed": 0, "sub_id": 0, "tag": "", "dhcp": null, "vrf_ipv4": 0, "vrf_ipv6": 0, "pci": 0 } } }

When I ran the VPP-agent, it showed warning for the resync plugin. Is there some issue with the synchronization between the VPP and VPP agent? Am I missing some something?
Please, let me know your opinion on this.

test@2030023305:~/go/src/github.com/ligato/vpp-agent$ sudo ./cmd/vpp-agent/vpp-agent
[sudo] password for test:                                     __
_  _____  ___ _______ ____ ____ ___ / /_
| |/ / _ \/ _ /___/ _ '/ _ '/ -_/ _ / __/
|___/ .__/ .__/   \_'_/\_' /\__/_//_\__/  v2.1.1-1-g80401e6
   /_/  /_/           /___/
INFO[0000] Starting agent version: v2.1.1-1-g80401e6     BuildDate="2019-05-28T12:27+05:30" CommitHash=80401e66a0ef370ac61c404590de51d86f78bf4c loc="agent/agent.go(134)" logger=agent
DEBU[0000] Microservice label is set to vpp1             loc="servicelabel/plugin_impl_servicelabel.go(44)" logger=defaultLogger
INFO[0000] ETCD config not found, skip loading this plugin  loc="etcd/plugin_impl_etcd.go(283)" logger=etcd
INFO[0000] Consul config not found, skip loading this plugin  loc="consul/plugin.go(137)" logger=consul
INFO[0000] Redis config not found, skip loading this plugin  loc="redis/plugin_impl_redis.go(123)" logger=red

DEBU[0000] found compatible version:vpp1901              loc="logging/log_api.go(34)" logger=defaultLogger
INFO[0000] VPP version: 19.04.1-rc0~5-g6f05f72           loc="govppmux/plugin_impl_govppmux.go(408)" logger=govpp
INFO[0000] Namespace plugin initialized                  loc="nsplugin/ns_plugin.go(110)" logger=linux-nsplugin
INFO[0000] kafka config not found  - skip loading this plugin  loc="kafka/plugin_impl_kafka.go(72)" logger=kafka
INFO[0000] Interface state updater initialized           loc="ifplugin/interface_state.go(118)" logger=vpp-ifplugin.if-state
INFO[0000] GRPC config not found, skip loading this plugin  loc="grpc/plugin_impl_grpc.go(135)" logger=grpc
INFO[0000] grpc server not available                     loc="orchestrator/orchestrator.go(68)" logger=orchestrator.dispatcher
INFO[0000] Watch starting for 25 registered NB prefixes  loc="orchestrator/orchestrator.go(73)" logger=orchestrator.dispatcher
DEBU[0000] found compatible version:vpp1901              loc="logging/log_api.go(34)" logger=defaultLogger
DEBU[0000] found compatible version: 19.04               loc="logging/log_api.go(34)" logger=defaultLogger
INFO[0000] Listening on http://0.0.0.0:9191              loc="rest/plugin_impl_rest.go(108)" logger=http
INFO[0000] Agent plugin state update.                    lastErr="<nil>" loc="statuscheck/plugin_impl_statuscheck.go(182)" logger=status-check plugin=govpp state=ok
INFO[0000] Starting health http-probe on port 9191       loc="probe/plugin_impl_probe.go(74)" logger=probe
INFO[0000] Unable to register prometheus-probe handler, Prometheus is nil  loc="probe/plugin_impl_probe.go(86)" logger=probe
INFO[0000] Serving /metrics on port 9191                 loc="prometheus/plugin_impl_prometheus.go(87)" logger=prometheus
INFO[0000] Serving /vpp on port 9191                     loc="prometheus/plugin_impl_prometheus.go(87)" logger=prometheus
WARN[0000] No registrations, skipping resync             loc="resync/plugin_impl_resync.go(98)" logger=resync

INFO[0000] Agent started with 37 plugins (took 19ms)     loc="agent/agent.go(179)" logger=agent

DEBU[0000] found compatible version: 19.04               loc="logging/log_api.go(34)" logger=defaultLogger

agentctl: watch command

support watching interfaces and statistics in runtime (as was possible in the agentctlv1)

message acl_add_replace with CRC 3c317936 is not compatible with the VPP we are connected to

Hi,

I'm trying to use the framework with k8s based on the perf-demo code. If I try to add an ip address to a port of the VNF:

/vnf-agent/vnf1/vpp/config/v1/interface/port1
{"name":"port1","type":2,"enabled":true,"mtu":1500,"memif":{"id":1,"socket_filename":"/tmp/memif.sock"},"ip_addresses":["192.168.0.254/24"]}

I get this error from vpp-agent:

time="2017-11-13 12:57:08.01573" level=error msg="unable to retrieve message ID: unkonwn message" msg_crc=3c317936 msg_name="acl_add_replace"
time="2017-11-13 12:57:08.01576" level=error msg="message acl_add_replace with CRC 3c317936 is not compatible with the VPP we are connected to" loc="vppcalls/compatibility_check_vppcalls.go(47)" logger=default-plugins-acl-plugin tag=00000000
time="2017-11-13 12:57:08.01579" level=error msg="plugin VPP: Init error 'message acl_add_replace with CRC 3c317936 is not compatible with the VPP we are connected to', took 498.31µs" durationInNs=498310 loc="core/agent_core.go(206)" logger=defaultLogger tag=00000000

I'm using the docker hub ligato/vpp-agent:latest image.

Do you know why the binary api is not synced with the vpp included in the image ?

Role of kafka

Can somebody will tell me about role of Kafka in this whole processes?

API for DHCP proxy

Found that the binary API generation already exists for the equivalent cli commands https://github.com/ligato/vpp-binapi/blob/master/binapi/dhcp/dhcp.ba.go looking for some pointers on where in the code base to start looking for folding this support in.

Is the same process leveraged for grpc proto support or is this a different set of steps and areas in the code base ?

Thanks!

Using the RestAPI

Is there a way to use the Rest API separately without the full agent? I'd like to run it to interface with my vanilla VPP installation.

Thanks for open-sourcing btw.

vpp-agent strange behaviour on pod killed

We found strange behaviour when we have connection between two pods and one of the pods
dies.

Full logs

case1.txt
case2.txt
case3.txt

Case 1

Pods at same node with no closing connection on pod killed

+======================================================================================================================+
| Transaction #0                                                                          NB Transaction (Full Resync) |
+======================================================================================================================+
  * transaction arguments:
      - seq-num: 0
      - type: NB Transaction, Full Resync
      - values:


2019-06-04 09:51:26,906 DEBG 'agent' stdout output:
o----------------------------------------------------------------------------------------------------------------------o
  * executed operations:
      <NONE>
x----------------------------------------------------------------------------------------------------------------------x
| #0                                                                                                      took 283.7ms |
x----------------------------------------------------------------------------------------------------------------------x

+======================================================================================================================+
| Transaction #1                                                                                        NB Transaction |
+======================================================================================================================+
  * transaction arguments:
      - seq-num: 1
      - type: NB Transaction
      - values:
          - key: config/vpp/acls/v2/acl/NSMmgmtInterfaceACL
            val: { name:"NSMmgmtInterfaceACL" rules:<action:PERMIT ip_rule:<ip:<destination_network:"10.32.0.4/32" source_network:"0.0.0.0/0" > udp:<destination_port_range:<lower_port:4789 upper_port:4789 > source_port_range:<upper_port:65535 > > > > interfaces:<ingress:"mgmt" >  }
          - key: config/vpp/v2/interfaces/mgmt
            val: { name:"mgmt" type:AF_PACKET enabled:true phys_address:"fa:0a:a2:fd:db:98" ip_addresses:"10.32.0.4/12" afpacket:<host_if_name:"eth0" >  }
          - key: config/vpp/v2/route/vrf/0/dst/0.0.0.0/0/gw/10.32.0.1
            val: { type:INTER_VRF dst_network:"0.0.0.0/0" next_hop_addr:"10.32.0.1" outgoing_interface:"mgmt" weight:1  }


2019-06-04 09:51:26,973 DEBG 'agent' stdout output:
time="2019-06-04 09:51:26.97333" level=debug msg="publish add to ifplugin_ifstate{{mgmt false false vpp-interface-index} 0xc0005410e0}" loc="mem/inmemory_name_mapping.go(277)" logger=vpp-ifplugin.if-descriptor

2019-06-04 09:51:26,981 DEBG 'agent' stdout output:
time="2019-06-04 09:51:26.98128" level=debug msg="Interface 1 set to VRF 0" loc="vpp1904/vrf_vppcalls.go(54)" logger=vpp-ifplugin

2019-06-04 09:51:26,987 DEBG 'agent' stdout output:
o----------------------------------------------------------------------------------------------------------------------o
  * executed operations (2019-06-04 09:51:26.912 +0000 UTC -> 2019-06-04 09:51:26.987 +0000 UTC, dur: 75ms):
      1. CREATE:
          - key: config/vpp/acls/v2/acl/NSMmgmtInterfaceACL
          - value: { name:"NSMmgmtInterfaceACL" rules:<action:PERMIT ip_rule:<ip:<destination_network:"10.32.0.4/32" source_network:"0.0.0.0/0" > udp:<destination_port_range:<lower_port:4789 upper_port:4789 > source_port_range:<upper_port:65535 > > > > interfaces:<ingress:"mgmt" >  } 
      2. CREATE:
          - key: config/vpp/v2/interfaces/mgmt
          - value: { name:"mgmt" type:AF_PACKET enabled:true phys_address:"fa:0a:a2:fd:db:98" ip_addresses:"10.32.0.4/12" afpacket:<host_if_name:"eth0" >  } 
      3. CREATE [DERIVED]:
          - key: vpp/acl/NSMmgmtInterfaceACL/interface/ingress/mgmt
          - value: <EMPTY> 
      4. CREATE [DERIVED]:
          - key: vpp/interface/mgmt/vrf/0/ip-version/v4
          - value: <EMPTY> 
      5. CREATE [DERIVED]:
          - key: vpp/interface/mgmt/address/10.32.0.4/12
          - value: <EMPTY> 
      6. CREATE:
          - key: config/vpp/v2/route/vrf/0/dst/0.0.0.0/0/gw/10.32.0.1
          - value: { type:INTER_VRF dst_network:"0.0.0.0/0" next_hop_addr:"10.32.0.1" outgoing_interface:"mgmt" weight:1  } 
x----------------------------------------------------------------------------------------------------------------------x
| #1                                                                                                         took 75ms |
x----------------------------------------------------------------------------------------------------------------------x

2019-06-04 09:51:44,820 DEBG 'vppagent-dataplane' stdout output:
time="2019-06-04T09:51:44Z" level=info msg="Sending DataChange to vppagent: vpp_config: <\n  interfaces: <\n    name: \"SRC-1\"\n    type: TAP\n    enabled: true\n    tap: <\n      version: 2\n    >\n  >\n  interfaces: <\n    name: \"DST-1\"\n    type: TAP\n    enabled: true\n    tap: <\n      version: 2\n    >\n  >\n  xconnect_pairs: <\n    receive_interface: \"SRC-1\"\n    transmit_interface: \"DST-1\"\n  >\n  xconnect_pairs: <\n    receive_interface: \"DST-1\"\n    transmit_interface: \"SRC-1\"\n  >\n>\nlinux_config: <\n  interfaces: <\n    name: \"SRC-1\"\n    type: TAP_TO_VPP\n    namespace: <\n      type: FD\n      reference: \"/proc/6512/ns/net\"\n    >\n    host_if_name: \"nsm0\"\n    enabled: true\n    ip_addresses: \"172.16.1.1/30\"\n    tap: <\n      vpp_tap_if_name: \"SRC-1\"\n    >\n  >\n  interfaces: <\n    name: \"DST-1\"\n    type: TAP_TO_VPP\n    namespace: <\n      type: FD\n      reference: \"/proc/6262/ns/net\"\n    >\n    host_if_name: \"nsmdN5Q1HWVE\"\n    enabled: true\n    ip_addresses: \"172.16.1.2/30\"\n    tap: <\n      vpp_tap_if_name: \"DST-1\"\n    >\n  >\n  routes: <\n    outgoing_interface: \"SRC-1\"\n    scope: LINK\n    dst_network: \"8.8.8.8/30\"\n  >\n>\n"
2019/06/04 09:51:44 ERROR: Reference contains invalid type of SpanReference: {}

2019-06-04 09:51:44,945 DEBG 'agent' stdout output:
o----------------------------------------------------------------------------------------------------------------------o
  * executed operations (2019-06-04 09:51:44.822 +0000 UTC -> 2019-06-04 09:51:44.945 +0000 UTC, dur: 122ms):
      1. CREATE:
          - key: config/vpp/v2/interfaces/DST-1
          - value: { name:"DST-1" type:TAP enabled:true tap:<version:2 >  } 
      2. CREATE:
          - key: config/linux/interfaces/v2/interface/DST-1
          - value: { name:"DST-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/6262/ns/net" > host_if_name:"nsmdN5Q1HWVE" enabled:true ip_addresses:"172.16.1.2/30" tap:<vpp_tap_if_name:"DST-1" >  } 
      3. CREATE [PROPERTY]:
          - key: linux/interface/address/DST-1/172.16.1.2/30
          - value: <EMPTY> 
      4. CREATE [PROPERTY]:
          - key: linux/interface/state/DST-1/UP
          - value: <EMPTY> 
      5. CREATE:
          - key: config/vpp/v2/interfaces/SRC-1
          - value: { name:"SRC-1" type:TAP enabled:true tap:<version:2 >  } 
      6. CREATE:
          - key: config/linux/interfaces/v2/interface/SRC-1
          - value: { name:"SRC-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/6512/ns/net" > host_if_name:"nsm0" enabled:true ip_addresses:"172.16.1.1/30" tap:<vpp_tap_if_name:"SRC-1" >  } 
      7. CREATE [PROPERTY]:
          - key: linux/interface/address/SRC-1/172.16.1.1/30
          - value: <EMPTY> 
      8. CREATE [PROPERTY]:
          - key: linux/interface/state/SRC-1/UP
          - value: <EMPTY> 
      9. CREATE:
          - key: config/linux/l3/v2/route/8.8.8.8/30/SRC-1
          - value: { outgoing_interface:"SRC-1" scope:LINK dst_network:"8.8.8.8/30"  } 
      10. CREATE [PROPERTY]:
          - key: linux/link-local-route/SRC-1/8.8.8.8/30
          - value: <EMPTY> 
      11. CREATE:
          - key: config/vpp/l2/v2/xconnect/DST-1
          - value: { receive_interface:"DST-1" transmit_interface:"SRC-1"  } 
      12. CREATE:
          - key: config/vpp/l2/v2/xconnect/SRC-1
          - value: { receive_interface:"SRC-1" transmit_interface:"DST-1"  } 
x----------------------------------------------------------------------------------------------------------------------x
| #2                                                                                                      took 122.4ms |
x----------------------------------------------------------------------------------------------------------------------x
2019-06-04 09:51:54,223 DEBG 'vppagent-dataplane' stdout output:
time="2019-06-04T09:51:54Z" level=info msg="Sending DataChange to vppagent: vpp_config: <\n  interfaces: <\n    name: \"SRC-1\"\n    type: TAP\n    enabled: true\n    tap: <\n      version: 2\n    >\n  >\n  interfaces: <\n    name: \"DST-1\"\n    type: TAP\n    enabled: true\n    tap: <\n      version: 2\n    >\n  >\n  xconnect_pairs: <\n    receive_interface: \"SRC-1\"\n    transmit_interface: \"DST-1\"\n  >\n  xconnect_pairs: <\n    receive_interface: \"DST-1\"\n    transmit_interface: \"SRC-1\"\n  >\n>\nlinux_config: <\n  interfaces: <\n    name: \"SRC-1\"\n    type: TAP_TO_VPP\n    namespace: <\n      type: FD\n      reference: \"/proc/6512/ns/net\"\n    >\n    host_if_name: \"nsm0\"\n    enabled: true\n    ip_addresses: \"172.16.1.1/30\"\n    tap: <\n      vpp_tap_if_name: \"SRC-1\"\n    >\n  >\n  interfaces: <\n    name: \"DST-1\"\n    type: TAP_TO_VPP\n    namespace: <\n      type: FD\n      reference: \"/proc/6874/ns/net\"\n    >\n    host_if_name: \"nsmtm44rc5N\"\n    enabled: true\n    ip_addresses: \"172.16.1.2/30\"\n    tap: <\n      vpp_tap_if_name: \"DST-1\"\n    >\n  >\n  routes: <\n    outgoing_interface: \"SRC-1\"\n    scope: LINK\n    dst_network: \"8.8.8.8/30\"\n  >\n>\n"



+======================================================================================================================+
| Transaction #3                                                                                        NB Transaction |
+======================================================================================================================+
  * transaction arguments:
      - seq-num: 3
      - type: NB Transaction
      - values:
          - key: config/linux/interfaces/v2/interface/DST-1
            val: { name:"DST-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/6874/ns/net" > host_if_name:"nsmtm44rc5N" enabled:true ip_addresses:"172.16.1.2/30" tap:<vpp_tap_if_name:"DST-1" >  }
          - key: config/linux/interfaces/v2/interface/SRC-1
            val: { name:"SRC-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/6512/ns/net" > host_if_name:"nsm0" enabled:true ip_addresses:"172.16.1.1/30" tap:<vpp_tap_if_name:"SRC-1" >  }
          - key: config/linux/l3/v2/route/8.8.8.8/30/SRC-1
            val: { outgoing_interface:"SRC-1" scope:LINK dst_network:"8.8.8.8/30"  }
          - key: config/vpp/l2/v2/xconnect/DST-1
            val: { receive_interface:"DST-1" transmit_interface:"SRC-1"  }
          - key: config/vpp/l2/v2/xconnect/SRC-1
            val: { receive_interface:"SRC-1" transmit_interface:"DST-1"  }
          - key: config/vpp/v2/interfaces/DST-1
            val: { name:"DST-1" type:TAP enabled:true tap:<version:2 >  }
          - key: config/vpp/v2/interfaces/SRC-1
            val: { name:"SRC-1" type:TAP enabled:true tap:<version:2 >  }

time="2019-06-04 09:51:54.22526" level=error msg="failed to get or create namespace (type:FD reference:\"/proc/6262/ns/net\" ): failed to get file /proc/6262/ns/net from path: no such file or directory" loc="descriptor/interface.go(434)" logger=linux-ifplugin.if-descriptor
o----------------------------------------------------------------------------------------------------------------------o
  * executed operations (2019-06-04 09:51:54.225 +0000 UTC -> 2019-06-04 09:51:54.225 +0000 UTC, dur: 1ms):
      1. DELETE [PROPERTY RECREATE]:
          - key: linux/interface/address/DST-1/172.16.1.2/30
          - value: <EMPTY> 
      2. DELETE [PROPERTY RECREATE]:
          - key: linux/interface/state/DST-1/UP
          - value: <EMPTY> 
      3. DELETE [RECREATE FAILED]:
          - key: config/linux/interfaces/v2/interface/DST-1
          - value: { name:"DST-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/6262/ns/net" > host_if_name:"nsmdN5Q1HWVE" enabled:true ip_addresses:"172.16.1.2/30" tap:<vpp_tap_if_name:"DST-1" >  } 
          - **error: failed to get or create namespace (type:FD reference:"/proc/6262/ns/net" ): failed to get file /proc/6262/ns/net from path: no such file or directory**
x----------------------------------------------------------------------------------------------------------------------x
| #3                                                                                                        took 500µs |
x----------------------------------------------------------------------------------------------------------------------x
2019-06-04 09:51:54,897 DEBG 'vppagent-dataplane' stdout output:
time="2019-06-04T09:51:54Z" level=info msg="Sending DataChange to vppagent: vpp_config: <\n  interfaces: <\n    name: \"SRC-1\"\n    type: TAP\n    enabled: true\n    tap: <\n      version: 2\n    >\n  >\n  interfaces: <\n    name: \"DST-1\"\n    type: TAP\n    enabled: true\n    tap: <\n      version: 2\n    >\n  >\n  xconnect_pairs: <\n    receive_interface: \"SRC-1\"\n    transmit_interface: \"DST-1\"\n  >\n  xconnect_pairs: <\n    receive_interface: \"DST-1\"\n    transmit_interface: \"SRC-1\"\n  >\n>\nlinux_config: <\n  interfaces: <\n    name: \"SRC-1\"\n    type: TAP_TO_VPP\n    namespace: <\n      type: FD\n      reference: \"/proc/6512/ns/net\"\n    >\n    host_if_name: \"nsm0\"\n    enabled: true\n    ip_addresses: \"172.16.1.1/30\"\n    tap: <\n      vpp_tap_if_name: \"SRC-1\"\n    >\n  >\n  interfaces: <\n    name: \"DST-1\"\n    type: TAP_TO_VPP\n    namespace: <\n      type: FD\n      reference: \"/proc/6874/ns/net\"\n    >\n    host_if_name: \"nsmtm44rc5N\"\n    enabled: true\n    ip_addresses: \"172.16.1.2/30\"\n    tap: <\n      vpp_tap_if_name: \"DST-1\"\n    >\n  >\n  routes: <\n    outgoing_interface: \"SRC-1\"\n    scope: LINK\n    dst_network: \"8.8.8.8/30\"\n  >\n>\n"

+======================================================================================================================+
| Transaction #4                                                                                        NB Transaction |
+======================================================================================================================+
  * transaction arguments:
      - seq-num: 4
      - type: NB Transaction
      - values:
          - key: config/linux/interfaces/v2/interface/DST-1
            val: { name:"DST-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/6874/ns/net" > host_if_name:"nsmtm44rc5N" enabled:true ip_addresses:"172.16.1.2/30" tap:<vpp_tap_if_name:"DST-1" >  }
          - key: config/linux/interfaces/v2/interface/SRC-1
            val: { name:"SRC-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/6512/ns/net" > host_if_name:"nsm0" enabled:true ip_addresses:"172.16.1.1/30" tap:<vpp_tap_if_name:"SRC-1" >  }
          - key: config/linux/l3/v2/route/8.8.8.8/30/SRC-1
            val: { outgoing_interface:"SRC-1" scope:LINK dst_network:"8.8.8.8/30"  }
          - key: config/vpp/l2/v2/xconnect/DST-1
            val: { receive_interface:"DST-1" transmit_interface:"SRC-1"  }
          - key: config/vpp/l2/v2/xconnect/SRC-1
            val: { receive_interface:"SRC-1" transmit_interface:"DST-1"  }
          - key: config/vpp/v2/interfaces/DST-1
            val: { name:"DST-1" type:TAP enabled:true tap:<version:2 >  }
          - key: config/vpp/v2/interfaces/SRC-1
            val: { name:"SRC-1" type:TAP enabled:true tap:<version:2 >  }

time="2019-06-04 09:51:54.89912" level=error msg="Link not found" loc="descriptor/interface_tap.go(50)" logger=linux-ifplugin.if-descriptor
o----------------------------------------------------------------------------------------------------------------------o
  * executed operations (2019-06-04 09:51:54.898 +0000 UTC -> 2019-06-04 09:51:54.899 +0000 UTC, dur: 1ms):
      1. CREATE [STILL-FAILING]:
          - key: config/linux/interfaces/v2/interface/DST-1
          - value: { name:"DST-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/6874/ns/net" > host_if_name:"nsmtm44rc5N" enabled:true ip_addresses:"172.16.1.2/30" tap:<vpp_tap_if_name:"DST-1" >  } 
          - prev-error: **failed to get or create namespace (type:FD reference:"/proc/6262/ns/net" ): failed to get file /proc/6262/ns/net from path: no such file or directory**
          - error: Link not found
x----------------------------------------------------------------------------------------------------------------------x
| #4                                                                                                          took 1ms |
x----------------------------------------------------------------------------------------------------------------------x

Actual: request has error
Expected:?

Case 2

Pods at different nodes with no closing connection on pod killed

+======================================================================================================================+
| Transaction #0                                                                          NB Transaction (Full Resync) |
+======================================================================================================================+
  * transaction arguments:
      - seq-num: 0
      - type: NB Transaction, Full Resync
      - values:

o----------------------------------------------------------------------------------------------------------------------o
  * executed operations:
      <NONE>
x----------------------------------------------------------------------------------------------------------------------x
| #0                                                                                                      took 248.1ms |
x----------------------------------------------------------------------------------------------------------------------x
o----------------------------------------------------------------------------------------------------------------------o
  * executed operations (2019-06-05 04:40:04.618 +0000 UTC -> 2019-06-05 04:40:04.692 +0000 UTC, dur: 73ms):
      1. CREATE:
          - key: config/vpp/acls/v2/acl/NSMmgmtInterfaceACL
          - value: { name:"NSMmgmtInterfaceACL" rules:<action:PERMIT ip_rule:<ip:<destination_network:"10.32.0.5/32" source_network:"0.0.0.0/0" > udp:<destination_port_range:<lower_port:4789 upper_port:4789 > source_port_range:<upper_port:65535 > > > > interfaces:<ingress:"mgmt" >  } 
      2. CREATE:
          - key: config/vpp/v2/interfaces/mgmt
          - value: { name:"mgmt" type:AF_PACKET enabled:true phys_address:"5e:bd:a1:38:29:c8" ip_addresses:"10.32.0.5/12" afpacket:<host_if_name:"eth0" >  } 
      3. CREATE [DERIVED]:
          - key: vpp/acl/NSMmgmtInterfaceACL/interface/ingress/mgmt
          - value: <EMPTY> 
      4. CREATE [DERIVED]:
          - key: vpp/interface/mgmt/vrf/0/ip-version/v4
          - value: <EMPTY> 
      5. CREATE [DERIVED]:
          - key: vpp/interface/mgmt/address/10.32.0.5/12
          - value: <EMPTY> 
      6. CREATE:
          - key: config/vpp/v2/route/vrf/0/dst/0.0.0.0/0/gw/10.32.0.1
          - value: { type:INTER_VRF dst_network:"0.0.0.0/0" next_hop_addr:"10.32.0.1" outgoing_interface:"mgmt" weight:1  } 
x----------------------------------------------------------------------------------------------------------------------x
| #1                                                                                                       took 73.3ms |
x----------------------------------------------------------------------------------------------------------------------x

+======================================================================================================================+
| Transaction #2                                                                                        NB Transaction |
+======================================================================================================================+
  * transaction arguments:
      - seq-num: 2
      - type: NB Transaction
      - values:
          - key: config/linux/interfaces/v2/interface/DST-1
            val: { name:"DST-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/3189/ns/net" > host_if_name:"nsmSj3LWXchy" enabled:true ip_addresses:"172.16.1.2/30" tap:<vpp_tap_if_name:"DST-1" >  }
          - key: config/linux/interfaces/v2/interface/SRC-1
            val: { name:"SRC-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/3450/ns/net" > host_if_name:"nsm0" enabled:true ip_addresses:"172.16.1.1/30" tap:<vpp_tap_if_name:"SRC-1" >  }
          - key: config/linux/l3/v2/route/8.8.8.8/30/SRC-1
            val: { outgoing_interface:"SRC-1" scope:LINK dst_network:"8.8.8.8/30"  }
          - key: config/vpp/l2/v2/xconnect/DST-1
            val: { receive_interface:"DST-1" transmit_interface:"SRC-1"  }
          - key: config/vpp/l2/v2/xconnect/SRC-1
            val: { receive_interface:"SRC-1" transmit_interface:"DST-1"  }
          - key: config/vpp/v2/interfaces/DST-1
            val: { name:"DST-1" type:TAP enabled:true tap:<version:2 >  }
          - key: config/vpp/v2/interfaces/SRC-1
            val: { name:"SRC-1" type:TAP enabled:true tap:<version:2 >  }


2019-06-05 04:40:12,960 DEBG 'agent' stdout output:
time="2019-06-05 04:40:12.95892" level=debug msg="publish add to ifplugin_ifstate{{DST-1 false false vpp-interface-index} 0xc00045d830}" loc="mem/inmemory_name_mapping.go(277)" logger=vpp-ifplugin.if-descriptor

2019-06-05 04:40:13,027 DEBG 'agent' stdout output:
time="2019-06-05 04:40:13.02704" level=debug msg="publish add to ifplugin_ifstate{{SRC-1 false false vpp-interface-index} 0xc00045d470}" loc="mem/inmemory_name_mapping.go(277)" logger=vpp-ifplugin.if-descriptor

2019-06-05 04:40:13,063 DEBG 'agent' stdout output:
o----------------------------------------------------------------------------------------------------------------------o
  * executed operations (2019-06-05 04:40:12.934 +0000 UTC -> 2019-06-05 04:40:13.063 +0000 UTC, dur: 129ms):
      1. CREATE:
          - key: config/vpp/v2/interfaces/DST-1
          - value: { name:"DST-1" type:TAP enabled:true tap:<version:2 >  } 
      2. CREATE:
          - key: config/linux/interfaces/v2/interface/DST-1
          - value: { name:"DST-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/3189/ns/net" > host_if_name:"nsmSj3LWXchy" enabled:true ip_addresses:"172.16.1.2/30" tap:<vpp_tap_if_name:"DST-1" >  } 
      3. CREATE [PROPERTY]:
          - key: linux/interface/address/DST-1/172.16.1.2/30
          - value: <EMPTY> 
      4. CREATE [PROPERTY]:
          - key: linux/interface/state/DST-1/UP
          - value: <EMPTY> 
      5. CREATE:
          - key: config/vpp/v2/interfaces/SRC-1
          - value: { name:"SRC-1" type:TAP enabled:true tap:<version:2 >  } 
      6. CREATE:
          - key: config/linux/interfaces/v2/interface/SRC-1
          - value: { name:"SRC-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/3450/ns/net" > host_if_name:"nsm0" enabled:true ip_addresses:"172.16.1.1/30" tap:<vpp_tap_if_name:"SRC-1" >  } 
      7. CREATE [PROPERTY]:
          - key: linux/interface/address/SRC-1/172.16.1.1/30
          - value: <EMPTY> 
      8. CREATE [PROPERTY]:
          - key: linux/interface/state/SRC-1/UP
          - value: <EMPTY> 
      9. CREATE:
          - key: config/linux/l3/v2/route/8.8.8.8/30/SRC-1
          - value: { outgoing_interface:"SRC-1" scope:LINK dst_network:"8.8.8.8/30"  } 
      10. CREATE [PROPERTY]:
          - key: linux/link-local-route/SRC-1/8.8.8.8/30
          - value: <EMPTY> 
      11. CREATE:
          - key: config/vpp/l2/v2/xconnect/DST-1
          - value: { receive_interface:"DST-1" transmit_interface:"SRC-1"  } 
      12. CREATE:
          - key: config/vpp/l2/v2/xconnect/SRC-1
          - value: { receive_interface:"SRC-1" transmit_interface:"DST-1"  } 
x----------------------------------------------------------------------------------------------------------------------x
| #2                                                                                                      took 128.6ms |
x----------------------------------------------------------------------------------------------------------------------x
2019-06-05 04:40:22,441 DEBG 'vppagent-dataplane' stdout output:
time="2019-06-05T04:40:22Z" level=info msg="Sending DataChange to vppagent: vpp_config: <\n  interfaces: <\n    name: \"SRC-1\"\n    type: TAP\n    enabled: true\n    tap: <\n      version: 2\n    >\n  >\n  interfaces: <\n    name: \"DST-1\"\n    type: TAP\n    enabled: true\n    tap: <\n      version: 2\n    >\n  >\n  xconnect_pairs: <\n    receive_interface: \"SRC-1\"\n    transmit_interface: \"DST-1\"\n  >\n  xconnect_pairs: <\n    receive_interface: \"DST-1\"\n    transmit_interface: \"SRC-1\"\n  >\n>\nlinux_config: <\n  interfaces: <\n    name: \"SRC-1\"\n    type: TAP_TO_VPP\n    namespace: <\n      type: FD\n      reference: \"/proc/3450/ns/net\"\n    >\n    host_if_name: \"nsm0\"\n    enabled: true\n    ip_addresses: \"172.16.1.1/30\"\n    tap: <\n      vpp_tap_if_name: \"SRC-1\"\n    >\n  >\n  interfaces: <\n    name: \"DST-1\"\n    type: TAP_TO_VPP\n    namespace: <\n      type: FD\n    >\n    host_if_name: \"nsmSj3LWXchy\"\n    enabled: true\n    ip_addresses: \"172.16.1.2/30\"\n    tap: <\n      vpp_tap_if_name: \"DST-1\"\n    >\n  >\n  routes: <\n    outgoing_interface: \"SRC-1\"\n    scope: LINK\n    dst_network: \"8.8.8.8/30\"\n  >\n>\n"
+======================================================================================================================+
| Transaction #3                                                                                        NB Transaction |
+======================================================================================================================+
  * transaction arguments:
      - seq-num: 3
      - type: NB Transaction
      - values:
          - key: config/linux/interfaces/v2/interface/DST-1
            val: <NIL>
          - key: config/linux/interfaces/v2/interface/SRC-1
            val: <NIL>
          - key: config/linux/l3/v2/route/8.8.8.8/30/SRC-1
            val: <NIL>
          - key: config/vpp/l2/v2/xconnect/DST-1
            val: <NIL>
          - key: config/vpp/l2/v2/xconnect/SRC-1
            val: <NIL>
          - key: config/vpp/v2/interfaces/DST-1
            val: <NIL>
          - key: config/vpp/v2/interfaces/SRC-1
            val: <NIL>

time="2019-06-05 04:40:22.44312" level=error msg="failed to get or create namespace (type:FD reference:\"/proc/3189/ns/net\" ): failed to get file /proc/3189/ns/net from path: no such file or directory" loc="descriptor/interface.go(434)" logger=linux-ifplugin.if-descriptor

2019-06-05 04:40:22,503 DEBG 'agent' stdout output:
time="2019-06-05 04:40:22.50301" level=debug msg="publish del to ifplugin_ifstate{{DST-1 true false vpp-interface-index} 0xc00045d830}" loc="mem/inmemory_name_mapping.go(323)" logger=vpp-ifplugin.if-descriptor

2019-06-05 04:40:22,543 DEBG 'agent' stdout output:
time="2019-06-05 04:40:22.54359" level=debug msg="publish del to ifplugin_ifstate{{SRC-1 true false vpp-interface-index} 0xc00045d470}" loc="mem/inmemory_name_mapping.go(323)" logger=vpp-ifplugin.if-descriptor

2019-06-05 04:40:22,544 DEBG 'agent' stdout output:
o----------------------------------------------------------------------------------------------------------------------o
  * executed operations (2019-06-05 04:40:22.443 +0000 UTC -> 2019-06-05 04:40:22.544 +0000 UTC, dur: 101ms):
      1. DELETE [PROPERTY]:
          - key: linux/interface/address/DST-1/172.16.1.2/30
          - value: <EMPTY> 
      2. DELETE [PROPERTY]:
          - key: linux/interface/state/DST-1/UP
          - value: <EMPTY> 
      3. DELETE [FAILED]:
          - key: config/linux/interfaces/v2/interface/DST-1
          - value: { name:"DST-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/3189/ns/net" > host_if_name:"nsmSj3LWXchy" enabled:true ip_addresses:"172.16.1.2/30" tap:<vpp_tap_if_name:"DST-1" >  } 
          - error: failed to get or create namespace (type:FD reference:"/proc/3189/ns/net" ): failed to get file /proc/3189/ns/net from path: no such file or directory
      4. DELETE [PROPERTY]:
          - key: linux/link-local-route/SRC-1/8.8.8.8/30
          - value: <EMPTY> 
      5. DELETE:
          - key: config/linux/l3/v2/route/8.8.8.8/30/SRC-1
          - value: { outgoing_interface:"SRC-1" scope:LINK dst_network:"8.8.8.8/30"  } 
      6. DELETE [PROPERTY]:
          - key: linux/interface/address/SRC-1/172.16.1.1/30
          - value: <EMPTY> 
      7. DELETE [PROPERTY]:
          - key: linux/interface/state/SRC-1/UP
          - value: <EMPTY> 
      8. DELETE:
          - key: config/linux/interfaces/v2/interface/SRC-1
          - value: { name:"SRC-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/3450/ns/net" > host_if_name:"nsm0" enabled:true ip_addresses:"172.16.1.1/30" tap:<vpp_tap_if_name:"SRC-1" >  } 
      9. DELETE:
          - key: config/vpp/l2/v2/xconnect/DST-1
          - value: { receive_interface:"DST-1" transmit_interface:"SRC-1"  } 
      10. DELETE:
          - key: config/vpp/l2/v2/xconnect/SRC-1
          - value: { receive_interface:"SRC-1" transmit_interface:"DST-1"  } 
      11. DELETE:
          - key: config/vpp/v2/interfaces/DST-1
          - value: { name:"DST-1" type:TAP enabled:true tap:<version:2 >  } 
      12. DELETE:
          - key: config/vpp/v2/interfaces/SRC-1
          - value: { name:"SRC-1" type:TAP enabled:true tap:<version:2 >  } 
x----------------------------------------------------------------------------------------------------------------------x
| #3                                                                                                      took 100.7ms |
x----------------------------------------------------------------------------------------------------------------------x
2019-06-05 04:40:22,675 DEBG 'vppagent-dataplane' stdout output:
time="2019-06-05T04:40:22Z" level=info msg="Sending DataChange to vppagent: vpp_config: <\n  interfaces: <\n    name: \"SRC-1\"\n    type: TAP\n    enabled: true\n    tap: <\n      version: 2\n    >\n  >\n  interfaces: <\n    name: \"DST-1\"\n    type: TAP\n    enabled: true\n    tap: <\n      version: 2\n    >\n  >\n  xconnect_pairs: <\n    receive_interface: \"SRC-1\"\n    transmit_interface: \"DST-1\"\n  >\n  xconnect_pairs: <\n    receive_interface: \"DST-1\"\n    transmit_interface: \"SRC-1\"\n  >\n>\nlinux_config: <\n  interfaces: <\n    name: \"SRC-1\"\n    type: TAP_TO_VPP\n    namespace: <\n      type: FD\n      reference: \"/proc/3450/ns/net\"\n    >\n    host_if_name: \"nsm0\"\n    enabled: true\n    ip_addresses: \"172.16.1.1/30\"\n    tap: <\n      vpp_tap_if_name: \"SRC-1\"\n    >\n  >\n  interfaces: <\n    name: \"DST-1\"\n    type: TAP_TO_VPP\n    namespace: <\n      type: FD\n      reference: \"/proc/3854/ns/net\"\n    >\n    host_if_name: \"nsmZ5yEVPGTj\"\n    enabled: true\n    ip_addresses: \"172.16.1.2/30\"\n    tap: <\n      vpp_tap_if_name: \"DST-1\"\n    >\n  >\n  routes: <\n    outgoing_interface: \"SRC-1\"\n    scope: LINK\n    dst_network: \"8.8.8.8/30\"\n  >\n>\n"
+======================================================================================================================+
| Transaction #4                                                                                        NB Transaction |
+======================================================================================================================+
  * transaction arguments:
      - seq-num: 4
      - type: NB Transaction
      - values:
          - key: config/linux/interfaces/v2/interface/DST-1
            val: { name:"DST-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/3854/ns/net" > host_if_name:"nsmZ5yEVPGTj" enabled:true ip_addresses:"172.16.1.2/30" tap:<vpp_tap_if_name:"DST-1" >  }
          - key: config/linux/interfaces/v2/interface/SRC-1
            val: { name:"SRC-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/3450/ns/net" > host_if_name:"nsm0" enabled:true ip_addresses:"172.16.1.1/30" tap:<vpp_tap_if_name:"SRC-1" >  }
          - key: config/linux/l3/v2/route/8.8.8.8/30/SRC-1
            val: { outgoing_interface:"SRC-1" scope:LINK dst_network:"8.8.8.8/30"  }
          - key: config/vpp/l2/v2/xconnect/DST-1
            val: { receive_interface:"DST-1" transmit_interface:"SRC-1"  }
          - key: config/vpp/l2/v2/xconnect/SRC-1
            val: { receive_interface:"SRC-1" transmit_interface:"DST-1"  }
          - key: config/vpp/v2/interfaces/DST-1
            val: { name:"DST-1" type:TAP enabled:true tap:<version:2 >  }
          - key: config/vpp/v2/interfaces/SRC-1
            val: { name:"SRC-1" type:TAP enabled:true tap:<version:2 >  }


2019-06-05 04:40:22,706 DEBG 'agent' stdout output:
time="2019-06-05 04:40:22.70575" level=debug msg="publish add to ifplugin_ifstate{{DST-1 false false vpp-interface-index} 0xc00045c630}" loc="mem/inmemory_name_mapping.go(277)" logger=vpp-ifplugin.if-descriptor

2019-06-05 04:40:22,774 DEBG 'agent' stdout output:
time="2019-06-05 04:40:22.77358" level=debug msg="publish add to ifplugin_ifstate{{SRC-1 false false vpp-interface-index} 0xc000229b90}" loc="mem/inmemory_name_mapping.go(277)" logger=vpp-ifplugin.if-descriptor

2019-06-05 04:40:22,814 DEBG 'agent' stdout output:
o----------------------------------------------------------------------------------------------------------------------o
  * executed operations (2019-06-05 04:40:22.677 +0000 UTC -> 2019-06-05 04:40:22.814 +0000 UTC, dur: 137ms):
      1. CREATE:
          - key: config/vpp/v2/interfaces/DST-1
          - value: { name:"DST-1" type:TAP enabled:true tap:<version:2 >  } 
      2. CREATE [FIXED]:
          - key: config/linux/interfaces/v2/interface/DST-1
          - value: { name:"DST-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/3854/ns/net" > host_if_name:"nsmZ5yEVPGTj" enabled:true ip_addresses:"172.16.1.2/30" tap:<vpp_tap_if_name:"DST-1" >  } 
          - prev-error: failed to get or create namespace (type:FD reference:"/proc/3189/ns/net" ): failed to get file /proc/3189/ns/net from path: no such file or directory
      3. CREATE [PROPERTY]:
          - key: linux/interface/address/DST-1/172.16.1.2/30
          - value: <EMPTY> 
      4. CREATE [PROPERTY]:
          - key: linux/interface/state/DST-1/UP
          - value: <EMPTY> 
      5. CREATE:
          - key: config/vpp/v2/interfaces/SRC-1
          - value: { name:"SRC-1" type:TAP enabled:true tap:<version:2 >  } 
      6. CREATE:
          - key: config/linux/interfaces/v2/interface/SRC-1
          - value: { name:"SRC-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/3450/ns/net" > host_if_name:"nsm0" enabled:true ip_addresses:"172.16.1.1/30" tap:<vpp_tap_if_name:"SRC-1" >  } 
      7. CREATE [PROPERTY]:
          - key: linux/interface/address/SRC-1/172.16.1.1/30
          - value: <EMPTY> 
      8. CREATE [PROPERTY]:
          - key: linux/interface/state/SRC-1/UP
          - value: <EMPTY> 
      9. CREATE:
          - key: config/linux/l3/v2/route/8.8.8.8/30/SRC-1
          - value: { outgoing_interface:"SRC-1" scope:LINK dst_network:"8.8.8.8/30"  } 
      10. CREATE [PROPERTY]:
          - key: linux/link-local-route/SRC-1/8.8.8.8/30
          - value: <EMPTY> 
      11. CREATE:
          - key: config/vpp/l2/v2/xconnect/DST-1
          - value: { receive_interface:"DST-1" transmit_interface:"SRC-1"  } 
      12. CREATE:
          - key: config/vpp/l2/v2/xconnect/SRC-1
          - value: { receive_interface:"SRC-1" transmit_interface:"DST-1"  } 
x----------------------------------------------------------------------------------------------------------------------x
| #4                                                                                                      took 136.9ms |
x----------------------------------------------------------------------------------------------------------------------x

Actual: request has not error
Expected:?

Case 3

Pods at same node with closing connection on pod killed.

+======================================================================================================================+
| Transaction #0                                                                          NB Transaction (Full Resync) |
+======================================================================================================================+
  * transaction arguments:
      - seq-num: 0
      - type: NB Transaction, Full Resync
      - values:

o----------------------------------------------------------------------------------------------------------------------o
  * executed operations:
      <NONE>
x----------------------------------------------------------------------------------------------------------------------x
| #0                                                                                                      took 264.4ms |
x----------------------------------------------------------------------------------------------------------------------x
2019-06-06 04:38:50,129 DEBG 'agent' stdout output:
+======================================================================================================================+
| Transaction #1                                                                                        NB Transaction |
+======================================================================================================================+
  * transaction arguments:
      - seq-num: 1
      - type: NB Transaction
      - values:
          - key: config/vpp/acls/v2/acl/NSMmgmtInterfaceACL
            val: { name:"NSMmgmtInterfaceACL" rules:<action:PERMIT ip_rule:<ip:<destination_network:"10.32.0.4/32" source_network:"0.0.0.0/0" > udp:<destination_port_range:<lower_port:4789 upper_port:4789 > source_port_range:<upper_port:65535 > > > > interfaces:<ingress:"mgmt" >  }
          - key: config/vpp/v2/interfaces/mgmt
            val: { name:"mgmt" type:AF_PACKET enabled:true phys_address:"72:84:3d:b9:41:35" ip_addresses:"10.32.0.4/12" afpacket:<host_if_name:"eth0" >  }
          - key: config/vpp/v2/route/vrf/0/dst/0.0.0.0/0/gw/10.32.0.1
            val: { type:INTER_VRF dst_network:"0.0.0.0/0" next_hop_addr:"10.32.0.1" outgoing_interface:"mgmt" weight:1  }


2019-06-06 04:38:50,184 DEBG 'agent' stdout output:
time="2019-06-06 04:38:50.18410" level=debug msg="publish add to ifplugin_ifstate{{mgmt false false vpp-interface-index} 0xc0004409c0}" loc="mem/inmemory_name_mapping.go(277)" logger=vpp-ifplugin.if-descriptor

2019-06-06 04:38:50,193 DEBG 'agent' stdout output:
time="2019-06-06 04:38:50.19352" level=debug msg="Interface 1 set to VRF 0" loc="vpp1904/vrf_vppcalls.go(54)" logger=vpp-ifplugin

2019-06-06 04:38:50,201 DEBG 'vppagent-dataplane' stdout output:
2019/06/06 04:38:50 Reporting span 23eb69cd153b18b1:23eb69cd153b18b1:0:1
time="2019-06-06T04:38:50Z" level=info msg="Metics collector is disabled"

2019-06-06 04:38:50,201 DEBG 'agent' stdout output:
o----------------------------------------------------------------------------------------------------------------------o
  * executed operations (2019-06-06 04:38:50.129 +0000 UTC -> 2019-06-06 04:38:50.2 +0000 UTC, dur: 71ms):
      1. CREATE:
          - key: config/vpp/acls/v2/acl/NSMmgmtInterfaceACL
          - value: { name:"NSMmgmtInterfaceACL" rules:<action:PERMIT ip_rule:<ip:<destination_network:"10.32.0.4/32" source_network:"0.0.0.0/0" > udp:<destination_port_range:<lower_port:4789 upper_port:4789 > source_port_range:<upper_port:65535 > > > > interfaces:<ingress:"mgmt" >  } 
      2. CREATE:
          - key: config/vpp/v2/interfaces/mgmt
          - value: { name:"mgmt" type:AF_PACKET enabled:true phys_address:"72:84:3d:b9:41:35" ip_addresses:"10.32.0.4/12" afpacket:<host_if_name:"eth0" >  } 
      3. CREATE [DERIVED]:
          - key: vpp/acl/NSMmgmtInterfaceACL/interface/ingress/mgmt
          - value: <EMPTY> 
      4. CREATE [DERIVED]:
          - key: vpp/interface/mgmt/vrf/0/ip-version/v4
          - value: <EMPTY> 
      5. CREATE [DERIVED]:
          - key: vpp/interface/mgmt/address/10.32.0.4/12
          - value: <EMPTY> 
      6. CREATE:
          - key: config/vpp/v2/route/vrf/0/dst/0.0.0.0/0/gw/10.32.0.1
          - value: { type:INTER_VRF dst_network:"0.0.0.0/0" next_hop_addr:"10.32.0.1" outgoing_interface:"mgmt" weight:1  } 
x----------------------------------------------------------------------------------------------------------------------x
| #1                                                                                                         took 71ms |
x----------------------------------------------------------------------------------------------------------------------x
2019-06-06 04:39:08,080 DEBG 'vppagent-dataplane' stdout output:
time="2019-06-06T04:39:08Z" level=info msg="Sending DataChange to vppagent: vpp_config: <\n  interfaces: <\n    name: \"SRC-1\"\n    type: TAP\n    enabled: true\n    tap: <\n      version: 2\n    >\n  >\n  interfaces: <\n    name: \"DST-1\"\n    type: TAP\n    enabled: true\n    tap: <\n      version: 2\n    >\n  >\n  xconnect_pairs: <\n    receive_interface: \"SRC-1\"\n    transmit_interface: \"DST-1\"\n  >\n  xconnect_pairs: <\n    receive_interface: \"DST-1\"\n    transmit_interface: \"SRC-1\"\n  >\n>\nlinux_config: <\n  interfaces: <\n    name: \"SRC-1\"\n    type: TAP_TO_VPP\n    namespace: <\n      type: FD\n      reference: \"/proc/26312/ns/net\"\n    >\n    host_if_name: \"nsm0\"\n    enabled: true\n    ip_addresses: \"172.16.1.1/30\"\n    tap: <\n      vpp_tap_if_name: \"SRC-1\"\n    >\n  >\n  interfaces: <\n    name: \"DST-1\"\n    type: TAP_TO_VPP\n    namespace: <\n      type: FD\n      reference: \"/proc/26052/ns/net\"\n    >\n    host_if_name: \"nsmx4UlWGtsv\"\n    enabled: true\n    ip_addresses: \"172.16.1.2/30\"\n    tap: <\n      vpp_tap_if_name: \"DST-1\"\n    >\n  >\n  routes: <\n    outgoing_interface: \"SRC-1\"\n    scope: LINK\n    dst_network: \"8.8.8.8/30\"\n  >\n>\n"
+======================================================================================================================+
| Transaction #2                                                                                        NB Transaction |
+======================================================================================================================+
  * transaction arguments:
      - seq-num: 2
      - type: NB Transaction
      - values:
          - key: config/linux/interfaces/v2/interface/DST-1
            val: { name:"DST-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/26052/ns/net" > host_if_name:"nsmx4UlWGtsv" enabled:true ip_addresses:"172.16.1.2/30" tap:<vpp_tap_if_name:"DST-1" >  }
          - key: config/linux/interfaces/v2/interface/SRC-1
            val: { name:"SRC-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/26312/ns/net" > host_if_name:"nsm0" enabled:true ip_addresses:"172.16.1.1/30" tap:<vpp_tap_if_name:"SRC-1" >  }
          - key: config/linux/l3/v2/route/8.8.8.8/30/SRC-1
            val: { outgoing_interface:"SRC-1" scope:LINK dst_network:"8.8.8.8/30"  }
          - key: config/vpp/l2/v2/xconnect/DST-1
            val: { receive_interface:"DST-1" transmit_interface:"SRC-1"  }
          - key: config/vpp/l2/v2/xconnect/SRC-1
            val: { receive_interface:"SRC-1" transmit_interface:"DST-1"  }
          - key: config/vpp/v2/interfaces/DST-1
            val: { name:"DST-1" type:TAP enabled:true tap:<version:2 >  }
          - key: config/vpp/v2/interfaces/SRC-1
            val: { name:"SRC-1" type:TAP enabled:true tap:<version:2 >  }


2019-06-06 04:39:08,116 DEBG 'agent' stdout output:
time="2019-06-06 04:39:08.11246" level=debug msg="publish add to ifplugin_ifstate{{DST-1 false false vpp-interface-index} 0xc000614630}" loc="mem/inmemory_name_mapping.go(277)" logger=vpp-ifplugin.if-descriptor

2019-06-06 04:39:08,185 DEBG 'agent' stdout output:
time="2019-06-06 04:39:08.18480" level=debug msg="publish add to ifplugin_ifstate{{SRC-1 false false vpp-interface-index} 0xc00042fe00}" loc="mem/inmemory_name_mapping.go(277)" logger=vpp-ifplugin.if-descriptor

2019-06-06 04:39:08,224 DEBG 'agent' stdout output:
o----------------------------------------------------------------------------------------------------------------------o
  * executed operations (2019-06-06 04:39:08.082 +0000 UTC -> 2019-06-06 04:39:08.224 +0000 UTC, dur: 143ms):
      1. CREATE:
          - key: config/vpp/v2/interfaces/DST-1
          - value: { name:"DST-1" type:TAP enabled:true tap:<version:2 >  } 
      2. CREATE:
          - key: config/linux/interfaces/v2/interface/DST-1
          - value: { name:"DST-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/26052/ns/net" > host_if_name:"nsmx4UlWGtsv" enabled:true ip_addresses:"172.16.1.2/30" tap:<vpp_tap_if_name:"DST-1" >  } 
      3. CREATE [PROPERTY]:
          - key: linux/interface/address/DST-1/172.16.1.2/30
          - value: <EMPTY> 
      4. CREATE [PROPERTY]:
          - key: linux/interface/state/DST-1/UP
          - value: <EMPTY> 
      5. CREATE:
          - key: config/vpp/v2/interfaces/SRC-1
          - value: { name:"SRC-1" type:TAP enabled:true tap:<version:2 >  } 
      6. CREATE:
          - key: config/linux/interfaces/v2/interface/SRC-1
          - value: { name:"SRC-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/26312/ns/net" > host_if_name:"nsm0" enabled:true ip_addresses:"172.16.1.1/30" tap:<vpp_tap_if_name:"SRC-1" >  } 
      7. CREATE [PROPERTY]:
          - key: linux/interface/address/SRC-1/172.16.1.1/30
          - value: <EMPTY> 
      8. CREATE [PROPERTY]:
          - key: linux/interface/state/SRC-1/UP
          - value: <EMPTY> 
      9. CREATE:
          - key: config/linux/l3/v2/route/8.8.8.8/30/SRC-1
          - value: { outgoing_interface:"SRC-1" scope:LINK dst_network:"8.8.8.8/30"  } 
      10. CREATE [PROPERTY]:
          - key: linux/link-local-route/SRC-1/8.8.8.8/30
          - value: <EMPTY> 
      11. CREATE:
          - key: config/vpp/l2/v2/xconnect/DST-1
          - value: { receive_interface:"DST-1" transmit_interface:"SRC-1"  } 
      12. CREATE:
          - key: config/vpp/l2/v2/xconnect/SRC-1
          - value: { receive_interface:"SRC-1" transmit_interface:"DST-1"  } 
x----------------------------------------------------------------------------------------------------------------------x
| #2                                                                                                      took 142.7ms |
x----------------------------------------------------------------------------------------------------------------------x
2019-06-06 04:39:17,493 DEBG 'vppagent-dataplane' stdout output:
time="2019-06-06T04:39:17Z" level=info msg="Sending DataChange to vppagent: vpp_config: <\n  interfaces: <\n    name: \"SRC-1\"\n    type: TAP\n    enabled: true\n    tap: <\n      version: 2\n    >\n  >\n  interfaces: <\n    name: \"DST-1\"\n    type: TAP\n    enabled: true\n    tap: <\n      version: 2\n    >\n  >\n  xconnect_pairs: <\n    receive_interface: \"SRC-1\"\n    transmit_interface: \"DST-1\"\n  >\n  xconnect_pairs: <\n    receive_interface: \"DST-1\"\n    transmit_interface: \"SRC-1\"\n  >\n>\nlinux_config: <\n  interfaces: <\n    name: \"SRC-1\"\n    type: TAP_TO_VPP\n    namespace: <\n      type: FD\n      reference: \"/proc/26312/ns/net\"\n    >\n    host_if_name: \"nsm0\"\n    enabled: true\n    ip_addresses: \"172.16.1.1/30\"\n    tap: <\n      vpp_tap_if_name: \"SRC-1\"\n    >\n  >\n  interfaces: <\n    name: \"DST-1\"\n    type: TAP_TO_VPP\n    namespace: <\n      type: FD\n    >\n    host_if_name: \"nsmx4UlWGtsv\"\n    enabled: true\n    ip_addresses: \"172.16.1.2/30\"\n    tap: <\n      vpp_tap_if_name: \"DST-1\"\n    >\n  >\n  routes: <\n    outgoing_interface: \"SRC-1\"\n    scope: LINK\n    dst_network: \"8.8.8.8/30\"\n  >\n>\n"
======================================================================================================================+
| Transaction #3                                                                                        NB Transaction |
+======================================================================================================================+
  * transaction arguments:
      - seq-num: 3
      - type: NB Transaction
      - values:
          - key: config/linux/interfaces/v2/interface/DST-1
            val: <NIL>
          - key: config/linux/interfaces/v2/interface/SRC-1
            val: <NIL>
          - key: config/linux/l3/v2/route/8.8.8.8/30/SRC-1
            val: <NIL>
          - key: config/vpp/l2/v2/xconnect/DST-1
            val: <NIL>
          - key: config/vpp/l2/v2/xconnect/SRC-1
            val: <NIL>
          - key: config/vpp/v2/interfaces/DST-1
            val: <NIL>
          - key: config/vpp/v2/interfaces/SRC-1
            val: <NIL>


2019-06-06 04:39:17,495 DEBG 'agent' stdout output:
time="2019-06-06 04:39:17.49577" level=error msg="failed to get or create namespace (type:FD reference:\"/proc/26052/ns/net\" ): failed to get file /proc/26052/ns/net from path: no such file or directory" loc="descriptor/interface.go(434)" logger=linux-ifplugin.if-descriptor

2019-06-06 04:39:17,572 DEBG 'agent' stdout output:
time="2019-06-06 04:39:17.57222" level=debug msg="publish del to ifplugin_ifstate{{DST-1 true false vpp-interface-index} 0xc000614630}" loc="mem/inmemory_name_mapping.go(323)" logger=vpp-ifplugin.if-descriptor

2019-06-06 04:39:17,632 DEBG 'agent' stdout output:
time="2019-06-06 04:39:17.63126" level=debug msg="publish del to ifplugin_ifstate{{SRC-1 true false vpp-interface-index} 0xc00042fe00}" loc="mem/inmemory_name_mapping.go(323)" logger=vpp-ifplugin.if-descriptor
o----------------------------------------------------------------------------------------------------------------------o
  * executed operations (2019-06-06 04:39:17.496 +0000 UTC -> 2019-06-06 04:39:17.631 +0000 UTC, dur: 136ms):
      1. DELETE [PROPERTY]:
          - key: linux/interface/address/DST-1/172.16.1.2/30
          - value: <EMPTY> 
      2. DELETE [PROPERTY]:
          - key: linux/interface/state/DST-1/UP
          - value: <EMPTY> 
      3. DELETE [FAILED]:
          - key: config/linux/interfaces/v2/interface/DST-1
          - value: { name:"DST-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/26052/ns/net" > host_if_name:"nsmx4UlWGtsv" enabled:true ip_addresses:"172.16.1.2/30" tap:<vpp_tap_if_name:"DST-1" >  } 
          - error: failed to get or create namespace (type:FD reference:"/proc/26052/ns/net" ): failed to get file /proc/26052/ns/net from path: no such file or directory
      4. DELETE [PROPERTY]:
          - key: linux/link-local-route/SRC-1/8.8.8.8/30
          - value: <EMPTY> 
      5. DELETE:
          - key: config/linux/l3/v2/route/8.8.8.8/30/SRC-1
          - value: { outgoing_interface:"SRC-1" scope:LINK dst_network:"8.8.8.8/30"  } 
      6. DELETE [PROPERTY]:
          - key: linux/interface/address/SRC-1/172.16.1.1/30
          - value: <EMPTY> 
      7. DELETE [PROPERTY]:
          - key: linux/interface/state/SRC-1/UP
          - value: <EMPTY> 
      8. DELETE:
          - key: config/linux/interfaces/v2/interface/SRC-1
          - value: { name:"SRC-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/26312/ns/net" > host_if_name:"nsm0" enabled:true ip_addresses:"172.16.1.1/30" tap:<vpp_tap_if_name:"SRC-1" >  } 
      9. DELETE:
          - key: config/vpp/l2/v2/xconnect/DST-1
          - value: { receive_interface:"DST-1" transmit_interface:"SRC-1"  } 
      10. DELETE:
          - key: config/vpp/l2/v2/xconnect/SRC-1
          - value: { receive_interface:"SRC-1" transmit_interface:"DST-1"  } 
      11. DELETE:
          - key: config/vpp/v2/interfaces/DST-1
          - value: { name:"DST-1" type:TAP enabled:true tap:<version:2 >  } 
      12. DELETE:
          - key: config/vpp/v2/interfaces/SRC-1
          - value: { name:"SRC-1" type:TAP enabled:true tap:<version:2 >  } 
x----------------------------------------------------------------------------------------------------------------------x
| #3                                                                                                      took 135.8ms |
x----------------------------------------------------------------------------------------------------------------------x

2019-06-06 04:39:17,766 DEBG 'vppagent-dataplane' stdout output:
time="2019-06-06T04:39:17Z" level=info msg="Sending DataChange to vppagent: vpp_config: <\n  interfaces: <\n    name: \"SRC-1\"\n    type: TAP\n    enabled: true\n    tap: <\n      version: 2\n    >\n  >\n  interfaces: <\n    name: \"DST-1\"\n    type: TAP\n    enabled: true\n    tap: <\n      version: 2\n    >\n  >\n  xconnect_pairs: <\n    receive_interface: \"SRC-1\"\n    transmit_interface: \"DST-1\"\n  >\n  xconnect_pairs: <\n    receive_interface: \"DST-1\"\n    transmit_interface: \"SRC-1\"\n  >\n>\nlinux_config: <\n  interfaces: <\n    name: \"SRC-1\"\n    type: TAP_TO_VPP\n    namespace: <\n      type: FD\n      reference: \"/proc/26312/ns/net\"\n    >\n    host_if_name: \"nsm0\"\n    enabled: true\n    ip_addresses: \"172.16.1.1/30\"\n    tap: <\n      vpp_tap_if_name: \"SRC-1\"\n    >\n  >\n  interfaces: <\n    name: \"DST-1\"\n    type: TAP_TO_VPP\n    namespace: <\n      type: FD\n      reference: \"/proc/26869/ns/net\"\n    >\n    host_if_name: \"nsm3LYSTrsC\"\n    enabled: true\n    ip_addresses: \"172.16.1.2/30\"\n    tap: <\n      vpp_tap_if_name: \"DST-1\"\n    >\n  >\n  routes: <\n    outgoing_interface: \"SRC-1\"\n    scope: LINK\n    dst_network: \"8.8.8.8/30\"\n  >\n>\n"
+======================================================================================================================+
| Transaction #4                                                                                        NB Transaction |
+======================================================================================================================+
  * transaction arguments:
      - seq-num: 4
      - type: NB Transaction
      - values:
          - key: config/linux/interfaces/v2/interface/DST-1
            val: { name:"DST-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/26869/ns/net" > host_if_name:"nsm3LYSTrsC" enabled:true ip_addresses:"172.16.1.2/30" tap:<vpp_tap_if_name:"DST-1" >  }
          - key: config/linux/interfaces/v2/interface/SRC-1
            val: { name:"SRC-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/26312/ns/net" > host_if_name:"nsm0" enabled:true ip_addresses:"172.16.1.1/30" tap:<vpp_tap_if_name:"SRC-1" >  }
          - key: config/linux/l3/v2/route/8.8.8.8/30/SRC-1
            val: { outgoing_interface:"SRC-1" scope:LINK dst_network:"8.8.8.8/30"  }
          - key: config/vpp/l2/v2/xconnect/DST-1
            val: { receive_interface:"DST-1" transmit_interface:"SRC-1"  }
          - key: config/vpp/l2/v2/xconnect/SRC-1
            val: { receive_interface:"SRC-1" transmit_interface:"DST-1"  }
          - key: config/vpp/v2/interfaces/DST-1
            val: { name:"DST-1" type:TAP enabled:true tap:<version:2 >  }
          - key: config/vpp/v2/interfaces/SRC-1
            val: { name:"SRC-1" type:TAP enabled:true tap:<version:2 >  }


2019-06-06 04:39:17,806 DEBG 'agent' stdout output:
time="2019-06-06 04:39:17.80544" level=debug msg="publish add to ifplugin_ifstate{{DST-1 false false vpp-interface-index} 0xc0006147e0}" loc="mem/inmemory_name_mapping.go(277)" logger=vpp-ifplugin.if-descriptor

2019-06-06 04:39:17,863 DEBG 'agent' stdout output:
time="2019-06-06 04:39:17.86273" level=debug msg="publish add to ifplugin_ifstate{{SRC-1 false false vpp-interface-index} 0xc000488660}" loc="mem/inmemory_name_mapping.go(277)" logger=vpp-ifplugin.if-descriptor

2019-06-06 04:39:17,906 DEBG 'agent' stdout output:
o----------------------------------------------------------------------------------------------------------------------o
  * executed operations (2019-06-06 04:39:17.768 +0000 UTC -> 2019-06-06 04:39:17.906 +0000 UTC, dur: 138ms):
      1. CREATE:
          - key: config/vpp/v2/interfaces/DST-1
          - value: { name:"DST-1" type:TAP enabled:true tap:<version:2 >  } 
      2. CREATE [FIXED]:
          - key: config/linux/interfaces/v2/interface/DST-1
          - value: { name:"DST-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/26869/ns/net" > host_if_name:"nsm3LYSTrsC" enabled:true ip_addresses:"172.16.1.2/30" tap:<vpp_tap_if_name:"DST-1" >  } 
          - prev-error: failed to get or create namespace (type:FD reference:"/proc/26052/ns/net" ): failed to get file /proc/26052/ns/net from path: no such file or directory
      3. CREATE [PROPERTY]:
          - key: linux/interface/address/DST-1/172.16.1.2/30
          - value: <EMPTY> 
      4. CREATE [PROPERTY]:
          - key: linux/interface/state/DST-1/UP
          - value: <EMPTY> 
      5. CREATE:
          - key: config/vpp/v2/interfaces/SRC-1
          - value: { name:"SRC-1" type:TAP enabled:true tap:<version:2 >  } 
      6. CREATE:
          - key: config/linux/interfaces/v2/interface/SRC-1
          - value: { name:"SRC-1" type:TAP_TO_VPP namespace:<type:FD reference:"/proc/26312/ns/net" > host_if_name:"nsm0" enabled:true ip_addresses:"172.16.1.1/30" tap:<vpp_tap_if_name:"SRC-1" >  } 
      7. CREATE [PROPERTY]:
          - key: linux/interface/address/SRC-1/172.16.1.1/30
          - value: <EMPTY> 
      8. CREATE [PROPERTY]:
          - key: linux/interface/state/SRC-1/UP
          - value: <EMPTY> 
      9. CREATE:
          - key: config/linux/l3/v2/route/8.8.8.8/30/SRC-1
          - value: { outgoing_interface:"SRC-1" scope:LINK dst_network:"8.8.8.8/30"  } 
      10. CREATE [PROPERTY]:
          - key: linux/link-local-route/SRC-1/8.8.8.8/30
          - value: <EMPTY> 
      11. CREATE:
          - key: config/vpp/l2/v2/xconnect/DST-1
          - value: { receive_interface:"DST-1" transmit_interface:"SRC-1"  } 
      12. CREATE:
          - key: config/vpp/l2/v2/xconnect/SRC-1
          - value: { receive_interface:"SRC-1" transmit_interface:"DST-1"  } 
x----------------------------------------------------------------------------------------------------------------------x
| #4                                                                                                      took 137.8ms |
x----------------------------------------------------------------------------------------------------------------------x

Actual: request has not error
Expected: ?

Add support for Geneve tunnel

Binary API: https://github.com/ligato/vpp-binapi/blob/master/binapi/geneve/geneve.ba.go

agentctl: create command

Currently as put. However it should rather be named create to unify for any northbound access (etcd, grpc, ..). It would be best to follow CRUD for all commands - Create, Retrieve, Update, Delete.

not convenient. Currently you need to use the generate to print the data and then put . Too much work comparing to the vpp-agent-ctl where it works with single command. Make the command similar to the generate, so we can do something like put interface . We would at least found a use for the –label flag which is useless here. The option putting user-defined key-value pair should be persisted.
allow other database types, not just ETCD (at least Redis, but it would be nice to have all supported by the agent)

agentctl: gRPC support

delete config
create/update config
dump running
get config

Update/Delete/Update didn't work if same names are passed

Steps performed:

Create Connection between source/destination.
Let destination die, namespace is not available anymore.
Request Delete on vpp-agent with connection information passed, got error in logs.
Request new Connection betwrrn source and new destination but use same interface names.
Expect to see connection created but receive error.

time="2019-04-12 17:38:14.97837" level=error msg="failed to get link for interface veth-1649917878: Link not found" loc="descriptor/interface_veth.go(71)" logger=linux-ifplugin.if-descriptor

More details with logs:

1. Init connection between two pods in kubernetes (source/destination):

Update with following Config Object:

vpp_config:<interfaces:<name:\"SRC-1\" type:AF_PACKET enabled:true afpacket:<host_if_name:\"SRC-1-veth\" > > interfaces:<name:\"DST-1\" type:AF_PACKET enabled:true afpacket:<host_if_name:\"DST-1-veth\" > > xconnect_pairs:<receive_interface:\"SRC-1\" transmit_interface:\"DST-1\" > xconnect_pairs:<receive_interface:\"DST-1\" transmit_interface:\"SRC-1\" > > linux_config:<interfaces:<name:\"SRC-1-veth\" type:VETH host_if_name:\"SRC-1-veth\" enabled:true ip_addresses:\"10.20.1.1/30\" veth:<peer_if_name:\"SRC-1\" > > interfaces:<name:\"SRC-1\" type:VETH namespace:<type:FD reference:\"/proc/85977/ns/net\" > host_if_name:\"nsm\" enabled:true ip_addresses:\"10.20.1.1/30\" veth:<peer_if_name:\"SRC-1-veth\" > > interfaces:<name:\"DST-1-veth\" type:VETH host_if_name:\"DST-1-veth\" enabled:true ip_addresses:\"10.20.1.2/30\" veth:<peer_if_name:\"DST-1\" > > interfaces:<name:\"DST-1\" type:VETH namespace:<type:FD reference:\"/proc/85844/ns/net\" > host_if_name:\"nsmXEhBAHC5Y\" enabled:true ip_addresses:\"10.20.1.2/30\" veth:<peer_if_name:\"DST-1-veth\" > > arp_entries:<interface:\"SRC-1\" ip_address:\"10.40.1.135\" hw_address:\"0a:58:0a:28:01:87\" > arp_entries:<interface:\"SRC-1\" ip_address:\"fe80::d892:1bff:fe75:a2c3\" hw_address:\"0a:58:0a:28:01:87\" > routes:<outgoing_interface:\"SRC-1\" scope:LINK dst_network:\"8.8.8.8/30\" > > "

Got following in vpp-agent logs:

+======================================================================================================================+
| Transaction #2                                                                                        NB Transaction |
+======================================================================================================================+
 * transaction arguments:
      - seq-num: 2
      - type: NB Transaction
      - values:
          - key: config/linux/interfaces/v2/interface/DST-1
            val: { name:"DST-1" type:VETH namespace:<type:FD reference:"/proc/85844/ns/net" > host_if_name:"nsmXEhBAHC5Y" enabled:true ip_addresses:"10.20.1.2/30" veth:<peer_if_name:"DST-1-veth" >  }
          - key: config/linux/interfaces/v2/interface/DST-1-veth
            val: { name:"DST-1-veth" type:VETH host_if_name:"DST-1-veth" enabled:true ip_addresses:"10.20.1.2/30" veth:<peer_if_name:"DST-1" >  }
          - key: config/linux/interfaces/v2/interface/SRC-1
            val: { name:"SRC-1" type:VETH namespace:<type:FD reference:"/proc/85977/ns/net" > host_if_name:"nsm" enabled:true ip_addresses:"10.20.1.1/30" veth:<peer_if_name:"SRC-1-veth" >  }
          - key: config/linux/interfaces/v2/interface/SRC-1-veth
            val: { name:"SRC-1-veth" type:VETH host_if_name:"SRC-1-veth" enabled:true ip_addresses:"10.20.1.1/30" veth:<peer_if_name:"SRC-1" >  }
          - key: config/linux/l3/v2/arp/SRC-1/10.40.1.135
            val: { interface:"SRC-1" ip_address:"10.40.1.135" hw_address:"0a:58:0a:28:01:87"  }
          - key: config/linux/l3/v2/arp/SRC-1/fe80::d892:1bff:fe75:a2c3
            val: { interface:"SRC-1" ip_address:"fe80::d892:1bff:fe75:a2c3" hw_address:"0a:58:0a:28:01:87"  }
          - key: config/linux/l3/v2/route/8.8.8.8/30/SRC-1
            val: { outgoing_interface:"SRC-1" scope:LINK dst_network:"8.8.8.8/30"  }
          - key: config/vpp/l2/v2/xconnect/DST-1
            val: { receive_interface:"DST-1" transmit_interface:"SRC-1"  }
          - key: config/vpp/l2/v2/xconnect/SRC-1
            val: { receive_interface:"SRC-1" transmit_interface:"DST-1"  }
          - key: config/vpp/v2/interfaces/DST-1
            val: { name:"DST-1" type:AF_PACKET enabled:true afpacket:<host_if_name:"DST-1-veth" >  }
          - key: config/vpp/v2/interfaces/SRC-1
            val: { name:"SRC-1" type:AF_PACKET enabled:true afpacket:<host_if_name:"SRC-1-veth" >  }

2 Delete destination POD, send Cancel to vpp-agent.

Sending Delete with following Config object:

vpp_config:<interfaces:<name:\"SRC-1\" type:AF_PACKET enabled:true afpacket:<host_if_name:\"SRC-1-veth\" > > interfaces:<name:\"DST-1\" type:AF_PACKET enabled:true afpacket:<host_if_name:\"DST-1-veth\" > > xconnect_pairs:<receive_interface:\"SRC-1\" transmit_interface:\"DST-1\" > xconnect_pairs:<receive_interface:\"DST-1\" transmit_interface:\"SRC-1\" > > linux_config:<interfaces:<name:\"SRC-1-veth\" type:VETH host_if_name:\"SRC-1-veth\" enabled:true ip_addresses:\"10.20.1.1/30\" veth:<peer_if_name:\"SRC-1\" > > interfaces:<name:\"SRC-1\" type:VETH namespace:<type:FD reference:\"/proc/85977/ns/net\" > host_if_name:\"nsm\" enabled:true ip_addresses:\"10.20.1.1/30\" veth:<peer_if_name:\"SRC-1-veth\" > > interfaces:<name:\"DST-1-veth\" type:VETH host_if_name:\"DST-1-veth\" enabled:true ip_addresses:\"10.20.1.2/30\" veth:<peer_if_name:\"DST-1\" > > interfaces:<name:\"DST-1\" type:VETH namespace:<type:FD > host_if_name:\"nsmXEhBAHC5Y\" enabled:true ip_addresses:\"10.20.1.2/30\" veth:<peer_if_name:\"DST-1-veth\" > > arp_entries:<interface:\"SRC-1\" ip_address:\"10.40.1.135\" hw_address:\"0a:58:0a:28:01:87\" > arp_entries:<interface:\"SRC-1\" ip_address:\"fe80::d892:1bff:fe75:a2c3\" hw_address:\"0a:58:0a:28:01:87\" > routes:<outgoing_interface:\"SRC-1\" scope:LINK dst_network:\"8.8.8.8/30\" > > "

Got following in logs:

2019-04-12 17:38:14,749 DEBG 'agent' stdout output:
time="2019-04-12 17:38:14.74899" level=debug msg="Pushing data with 11 KV pairs (source: grpc)" loc="orchestrator/dispatcher.go(73)" logger=orchestrator.dispatcher

2019-04-12 17:38:14,749 DEBG 'agent' stdout output:
time="2019-04-12 17:38:14.74907" level=debug msg=" - DELETE: \"config/vpp/v2/interfaces/SRC-1\"" loc="orchestrator/dispatcher.go(96)" logger=orchestrator.dispatcher
time="2019-04-12 17:38:14.74914" level=debug msg=" - DELETE: \"config/vpp/v2/interfaces/DST-1\"" loc="orchestrator/dispatcher.go(96)" logger=orchestrator.dispatcher
time="2019-04-12 17:38:14.74920" level=debug msg=" - DELETE: \"config/vpp/l2/v2/xconnect/SRC-1\"" loc="orchestrator/dispatcher.go(96)" logger=orchestrator.dispatcher
time="2019-04-12 17:38:14.74925" level=debug msg=" - DELETE: \"config/vpp/l2/v2/xconnect/DST-1\"" loc="orchestrator/dispatcher.go(96)" logger=orchestrator.dispatcher
time="2019-04-12 17:38:14.74930" level=debug msg=" - DELETE: \"config/linux/interfaces/v2/interface/SRC-1-veth\"" loc="orchestrator/dispatcher.go(96)" logger=orchestrator.dispatcher
time="2019-04-12 17:38:14.74936" level=debug msg=" - DELETE: \"config/linux/interfaces/v2/interface/SRC-1\"" loc="orchestrator/dispatcher.go(96)" logger=orchestrator.dispatcher
time="2019-04-12 17:38:14.74941" level=debug msg=" - DELETE: \"config/linux/interfaces/v2/interface/DST-1-veth\"" loc="orchestrator/dispatcher.go(96)" logger=orchestrator.dispatcher

2019-04-12 17:38:14,749 DEBG 'agent' stdout output:
time="2019-04-12 17:38:14.74951" level=debug msg=" - DELETE: \"config/linux/interfaces/v2/interface/DST-1\"" loc="orchestrator/dispatcher.go(96)" logger=orchestrator.dispatcher
time="2019-04-12 17:38:14.74960" level=debug msg=" - DELETE: \"config/linux/l3/v2/arp/SRC-1/10.40.1.135\"" loc="orchestrator/dispatcher.go(96)" logger=orchestrator.dispatcher
time="2019-04-12 17:38:14.74967" level=debug msg=" - DELETE: \"config/linux/l3/v2/arp/SRC-1/fe80::d892:1bff:fe75:a2c3\"" loc="orchestrator/dispatcher.go(96)" logger=orchestrator.dispatcher

2019-04-12 17:38:14,750 DEBG 'agent' stdout output:
time="2019-04-12 17:38:14.74973" level=debug msg=" - DELETE: \"config/linux/l3/v2/route/8.8.8.8/30/SRC-1\"" loc="orchestrator/dispatcher.go(96)" logger=orchestrator.dispatcher
+======================================================================================================================+
| Transaction #6                                                                                        NB Transaction |
+======================================================================================================================+
  * transaction arguments:
      - seq-num: 6
      - type: NB Transaction
      - values:
          - key: config/linux/interfaces/v2/interface/DST-1
            val: <NIL>
          - key: config/linux/interfaces/v2/interface/DST-1-veth
            val: <NIL>
          - key: config/linux/interfaces/v2/interface/SRC-1
            val: <NIL>
          - key: config/linux/interfaces/v2/interface/SRC-1-veth
            val: <NIL>
          - key: config/linux/l3/v2/arp/SRC-1/10.40.1.135
            val: <NIL>
          - key: config/linux/l3/v2/arp/SRC-1/fe80::d892:1bff:fe75:a2c3
            val: <NIL>
          - key: config/linux/l3/v2/route/8.8.8.8/30/SRC-1
            val: <NIL>
          - key: config/vpp/l2/v2/xconnect/DST-1
            val: <NIL>
          - key: config/vpp/l2/v2/xconnect/SRC-1
            val: <NIL>
          - key: config/vpp/v2/interfaces/DST-1
            val: <NIL>
          - key: config/vpp/v2/interfaces/SRC-1
            val: <NIL>


2019-04-12 17:38:14,750 DEBG 'agent' stdout output:
time="2019-04-12 17:38:14.75036" level=error msg="failed to remove IP address 10.20.1.2/30 from linux interface DST-1-veth: Link not found" loc="descriptor/interface.go(417)" logger=linux-ifplugin.if-descriptor

2019-04-12 17:38:14,750 DEBG 'agent' stdout output:
time="2019-04-12 17:38:14.75062" level=error msg="failed to get or create namespace (type:FD reference:\"/proc/85844/ns/net\" ): failed to get file /proc/85844/ns/net from path: no such file or directory" loc="descriptor/interface.go(399)" logger=linux-ifplugin.if-descriptor

2019-04-12 17:38:14,790 DEBG 'vpp' stdout output:
/usr/bin/vpp[85783]: linux_epoll_file_update:120: epoll_ctl: Bad file descriptor (errno 9)

2019-04-12 17:38:14,803 DEBG 'agent' stdout output:
time="2019-04-12 17:38:14.80333" level=debug msg="publish del to ifplugin_ifstate{{SRC-1 true false vpp-interface-index} 0xc00025cea0}" loc="mem/inmemory_name_mapping.go(323)" logger=vpp-ifplugin.if-descriptor

2019-04-12 17:38:14,805 DEBG 'agent' stdout output:
o----------------------------------------------------------------------------------------------------------------------o
  * executed operations (2019-04-12 17:38:14.75 +0000 UTC -> 2019-04-12 17:38:14.804 +0000 UTC, dur: 54ms):
      1. DELETE [PROPERTY]:
          - key: linux/interface/address/DST-1/10.20.1.2/30
          - value: <EMPTY> 
      2. DELETE [PROPERTY]:
          - key: linux/interface/state/DST-1/UP
          - value: <EMPTY> 
      3. DELETE [PROPERTY]:
          - key: linux/interface/address/DST-1-veth/10.20.1.2/30
          - value: <EMPTY> 
      4. DELETE [PROPERTY]:
          - key: linux/interface/state/DST-1-veth/UP
          - value: <EMPTY> 
      5. DELETE [FAILED]:
          - key: config/linux/interfaces/v2/interface/DST-1-veth
          - value: { name:"DST-1-veth" type:VETH host_if_name:"DST-1-veth" enabled:true ip_addresses:"10.20.1.2/30" veth:<peer_if_name:"DST-1" >  } 
          - error: failed to remove IP address 10.20.1.2/30 from linux interface DST-1-veth: Link not found
      6. DELETE [FAILED]:
          - key: config/linux/interfaces/v2/interface/DST-1
          - value: { name:"DST-1" type:VETH namespace:<type:FD reference:"/proc/85844/ns/net" > host_if_name:"nsmXEhBAHC5Y" enabled:true ip_addresses:"10.20.1.2/30" veth:<peer_if_name:"DST-1-veth" >  } 
          - error: failed to get or create namespace (type:FD reference:"/proc/85844/ns/net" ): failed to get file /proc/85844/ns/net from path: no such file or directory
      7. DELETE:
          - key: config/linux/l3/v2/arp/SRC-1/10.40.1.135
          - value: { interface:"SRC-1" ip_address:"10.40.1.135" hw_address:"0a:58:0a:28:01:87"  } 
      8. DELETE:
          - key: config/linux/l3/v2/arp/SRC-1/fe80::d892:1bff:fe75:a2c3
          - value: { interface:"SRC-1" ip_address:"fe80::d892:1bff:fe75:a2c3" hw_address:"0a:58:0a:28:01:87"  } 
      9. DELETE [PROPERTY]:
          - key: linux/link-local-route/SRC-1/8.8.8.8/30
          - value: <EMPTY> 
      10. DELETE:
          - key: config/linux/l3/v2/route/8.8.8.8/30/SRC-1
          - value: { outgoing_interface:"SRC-1" scope:LINK dst_network:"8.8.8.8/30"  } 
      11. DELETE [PROPERTY]:
          - key: linux/interface/address/SRC-1/10.20.1.1/30
          - value: <EMPTY> 
      12. DELETE [PROPERTY]:
          - key: linux/interface/state/SRC-1/UP
          - value: <EMPTY> 
      13. DELETE [PROPERTY]:
          - key: linux/interface/address/SRC-1-veth/10.20.1.1/30
          - value: <EMPTY> 
      14. DELETE [PROPERTY]:
          - key: linux/interface/state/SRC-1-veth/UP
          - value: <EMPTY> 
      15. DELETE:
          - key: config/linux/interfaces/v2/interface/SRC-1-veth
          - value: { name:"SRC-1-veth" type:VETH host_if_name:"SRC-1-veth" enabled:true ip_addresses:"10.20.1.1/30" veth:<peer_if_name:"SRC-1" >  } 
      16. DELETE:
          - key: config/linux/interfaces/v2/interface/SRC-1
          - value: { name:"SRC-1" type:VETH namespace:<type:FD reference:"/proc/85977/ns/net" > host_if_name:"nsm" enabled:true ip_addresses:"10.20.1.1/30" veth:<peer_if_name:"SRC-1-veth" >  } 
      17. DELETE [NOOP WAS-PENDING]:
          - key: config/vpp/l2/v2/xconnect/DST-1
          - value: { receive_interface:"DST-1" transmit_interface:"SRC-1"  } 
      18. DELETE [NOOP WAS-PENDING]:
          - key: config/vpp/l2/v2/xconnect/SRC-1
          - value: { receive_interface:"SRC-1" transmit_interface:"DST-1"  } 
      19. DELETE [NOOP WAS-PENDING]:
          - key: config/vpp/v2/interfaces/DST-1
          - value: { name:"DST-1" type:AF_PACKET enabled:true afpacket:<host_if_name:"DST-1-veth" >  } 
      20. DELETE:
          - key: config/vpp/v2/interfaces/SRC-1
          - value: { name:"SRC-1" type:AF_PACKET enabled:true afpacket:<host_if_name:"SRC-1-veth" >  } 
x----------------------------------------------------------------------------------------------------------------------x
| #6                                                                                                       took 53.7ms |
x----------------------------------------------------------------------------------------------------------------------x

time="2019-04-12 17:38:14.80428" level=debug msg="Global interface state data updated: internal_name:\"host-SRC-1-veth\" index:3 status:\"DELETED\" mac_address:\"02:fe:43:f1:81:d8\" " loc="statuscheck/plugin_impl_statuscheck.go(251)" logger=status-check
time="2019-04-12 17:38:14.80480" level=debug msg="linux-interface descriptor retrieved 0 items: " loc="kvscheduler/refresh.go(129)" logger=kvscheduler
+======================================================================================================================+
| Transaction #7                                                                                       SB Notification |
+======================================================================================================================+
  * transaction arguments:
      - seq-num: 7
      - type: SB Notification
      - values:
          - key: linux/interface/host-name/SRC-1-veth
            val: <NIL>

o----------------------------------------------------------------------------------------------------------------------o
  * executed operations (2019-04-12 17:38:14.805 +0000 UTC -> 2019-04-12 17:38:14.805 +0000 UTC, dur: 0s):
      1. DELETE [WAS-OBTAINED]:
          - key: linux/interface/host-name/SRC-1-veth
          - value: <EMPTY> 
x----------------------------------------------------------------------------------------------------------------------x
| #7                                                                                                        took 100µs |
x----------------------------------------------------------------------------------------------------------------------x

time="2019-04-12 17:38:14.80524" level=error msg="Transaction #6 finished with 2 errors" loc="orchestrator/dispatcher.go(119)" logger=orchestrator.dispatcher
 -   1. error (DELETE) config/linux/interfaces/v2/interface/DST-1-veth - failed to remove IP address 10.20.1.2/30 from linux interface DST-1-veth: Link not found
 -   2. error (DELETE) config/linux/interfaces/v2/interface/DST-1 - failed to get or create namespace (type:FD reference:"/proc/85844/ns/net" ): failed to get file /proc/85844/ns/net from path: no such file or directory

time="2019-04-12 17:38:14.80529" level=info msg="Transaction 7 successful!" loc="kvscheduler/txn_process.go(487)" logger=kvscheduler

3 trying to configure new connection but with same interface names, I'm assume previous delete was ok.

 vpp_config:<interfaces:<name:\"SRC-1\" type:AF_PACKET enabled:true afpacket:<host_if_name:\"SRC-1-veth\" > > interfaces:<name:\"DST-1\" type:AF_PACKET enabled:true afpacket:<host_if_name:\"DST-1-veth\" > > xconnect_pairs:<receive_interface:\"SRC-1\" transmit_interface:\"DST-1\" > xconnect_pairs:<receive_interface:\"DST-1\" transmit_interface:\"SRC-1\" > > linux_config:<interfaces:<name:\"SRC-1-veth\" type:VETH host_if_name:\"SRC-1-veth\" enabled:true ip_addresses:\"10.20.1.1/30\" veth:<peer_if_name:\"SRC-1\" > > interfaces:<name:\"SRC-1\" type:VETH namespace:<type:FD reference:\"/proc/85977/ns/net\" > host_if_name:\"nsm\" enabled:true ip_addresses:\"10.20.1.1/30\" veth:<peer_if_name:\"SRC-1-veth\" > > interfaces:<name:\"DST-1-veth\" type:VETH host_if_name:\"DST-1-veth\" enabled:true ip_addresses:\"10.20.1.2/30\" veth:<peer_if_name:\"DST-1\" > > interfaces:<name:\"DST-1\" type:VETH namespace:<type:FD reference:\"/proc/86609/ns/net\" > host_if_name:\"nsmRMj5VM6fj\" enabled:true ip_addresses:\"10.20.1.2/30\" veth:<peer_if_name:\"DST-1-veth\" > > arp_entries:<interface:\"SRC-1\" ip_address:\"10.40.1.137\" hw_address:\"0a:58:0a:28:01:89\" > arp_entries:<interface:\"SRC-1\" ip_address:\"fe80::6cf6:b9ff:fe91:1fe8\" hw_address:\"0a:58:0a:28:01:89\" > routes:<outgoing_interface:\"SRC-1\" scope:LINK dst_network:\"8.8.8.8/30\" > > "

And following in vpp-agent logs:

+======================================================================================================================+
| Transaction #8                                                                                        NB Transaction |
+======================================================================================================================+
  * transaction arguments:
      - seq-num: 8
      - type: NB Transaction
      - values:
          - key: config/linux/interfaces/v2/interface/DST-1
            val: { name:"DST-1" type:VETH namespace:<type:FD reference:"/proc/86609/ns/net" > host_if_name:"nsmRMj5VM6fj" enabled:true ip_addresses:"10.20.1.2/30" veth:<peer_if_name:"DST-1-veth" >  }
          - key: config/linux/interfaces/v2/interface/DST-1-veth
            val: { name:"DST-1-veth" type:VETH host_if_name:"DST-1-veth" enabled:true ip_addresses:"10.20.1.2/30" veth:<peer_if_name:"DST-1" >  }
          - key: config/linux/interfaces/v2/interface/SRC-1
            val: { name:"SRC-1" type:VETH namespace:<type:FD reference:"/proc/85977/ns/net" > host_if_name:"nsm" enabled:true ip_addresses:"10.20.1.1/30" veth:<peer_if_name:"SRC-1-veth" >  }
          - key: config/linux/interfaces/v2/interface/SRC-1-veth
            val: { name:"SRC-1-veth" type:VETH host_if_name:"SRC-1-veth" enabled:true ip_addresses:"10.20.1.1/30" veth:<peer_if_name:"SRC-1" >  }
          - key: config/linux/l3/v2/arp/SRC-1/10.40.1.137
            val: { interface:"SRC-1" ip_address:"10.40.1.137" hw_address:"0a:58:0a:28:01:89"  }
          - key: config/linux/l3/v2/arp/SRC-1/fe80::6cf6:b9ff:fe91:1fe8
            val: { interface:"SRC-1" ip_address:"fe80::6cf6:b9ff:fe91:1fe8" hw_address:"0a:58:0a:28:01:89"  }
          - key: config/linux/l3/v2/route/8.8.8.8/30/SRC-1
            val: { outgoing_interface:"SRC-1" scope:LINK dst_network:"8.8.8.8/30"  }
          - key: config/vpp/l2/v2/xconnect/DST-1
            val: { receive_interface:"DST-1" transmit_interface:"SRC-1"  }
          - key: config/vpp/l2/v2/xconnect/SRC-1
            val: { receive_interface:"SRC-1" transmit_interface:"DST-1"  }
          - key: config/vpp/v2/interfaces/DST-1
            val: { name:"DST-1" type:AF_PACKET enabled:true afpacket:<host_if_name:"DST-1-veth" >  }
          - key: config/vpp/v2/interfaces/SRC-1
            val: { name:"SRC-1" type:AF_PACKET enabled:true afpacket:<host_if_name:"SRC-1-veth" >  }


2019-04-12 17:38:14,978 DEBG 'agent' stdout output:
time="2019-04-12 17:38:14.97837" level=error msg="failed to get link for interface veth-1649917878: Link not found" loc="descriptor/interface_veth.go(71)" logger=linux-ifplugin.if-descriptor

2019-04-12 17:38:14,978 DEBG 'agent' stdout output:
time="2019-04-12 17:38:14.97880" level=error msg="failed to get link for interface veth-2364841676: Link not found" loc="descriptor/interface_veth.go(71)" logger=linux-ifplugin.if-descriptor

2019-04-12 17:38:15,022 DEBG 'agent' stdout output:
o----------------------------------------------------------------------------------------------------------------------o
  * executed operations (2019-04-12 17:38:14.977 +0000 UTC -> 2019-04-12 17:38:15.022 +0000 UTC, dur: 45ms):
      1. CREATE [STILL-FAILING]:
          - key: config/linux/interfaces/v2/interface/DST-1
          - value: { name:"DST-1" type:VETH namespace:<type:FD reference:"/proc/86609/ns/net" > host_if_name:"nsmRMj5VM6fj" enabled:true ip_addresses:"10.20.1.2/30" veth:<peer_if_name:"DST-1-veth" >  } 
          - prev-error: failed to get or create namespace (type:FD reference:"/proc/85844/ns/net" ): failed to get file /proc/85844/ns/net from path: no such file or directory
          - error: failed to get link for interface veth-1649917878: Link not found
      2. CREATE [STILL-FAILING]:
          - key: config/linux/interfaces/v2/interface/DST-1-veth
          - value: { name:"DST-1-veth" type:VETH host_if_name:"DST-1-veth" enabled:true ip_addresses:"10.20.1.2/30" veth:<peer_if_name:"DST-1" >  } 
          - prev-error: failed to remove IP address 10.20.1.2/30 from linux interface DST-1-veth: Link not found
          - error: failed to get link for interface veth-2364841676: Link not found
      3. CREATE:
          - key: config/linux/interfaces/v2/interface/SRC-1-veth
          - value: { name:"SRC-1-veth" type:VETH host_if_name:"SRC-1-veth" enabled:true ip_addresses:"10.20.1.1/30" veth:<peer_if_name:"SRC-1" >  } 
      4. CREATE:
          - key: config/linux/interfaces/v2/interface/SRC-1
          - value: { name:"SRC-1" type:VETH namespace:<type:FD reference:"/proc/85977/ns/net" > host_if_name:"nsm" enabled:true ip_addresses:"10.20.1.1/30" veth:<peer_if_name:"SRC-1-veth" >  } 
      5. CREATE [PROPERTY]:
          - key: linux/interface/address/SRC-1/10.20.1.1/30
          - value: <EMPTY> 
      6. CREATE [PROPERTY]:
          - key: linux/interface/state/SRC-1/UP
          - value: <EMPTY> 
      7. CREATE [PROPERTY]:
          - key: linux/interface/address/SRC-1-veth/10.20.1.1/30
          - value: <EMPTY> 
      8. CREATE [PROPERTY]:
          - key: linux/interface/state/SRC-1-veth/UP
          - value: <EMPTY> 
      9. CREATE:
          - key: config/linux/l3/v2/arp/SRC-1/10.40.1.137
          - value: { interface:"SRC-1" ip_address:"10.40.1.137" hw_address:"0a:58:0a:28:01:89"  } 
      10. CREATE:
          - key: config/linux/l3/v2/arp/SRC-1/fe80::6cf6:b9ff:fe91:1fe8
          - value: { interface:"SRC-1" ip_address:"fe80::6cf6:b9ff:fe91:1fe8" hw_address:"0a:58:0a:28:01:89"  } 
      11. CREATE:
          - key: config/linux/l3/v2/route/8.8.8.8/30/SRC-1
          - value: { outgoing_interface:"SRC-1" scope:LINK dst_network:"8.8.8.8/30"  } 
      12. CREATE [PROPERTY]:
          - key: linux/link-local-route/SRC-1/8.8.8.8/30
          - value: <EMPTY> 
      13. CREATE [NOOP IS-PENDING]:
          - key: config/vpp/l2/v2/xconnect/DST-1
          - value: { receive_interface:"DST-1" transmit_interface:"SRC-1"  } 
      14. CREATE [NOOP IS-PENDING]:
          - key: config/vpp/l2/v2/xconnect/SRC-1
          - value: { receive_interface:"SRC-1" transmit_interface:"DST-1"  } 
      15. CREATE [NOOP IS-PENDING]:
          - key: config/vpp/v2/interfaces/DST-1
          - value: { name:"DST-1" type:AF_PACKET enabled:true afpacket:<host_if_name:"DST-1-veth" >  } 
      16. CREATE [NOOP IS-PENDING]:
          - key: config/vpp/v2/interfaces/SRC-1
          - value: { name:"SRC-1" type:AF_PACKET enabled:true afpacket:<host_if_name:"SRC-1-veth" >  } 
x----------------------------------------------------------------------------------------------------------------------x
| #8                                                                                                         took 45ms |
x----------------------------------------------------------------------------------------------------------------------x


2019-04-12 17:38:15,026 DEBG 'agent' stdout output:
time="2019-04-12 17:38:15.02471" level=debug msg="linux-interface descriptor retrieved 2 items: \n -  1. [linux-interface]: \"config/linux/interfaces/v2/interface/SRC-1-veth\" (from-NB)\n   { name:\"SRC-1-veth\" type:VETH host_if_name:\"SRC-1-veth\" enabled:true ip_addresses:\"10.20.1.1/30\" phys_address:\"52:63:e8:21:49:fc\" mtu:1500 veth:<peer_if_name:\"SRC-1\" >  }\n   Metadata: &{LinuxIfIndex:9 VPPTapName: Namespace:<nil>}\n -  2. [linux-interface]: \"config/linux/interfaces/v2/interface/SRC-1\" (from-NB)\n   { name:\"SRC-1\" type:VETH namespace:<type:FD reference:\"/proc/85977/ns/net\" > host_if_name:\"nsm\" enabled:true ip_addresses:\"10.20.1.1/30\" phys_address:\"56:bd:c3:63:70:9a\" mtu:1500 veth:<peer_if_name:\"SRC-1-veth\" >  }\n   Metadata: &{LinuxIfIndex:8 VPPTapName: Namespace:type:FD reference:\"/proc/85977/ns/net\" }" loc="kvscheduler/refresh.go(129)" logger=kvscheduler

2019-04-12 17:38:15,027 DEBG 'agent' stdout output:
time="2019-04-12 17:38:15.02722" level=error msg="Transaction #8 finished with 2 errors" loc="orchestrator/dispatcher.go(119)" logger=orchestrator.dispatcher

2019-04-12 17:38:15,028 DEBG 'vppagent-dataplane' stdout output:
2019/04/12 17:38:15 Reporting span 7e2120e855ea193a:7e2120e855ea193a:0:1

2019-04-12 17:38:15,028 DEBG 'agent' stdout output:
 -   1. error (CREATE) config/linux/interfaces/v2/interface/DST-1 - failed to get link for interface veth-1649917878: Link not found
 -   2. error (CREATE) config/linux/interfaces/v2/interface/DST-1-veth - failed to get link for interface veth-2364841676: Link not found

Mounting host directory in Docker container returns: operation not permitted

ERRO[0000] mount --make-shared failed loc="nsplugin/namespaces.go(222)" logger=linux-plugin-ns-handler namespace=veth-cfg-ns
ERRO[0000] plugin Linux: Init failed: operation not permitted loc="core/agent_core.go(264)" logger=agentcore took="488.906µs"

Issues in running Performance demo Scenario 2 and Scenario 4

I am successfully able to run Scenario 1 in the performance demo without any issues. However, when I try to run Scenario 2 and Scenario 4, I see the following issues:

Scenario 2:

When I telnet into the vswitch, and run show int the NIC interfaces are not visible and I only see the four memif interfaces. I double checked the interface names in the file scenario4/etcd.txt and also deleted the etcd pod and recreated it to ensure that the etcd configuration from the scenario 1 is completely deleted.
Both the VNFs show two memif interfaces each which is the expected behavior

Scenario 4

On the vswitch I see host-vnf1_port1 and host-vnf1_port2 but dont see any interfaces on the vnf

Any idea where I am going wrong? I have followed exactly the same steps for scenario 1 and got it working but don't see the expected behavior for the other two scenarios.

vpp-agent features

is every feature of vpp which is provided here https://wiki.fd.io/view/VPP/Features is implemented in ligato/vpp-agent or there are some functionalities missing?

Add support for VPP 19.08-rc0 (master)

rx-placement issue

When setting rx placements like this:

value:
        name: "TenGigabitEthernet3/0/0"
        type: "DPDK"
        enabled: true
        rx_placements:
          - queue: 0
            worker: 0
          - queue: 1
            worker: 2
          - queue: 2
            worker: 4
          - queue: 3
            worker: 6

it logs the error queue 0 redefined rx placement. while when I am trying to set do different workers same interface with same queue using vpp cli it let me do that without any problem

Optimize updating SAs in IPSec tunnels

When the IPSec tunnel is updated and only SA keys change, descriptor should only call relevant VPP binary API instead of recreating the tunnel.

Relevant VPP binary API:

vpp-agent/plugins/vpp/binapi/vpp1908/ipsec/ipsec.ba.go

Lines 883 to 884 in 018da11

 api.RegisterMessage((*IpsecTunnelIfSetSa)(nil), "ipsec.IpsecTunnelIfSetSa") 

 api.RegisterMessage((*IpsecTunnelIfSetSaReply)(nil), "ipsec.IpsecTunnelIfSetSaReply")

how vpp-agent is putting data in to vpp from etcd

when i run this command vpp-agent-ctl /opt/vpp-agent/dev/etcd.conf -tap it dump my data in to etcd somewhere here

vpp-agent/cmd/vpp-agent-ctl/data_cmd.go

Line 324 in 0599866

 ctl.broker.Put(interfaces.InterfaceKey(tap.Interfaces[0].Name), tap.Interfaces[0]) 

my question is how vpp-agent is putting this above configuration from etcd in to vpp?
second thing where we are using vppcalls packages in this vpp-agent-ctl /opt/vpp-agent/dev/etcd.conf -tap command because the thing which i have understand is this that these vppcalls folder in every plugin is communicating with https://github.com/ligato/vpp-agent/tree/master/plugins/vpp/binapi these generated go code and from there we are inserting our connfiguration in to vpp.

testepic

Allow specification of vxlan src/dst ports

I realize this will also require vpp to add support for non-standard vxlan src/dst ports, but most other vxlan implementations allow this, and its important in circumstances where one or the other side of a tunnel cannot use the standard ports.

old vendored code for cn-infra

Hello,

Please update vendor code for cn-infra. Current code for infra that comes with vpp-agent does not support grpc unix sockets, but master of cn-infra supports it.

Ikev2ProfileSetAuth gives error

when i want to put data in this stuct of ikev2 it gives me this error:

ERROR: no reply received within the timeout period 1s
vac_disconnect:407: vac_disconnect cnacel
vl_client_disconnect:322: queue drain: 7
msg_handler_internal:432: no handler for msg id 7
vl_client_disconnect:310: peer unresponsive, give up

my data to struct is:

req:=&ipsec.Ikev2ProfileSetAuth{
    Name : []byte("Profile1"),
    AuthMethod : uint8(2),
    Data : []byte("vpp123"),
}
reply := &ipsec.Ikev2ProfileSetAuthReply{}

if err := ch.SendRequest(req).ReceiveReply(reply); err != nil {
    fmt.Println("ERROR:", err)
    return
}
if reply.Retval != 0 {
    fmt.Println("Retval:", reply.Retval)
    return
}

vpp-agent failes when reading statistics

agent version: v2.1.1-1-g80401e6" BuildDate="2019-05-27T23:57-07:00" CommitHash=80401e66a0ef370ac61c404590de51d86f78bf4c

vpp version 1904

Agent failes at some point after reading statistics with following log:

vswitch.log

Remove usage of vpp-agent-ctl in robot tests

agentctl: show command

show the vpp-agent plugin status (as now), and configuration as a tree by default. Add commands show config and show status (or similar) to print only the wanted data.
show config prints single line, it should print the same as show config –all. I think the –all in not needed here, we may add additional filtering commands like show config interfaces or show config interfaces –names=if1,if2. Cmd show config –key= could be useful as well.
show config key now prints all the keys. I would prefer to move it to a new list command instead.

vpp-agent does not configure rx-mode

vpp version 1904
vpp-agent version v2.2.0-alpha-45-ge39b5b6" BuildDate="2019-05-31T01:28-07:00" CommitHash=e39b5b6578699d99b7268034fe0de1aeb27b7fde

rx-mode is not always set up. sometimes I see that memif is set to polling mode even if we set it to adaptive..
etcdctl output ->

/vnf-agent/vpp-unit-18/config/vpp/l2/v2/bridge-domain/L2BR18101
{"name": "L2BR18101", "flood": true, "unknown_unicast_flood": true, "forward": true, "learn": true, "mac_age": 5, "interfaces": [{"name": "loop181", "bridged_virtual_interface": true}, {"name": "memif181"}]}
/vnf-agent/vpp-unit-18/config/vpp/nat/v2/nat44-global
{"nat_interfaces": [{"name": "loop181", "is_inside": true}, {"name": "memif180"}], "address_pool": [{"address": "66.253.160.17"}]}
/vnf-agent/vpp-unit-18/config/vpp/v2/dhcp-proxy/IPv4
{"source_ip_address": "192.168.0.19", "servers": [{"ip_address": "66.253.163.250"}]}
/vnf-agent/vpp-unit-18/config/vpp/v2/interfaces/loop181
{"name": "loop181", "type": "SOFTWARE_LOOPBACK", "enabled": true, "ip_addresses": ["100.70.17.1/24"]}
/vnf-agent/vpp-unit-18/config/vpp/v2/interfaces/loop182
{"name": "loop182", "type": "SOFTWARE_LOOPBACK", "enabled": true, "ip_addresses": ["66.253.160.17/32"]}
/vnf-agent/vpp-unit-18/config/vpp/v2/interfaces/memif180
{"name": "memif180", "type": "MEMIF", "enabled": true, "ip_addresses": ["192.168.0.19/22"], "rx_modes": [{"mode": "ADAPTIVE", "default_mode": true}], "memif": {"socket_filename": "/run/memif/18.sock"}}
/vnf-agent/vpp-unit-18/config/vpp/v2/interfaces/memif181
{"name": "memif181", "type": "MEMIF", "enabled": true, "rx_modes": [{"mode": "ADAPTIVE", "default_mode": true}], "memif": {"id": 1, "socket_filename": "/run/memif/18.sock"}}
/vnf-agent/vpp-unit-18/config/vpp/v2/route/vrf/0/dst/0.0.0.0/0/gw/192.168.0.1
{"dst_network": "0.0.0.0/0", "next_hop_addr": "192.168.0.1"}
/vnf-agent/vpp-unit-18/config/vpp/v2/route/vrf/0/dst/10.0.0.0/8/gw/0.0.0.0
{"dst_network": "10.0.0.0/8", "next_hop_addr": "0.0.0.0", "type": "DROP"}
/vnf-agent/vpp-unit-18/config/vpp/v2/route/vrf/0/dst/172.16.0.0/12/gw/0.0.0.0
{"dst_network": "172.16.0.0/12", "next_hop_addr": "0.0.0.0", "type": "DROP"}
/vnf-agent/vpp-unit-18/config/vpp/v2/route/vrf/0/dst/192.168.0.0/16/gw/0.0.0.0
{"dst_network": "192.168.0.0/16", "next_hop_addr": "0.0.0.0", "type": "DROP"}

vpp-agent log vpp (2).txt

localclient_linux example failure

Hi,
I was trying to run example github.com/ligato/vpp-agent/examples/localclient_linux/main.go, but console gave me this error:

INFO[0000] Starting the agent...                         BuildDate= BuildVersion= loc="core/agent_core.go(68)" tag=00000000
DEBU[0000] Microservice label is set to vpp1             loc="servicelabel/plugin_impl_servicelabel.go(49)" tag=00000000
INFO[0000] Plugin ServiceLabel: init success             loc="core/agent_core.go(148)" tag=00000000
INFO[0000] Plugin StatusCheck: init success              loc="core/agent_core.go(148)" tag=00000000
INFO[0000] Plugin LinuxLocalClient: init success         loc="core/agent_core.go(148)" tag=00000000
INFO[0000] Connected to VPP.                            
INFO[0000] Plugin govpp: status check probe registered   loc="statuscheck/plugin_impl_statuscheck.go(135)" tag=00000000
INFO[0000] Agent plugin state update.                    lastErr=<nil> loc="statuscheck/plugin_impl_statuscheck.go(163)" plugin=govpp state=ok tag=00000000
INFO[0000] Plugin GoVPP: init success                    loc="core/agent_core.go(148)" tag=00000000
DEBU[0000] Initializing Linux interface plugin           loc="linuxplugin/linuxplugin_init.go(66)" tag=00000000
DEBU[0000] Initializing LinuxInterfaceConfigurator       loc="linuxplugin/interface_config.go(101)" tag=00000000
INFO[0000] Plugin Linux: init success                    loc="core/agent_core.go(148)" tag=00000000
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x58 pc=0x97a271]

goroutine 8 [running]:
pantheon.tech/ligato-bgp/bgp-vpp-agent/vendor/github.com/ligato/vpp-agent/plugins/defaultplugins.(*Plugin).Init(0xc4200e1c88, 0xc42018b4f0, 0x1)
    /root/go/src/pantheon.tech/ligato-bgp/bgp-vpp-agent/vendor/github.com/ligato/vpp-agent/plugins/defaultplugins/defaultplugins_init.go:128 +0xd1
pantheon.tech/ligato-bgp/bgp-vpp-agent/vendor/github.com/ligato/cn-infra/core.(*Agent).initPlugins(0xc42019c030, 0x0, 0x0)
    /root/go/src/pantheon.tech/ligato-bgp/bgp-vpp-agent/vendor/github.com/ligato/cn-infra/core/agent_core.go:136 +0x76
pantheon.tech/ligato-bgp/bgp-vpp-agent/vendor/github.com/ligato/cn-infra/core.(*Agent).Start.func1(0xc42019c030, 0xc420019860, 0xc420019800)
    /root/go/src/pantheon.tech/ligato-bgp/bgp-vpp-agent/vendor/github.com/ligato/cn-infra/core/agent_core.go:78 +0x2f
created by pantheon.tech/ligato-bgp/bgp-vpp-agent/vendor/github.com/ligato/cn-infra/core.(*Agent).Start
    /root/go/src/pantheon.tech/ligato-bgp/bgp-vpp-agent/vendor/github.com/ligato/cn-infra/core/agent_core.go:89 +0x308
exit status 2

I was running your example inside your development docker image published on docker hub (https://hub.docker.com/r/ligato/vpp-agent/, label latest, build ~14:30 on 5.9.2017). I used similar setup like you described in https://github.com/ligato/vpp-agent/tree/master/docker/dev_vpp_agent#example-using-the-development-environment-on-a-macbook-with-gogland , just without using Kafka and Etcd. I was using master branch of vpp-agent (commit 600f99a).

I looked also inside your code and it looks like something Kafka related. Can someone help me with this?

vpp-agent integration/relationship with Contiv-VPP

Hi. I'm struggling to figure out the relationship between ligato/vpp-agent and contiv/vpp. I understand that contive/vpp allows inter-node networking, while ligato/vpp-agent provides the dataplane configuration interface via KV store. However this would suggest that these two projects would need to be tightly coupled in some way.

In the Deployment.md, it briefly states:

For integration with Contiv we decided to embed the VPP Agent directly into the netplugin process

It makes reference to TBD in numerous places. I see a contiv branch that is actively being maintained as well but 180 commits behind master.

Is ligato/vpp and contiv/vpp currently compatible with each other? I'm happy to help with the testing/docs if there is a little more info on how to get started using both Contiv and Ligato together.

agentctl: config/state/dump

show config
retrieve state (not yet available in agent for all config items)

There is a difference between showing desired configuration and actual state. The desired configuration is stored under ../config/.. and actual state is under ../status/... There is also the real current state that can be dumped directly from southbound. Compared to the previously mentioned state, which might just show cached info from agent.

How to run tests?

I see some tests defined here
https://github.com/ligato/vpp-agent/tree/master/tests/robot/suites

But I cant figure out how to run them. Could you provide an example?

Thanks.

Running vppagent and vpp through Supervisord resulting in goVPP timeout error

Hi, I am trying to run the vpp agent k8s setup explained here. I am facing issues running vpp and vpp-agent through supervisord and getting the following error: Error loading core: plugin GoVPP not completed before timeout. However, when I run both vpp and vpp agent manually, it runs okay. Can you please let me know if there is something in supervisord configuration that needs to be changed to get this running.

agentctl: generate command

Support changing any fields with customized values.

For example:

generate vpp.interface
uses defaults
.. vpp.interface name myiface
sets one field
.. vpp.interface name myiface ip 1.2.3.4
sets two fields
etc..
all commands should be lowercased
flag –label does not work, it should replace the microservice label with defined value
generated data should be by default as JSON (currently YAML)
flag –short works only with JSON (that’s ok), but it should print some message if used with different formats
data like IP addresses, MAC addresses or interface types should be already generated , thus random. However I would left this to later since it can introduce a lot of errors where the generated data are not valid (for example several IP addresses from the same network on single interface)
selected data should be configurable. Every generated item would have a set of some non-global flags which can be passed to generate to set specified data fields as required. E.g. generate random interface but of this type, or with this list of IP addresses or with this name which is referenced elsewhere, etc.
nice to have: generate data to file (and append them if the same file is referenced) in the format the FileDB plugin can work with
flag –endpoints has no use here (should not be global as well)

	api.RegisterMessage((*IpsecTunnelIfSetSa)(nil), "ipsec.IpsecTunnelIfSetSa")
	api.RegisterMessage((*IpsecTunnelIfSetSaReply)(nil), "ipsec.IpsecTunnelIfSetSaReply")