Comments (14)
addressed in pr #19
from network-resources-injector.
@pperiyasamy what's the use case for one net-attach-def to have two distinct resources? Does the network defined in net-attach-def need to configure two devices for a Pod? Could that be done by having two net-attach-def attached to the same Pod?
from network-resources-injector.
@zshi-redhat There is a need of two resources (VF resource pool and OVS bridge) in a net-attach-def object for attaching Mellanox ConnectX 5 smartNIC VFs into pod container.
The VF resource pool is for the device plugin to allocate a VF device inside the container and OVS bridge name is to attach VF 's net representer devices into the OVS bridge which is done by ovs-cni plugin. The net representer device is mainly used to offload VLANs, VxLANs etc.
Hence we need to define these resources through k8s.v1.cni.cncf.io/resourceName
and k8s.v1.cni.cncf.io/bridgeName
into network-resources-injector for appropriate pod placement on the nodes (where both resources exists) in k8s cluster.
from network-resources-injector.
@zshi-redhat There is a need of two resources (VF resource pool and OVS bridge) in a net-attach-def object for attaching Mellanox ConnectX 5 smartNIC VFs into pod container.
The VF resource pool is for the device plugin to allocate a VF device inside the container and OVS bridge name is to attach VF 's net representer devices into the OVS bridge which is done by ovs-cni plugin. The net representer device is mainly used to offload VLANs, VxLANs etc.Hence we need to define these resources through
k8s.v1.cni.cncf.io/resourceName
andk8s.v1.cni.cncf.io/bridgeName
into network-resources-injector for appropriate pod placement on the nodes (where both resources exists) in k8s cluster.
@pperiyasamy @phoracek My udnerstanding is that VF representor is 1:1 mapped to a VF device, there is essentially one network/interface attached to Pod. Why not use ovs-cni to take care of plugging of both devices like how ovn-kubernetes does?
Also adding @dougbtv who might have more insight on standardizing the use of k8s.v1.cni.cncf.io/resourceName
in NPWG. Not sure if it's allowed to have two distinct annotations for one resource in net-attach-def.
from network-resources-injector.
@zshi-redhat OVS CNI uses resourceName
and resource injector to make sure to schedule Pods on Nodes with the requested bridge available. It does not inject the bridge into the Pod, we just use this mechanism to handle scheduling. This is for vanilla OVS CNI.
With HW offload, we need to A) schedule the Pod on a Node with the bridge available and B) schedule the Pod on a node with the VF available. Since we can request only a single resource, it is not possible with the current network-resources-injector.
At least that's my understanding of what @pperiyasamy is aiming to solve.
from network-resources-injector.
@zshi-redhat OVS CNI uses
resourceName
and resource injector to make sure to schedule Pods on Nodes with the requested bridge available. It does not inject the bridge into the Pod, we just use this mechanism to handle scheduling. This is for vanilla OVS CNI.With HW offload, we need to A) schedule the Pod on a Node with the bridge available and B) schedule the Pod on a node with the VF available. Since we can request only a single resource, it is not possible with the current network-resources-injector.
At least that's my understanding of what @pperiyasamy is aiming to solve.
@phoracek you are correct.
We can also make sure that OVS bridge is created on the node(s) where these device pool would be available through provisioning tool and just refer to device pool name with k8s.v1.cni.cncf.io/resourceName
in net-attach-def object.
But this addition with network-resources-injector ensures everything in place on the node to schedule a pod and helps to detect any misconfiguration.
from network-resources-injector.
@zshi-redhat OVS CNI uses
resourceName
and resource injector to make sure to schedule Pods on Nodes with the requested bridge available. It does not inject the bridge into the Pod, we just use this mechanism to handle scheduling. This is for vanilla OVS CNI.
Right, OVS CNI adds VF representor device into OVS bridge, it doesn't attach actual VF device to Pod (which I assume is done by SR-IOV CNI ?). My question is should the work (moving actual VF to Pod) also be done by OVS-CNI? Is it possible?
With HW offload, we need to A) schedule the Pod on a Node with the bridge available and B) schedule the Pod on a node with the VF available. Since we can request only a single resource, it is not possible with the current network-resources-injector.
If OVS and VF pool are required to run OVS HW Offload, should user guarantee the two are available on node during deployment or provisioning with a high-level tool? For use of OVS-CNI as default plugin, should there be an OVS bridge created already on each node?
At least that's my understanding of what @pperiyasamy is aiming to solve.
from network-resources-injector.
@phoracek you are correct.
We can also make sure that OVS bridge is created on the node(s) where these device pool would be available through provisioning tool and just refer to device pool name withk8s.v1.cni.cncf.io/resourceName
in net-attach-def object.
Yes, if OVS HW offload is expected to be used in cluster, then OVS bridge and device pool need to be created first on the same node to avoid scheduling problem. Then it becomes a installer issue not network-resoruce-injector. If ovs brdige is a limited resource in the deployment, it sounds to me that you probably need a device plugin to manage ovs resource.
from network-resources-injector.
/cc @moshe010
from network-resources-injector.
/cc @JanScheurich
from network-resources-injector.
@zshi-redhat OVS CNI uses
resourceName
and resource injector to make sure to schedule Pods on Nodes with the requested bridge available. It does not inject the bridge into the Pod, we just use this mechanism to handle scheduling. This is for vanilla OVS CNI.Right, OVS CNI adds VF representor device into OVS bridge, it doesn't attach actual VF device to Pod (which I assume is done by SR-IOV CNI ?). My question is should the work (moving actual VF to Pod) also be done by OVS-CNI? Is it possible?
The latest OVS CNI already attaches the actual Mellanox VF netdev to the pod. The SRIOV CNI does not play a role here.
With HW offload, we need to A) schedule the Pod on a Node with the bridge available and B) schedule the Pod on a node with the VF available. Since we can request only a single resource, it is not possible with the current network-resources-injector.
If OVS and VF pool are required to run OVS HW Offload, should user guarantee the two are available on node during deployment or provisioning with a high-level tool? For use of OVS-CNI as default plugin, should there be an OVS bridge created already on each node?
At least that's my understanding of what @pperiyasamy is aiming to solve.
Yes, for OVS HW off-load to work, the operator needs to consistently configure an OVS bridge and the corresponding VF device pool in the SRIOV network device plugin. I believe a higher-level operator is called for here to automate correct configuration of all parts.
If the system is configured correctly, I doubt that for the specific use case of SmartNIC VF attachments in OVS-CNI we need more than one resource per NAD to be handled by the network resources injector. The device pool suffices.
Peri's request might be useful in other scenarios, however, for example for a hypothetical CNI that requires both a device as well as another, independent, resource, for example an automatically allocated VLAN tag (similar to how Neutron assigns segmentation IDs to internal neutron networks).
from network-resources-injector.
@zshi-redhat, it seem kubervirt has [1] a bridge resource which exposes available Open vSwitch nodes as node resources. I comment on the ovs-cni issue to better understand why it implemented like this and not with nfd for example.
@zshi-redhat , What is you main objections to inject more then one resources to the same network? it it just the use-case?
[1] - https://github.com/kubevirt/ovs-cni/blob/9115a6ca16fbda50e063429f8ceb3c8afab58856/docs/marker.md
from network-resources-injector.
@zshi-redhat, it seem kubervirt has [1] a bridge resource which exposes available Open vSwitch nodes as node resources. I comment on the ovs-cni issue to better understand why it implemented like this and not with nfd for example.
@zshi-redhat , What is you main objections to inject more then one resources to the same network? it it just the use-case?
I think this might not be a typical use of net-attach-def which requires two devices
for one network and the major concern is if I were to consider using it with NPWG compliant meta plugin like Multus for additional network, Multus doesn't have ability to read other resource annotations from net-attach-def as it assumes one device per network. but on the other hand, I don't know if ovs-cni will be used with any meta plugin for additional network.
from network-resources-injector.
The PR #19 addresses this issue, closing the issue.
from network-resources-injector.
Related Issues (20)
- resource not injected in a large K8s cluster HOT 7
- Certificates v1beta1 will no longer be support on k8s release 1.22 HOT 4
- Add stable tag for network resource injector container
- Release network-resources-injector v1.3 HOT 5
- Create a Deployment yaml for network-resource-injector
- Add Helm Chart
- NRI pod restart not handled HOT 3
- Health check for webhook server
- Release of Network resources injector HOT 3
- Too wide permissions given to service accounts
- CI lane is not able to finish the kind deployment HOT 2
- Help understanding webhook client certificates
- New release ? HOT 4
- 'User defined injections' feature does not react on ConfigMap remove HOT 1
- 'User defined injections' feature removes defined network annotations HOT 2
- create new release HOT 4
- User Defined Injections - does not take into account json path operation HOT 2
- User Defined Injections - injects only one property
- Can Network-Resources-Injector(NRI) to be deployed as a deployment HOT 2
- Resources limit and requests are added only to first container inside POD HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from network-resources-injector.