Any way to reliably detect infiniband eth1? about azhpc-images HOT 6 CLOSED

The extremely hacky approach I've applied right now is: https://gist.github.com/abagshaw/a7dba307e5513f35d9314563bcb41ba0

But I'm hoping there is a more reliable way to identify if the instance we are on is of RDMA type or some more reliable heuristic to trigger the renaming by.

from azhpc-images.

@abagshaw The eth1 MAC address is in the range 00155d33ff0b - 00155d34fe07 on current IB clusters. You can rely on this for current clusters, but please note that this may not be always true in future SKUs.

One other way is to check the ib0 MAC address (or deduce from ib0 ipv6 address) and identify/rename the corresponding IB eth1 interface.

eg:

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::519b:214f:8106:3711 prefixlen 64 scopeid 0x20
ether 00:15:5d:33:ff:35 txqueuelen 1000 (Ethernet)

ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044
inet 172.16.1.44 netmask 255.255.0.0 broadcast 172.16.255.255
inet6 fe80::215:5dff:fd33:ff35 prefixlen 64 scopeid 0x20

from azhpc-images.

@jithinjosepkl Hmm I see - unfortunately I can't run the risk that the MAC range changes and the rule breaks in the future, I need something more future proof.

I thought about doing it through ib0 but I found that ib0 isn't up when the udev rules run for bringing eth1 up. Also, ib0 only shows up on SR-IOV enabled devices right?

Is there any way to reliably figure out if I am on an RDMA enabled instance type before the network is up? That would be enough for me.

from azhpc-images.

How about lspci and grep for IB device?

Another thought is to check metadata information and do your rules based on SKU type. (https://docs.microsoft.com/en-us/azure/virtual-machines/linux/instance-metadata-service). "r" in vmSize field indicates RDMA enabled VM.

from azhpc-images.

I tried lspci but it doesn't seem to have the IB device populated at the time of the udev rule running for eth1.

IMDS doesn't work because the udev rule runs before the network is up so I can't access IMDS.

from azhpc-images.

@abagshaw , you may want to use two different images here - one for IB-enabled SKUs and one for non-IB. I dont see any other option especially since you are checking this in the udev rules processing time. Also, checking the MAC range is not a bad idea tbh, the range is ~65K. It may take sometime to go over that limit in a single cluster.

I am closing this issue as this is not related to the images.

from azhpc-images.