Failed to acquire license from license server about nvidia-docker HOT 5 CLOSED

Are there any differences between the nodes?

Could you also describe how your are deploying the NVIDIA GPU Driver and configuring the nodes for GPU access? Are you using the NVIDIA GPU Operator, for example?

cc @cdesiniotis @shivamerla since this is vGPU related.

from nvidia-docker.

After checking with the vGPU team, looks like its due to the 2KB limit we have to store host network information in vGPU 15.2 drivers. Can you confirm if there are large number of network interfaces in the worker node hitting the issue? Team is suggesting to use vGPU 15.3 which avoids this issue as the buffer is increased to 4KB now.

from nvidia-docker.

There is no difference with the tree nodes.

We installed NVIDIA GPU Driver manually by
sudo sh NVIDIA-Linux-x86_64-525.105.17-grid.run

We installed NVIDIA container toolkit according this guide:
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/1.14.1/index.html

We are not using NVIDIA GPU Operator in this cluster because of the limitation of the environment,
there is no private container registry, and they have very strict proxy control (all outbound traffic need white listed).

Only one network interface in each worker node.

We cannot use 15.3 driver since this version is still not certified by the host vendor.
Host driver is 15.2.

Thank you very much,

from nvidia-docker.

Successfully acquire the license after reboot the node.

from nvidia-docker.

I have the same problem with the latest 16.1 driver, and the host is also a k8s node. After turning off kubelet auto-start to prevent cni initialization, the license can be successfully obtained. I think you should probably increase the buffer size a little larger.

from nvidia-docker.