plugin resources changed to 0 and couldn't be updated about kubernetes HOT 6 OPEN

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

from kubernetes.

/sig node

from kubernetes.

hi, thanks for the report. It's possible we have some issues in handling device plugins reconnections. Do we have a simple reproducer by any chance?

from kubernetes.

Actually, It is hard to reproduce a same condition, I'm not sure the root reason is about reconnections, but I think the mechanism of reconnection between kubelet and plugin could be improved(I haven't found the reconnection action on kubelet. ), isn't it ?

from kubernetes.

Actually, It is hard to reproduce a same condition, I'm not sure the root reason is about reconnections, but I think the mechanism of reconnection between kubelet and plugin could be improved(I haven't found the reconnection action on kubelet. ), isn't it ?

It is a responsability of the device plugin to handle reconnections, both on crashes/restarts of the device plugin itself and tolerating kubelet restarts: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#handling-kubelet-restarts

I'd be more than supportive to make the kubelet code more robust in presence of such events, but we need to narrow down the perimeter a bit, hence my previous question.

from kubernetes.

Thanks for your support, I'm agree with your opinion. Here is some additional notes:

1. https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#handling-kubelet-restarts Kubelet re-registers the restore connection function by monitoring the sock file in the plugin directory. I have confirmed that there is no problem with the function. But when this issue occurred, I found that the sock file in the directory was normal and had not been removed. However, I restarted kubelet and found that the plugin was re-registered and the kubelet value was correctly updated.
2. In the kubelet logs, I did not clearly observe logs of disconnection from the plugin, I just found the log like this:
   
   
   However, the time of these two logs is after the failure occurs. :(
3. For details about this failure, I also carefully reviewed the plugin code and raised relevant issues to them. For details, please refer to ：https://github.com/Mellanox/k8s-rdma-shared-dev-plugin/issues/109

from kubernetes.