Error Handling for When the nvidia-smi fails on some nodes about rntop HOT 2 CLOSED

Hey @JunweiLiang and thanks for reaching out!
Happy to hear you use rntop.

As you see, rntop does not support handling connection issues at the moment, and I think it will not be that easy to add this support while making sure the data remains valid.

We are actually migrating the GPU cluster monitoring capabilities of rntop into our new and longer-term project Genv.
There, we handle connection issues properly.

Monitoring - and remote (cluster) monitoring in particular - in Genv is done using Prometheus and Grafana.
Here's a short document describing remote monitoring capabilities in Genv: https://docs.genv.dev/remote/monitoring.html
Here's a quick start tutorial to use monitoring capabilities in Genv with Prometheus and Grafana: https://docs.genv.dev/usage/monitoring.html#quick-start
And here's a list of all metrics exported today: https://docs.genv.dev/usage/monitoring.html#reference

Unfortunately, monitoring capabilities in Genv are not yet 100% compatible with the ones that exist in rntop.
For example, currently you have to install Genv in all remote machines before you can monitor them.
Secondly, at the moment, there are some metrics that are not yet exported in Genv, like device utilization, etc.

Is installing Genv on remote hosts an issue for you?
Is working with Prometheus and Grafana good for you?

If Genv can be used to monitor your cluster, let us know and we can estimate how much time it will take us to export the device utilization metric.

Waiting to hear from you!

from rntop.

Yeah I think this looks great after a quick peek. Installing some packages on every node is not a big issue. I think the direct web interface visualization will be of great help and much more convenient. With rntop we will need to download the log file and upload the file to every time see the visualization. Looking forward to the GPU util and memory usage functionalities!

Closing this issue.

from rntop.