Show CUDA Driver Version in the output about gpustat HOT 2 CLOSED

You are getting something wrong -- the CUDA version shown in nvidia-smi output has nothing to do with the actual CUDA runtime used in your system, it's just the latest CUDA version that the current NVIDIA Driver is compatible with.

So in your case, nvidia driver 535.113 can support CUDA 12.2 and below (as long as it's supported). But your ML environments might be using cuda 11.x.

To know the actual cuda runtime version on the current environment:

$ ldconfig -p | grep cudart
	libcudart.so.11.0 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudart.so.11.0
	libcudart.so.10.2 (libc6,x86-64) => /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudart.so.10.2
	libcudart.so (libc6,x86-64) => /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudart.so
	libcudart.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudart.so

Or from a specific DL framework, for instance:

print(torch.version.cuda)

from gpustat.

Discussion continued in #165

from gpustat.