"Unable to locate package nvidia-container-toolkit" on Debian (Ubuntu) x86_64 about h2ogpt HOT 4 CLOSED

Hi, please try the documentation here: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker

specifically try doing this first:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
      && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
      && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
            sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
            sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

This may be required to find the correct packages, and it was missed because on my system I had already done it before perhaps.

Let us know if this fixes it, in meantime I'll update instructions to include this step.

Thanks!

from h2ogpt.

Hi @pseudotensor, thank you for the commands. Yes it fixes the earlier problem but still having issues with the latter, which is;

nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory: unknown

Could you also specify the minimum CPU/Memory requirements for a machine to run this Docker container?

Thank you,
Best Regards

from h2ogpt.

The system requirements scale with the model size. E.g. 20B requires 4 48GB GPUs for generation unless use 8bit then 2 48GB GPUs is ok.

from h2ogpt.

Hi @iamdempa, just checking again if you are still experiencing issues with the latest changes.

If so, I would be happy to help, we typically use the steps here to setup cuda toolkit: https://github.com/h2oai/h2ogpt/blob/main/docs/INSTALL.md#installing-cuda-toolkit
but it could happen that under some different pre-conditions on your system the cuda libs are not found, in which case, one can check the /etc/ld.so.conf.d/cuda... and make sure it points to the right location of libnvidia-ml, that is if you can confirm that indeed libnvidia-ml.so.a is installed somewhere on your system (find / -name libnvidia-ml* 2> /dev/null).
If you can share the result of the find command, and how the ld cache is setup for your cuda install we debug.

from h2ogpt.