Thanks @benz0li. It makes sense to me 👍

from gpu-jupyter.

Setting LD_LIBRARY_PATH in the jupyter notebook does resolve the issue in notebooks.

It doesn't appear to be required in a terminal after sourcing the jovyan bashrc, and it's not set in that terminal session either.

Changing the start.sh to not check for an existing /home/{$NB_USER} does allow the script to copy over the jovyan directory correctly and then new terminals are properly set up, but it doesn't fix notebooks. I'll try preserving LD_LIBRARY_PATH and see how that does.

import os

os.environ["LD_LIBRARY_PATH"] = "/opt/conda/lib/:/usr/local/cuda/lib64/lib"

import bitsandbytes

from gpu-jupyter.

This path doesn't exist in this image or any of the other ones i've used recently. It is the default ld_library_path on the 11.6.2-cudnn-runtime base image, but those folders don't exist on that image either.

That is correct. See also jupyter/docker-stacks#1792 (comment).
ℹ️ These paths /usr/local/nvidia/lib:/usr/local/nvidia/lib64 are kept for legacy reasons.

@benz0li Applying this fix you provided does copy over the LD_LIBRARY_PATH to the environment of the jupyter notbooks, but libcudart.so is still not found.

'LD_LIBRARY_PATH': '/usr/local/nvidia/lib:/usr/local/nvidia/lib64',

You need to set/extend the path to LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib:/usr/local/cuda/lib64¹ beforehand.

from gpu-jupyter.

@njacobson-nci The problem is that LD_LIBRARY_PATH is not preserved, which is essential for CUDA images.

And the default LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 must be set/extended to LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib:/usr/local/cuda/lib64¹ beforehand.

FYI @mathbunnyru

from gpu-jupyter.

@benz0li as far as I understand, in docker-stacks images we do not rely on LD_LIBRARY_PATH.
And --preserve-env doesn't preserve this environment variable.
I'm not sure we should explicitly preserve it.

https://www.sudo.ws/docs/man/sudoers.man/

The dynamic linker on most operating systems will remove variables that can control dynamic linking from the environment of set-user-ID executables, including sudo.

from gpu-jupyter.

I'm not sure we should explicitly preserve it.

@mathbunnyru For the jupyter/docker-stacks you are not supposed to.

from gpu-jupyter.

But if someone builds the jupyter/docker-stacks on top of nvidia/cuda images, LD_LIBRARY_PATH must be preserved – i.e. start.sh modified accordingly.

from gpu-jupyter.

@benz0li Applying this fix you provided does copy over the LD_LIBRARY_PATH to the environment of the jupyter notbooks, but libcudart.so is still not found.

'LD_LIBRARY_PATH': '/usr/local/nvidia/lib:/usr/local/nvidia/lib64',

This path doesn't exist in this image or any of the other ones i've used recently. It is the default ld_library_path on the 11.6.2-cudnn-runtime base image, but those folders don't exist on that image either.

from gpu-jupyter.

Updating the start.sh to set the ld_library_path as you called out here still has issues, but that might be a bitsandbytes thing.
'LD_LIBRARY_PATH': '/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib:/usr/local/cuda/lib64',

Appending /opt/conda/lib/ does resolve notebooks being able to find the libcudart.so.

This is what i added to the start.sh to fix it now
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda/lib:/usr/local/cuda/lib64:/opt/conda/lib/" \

It still seems like there is a deeper issue with switching users in this manner, and I wonder if there are further bugs that will be experienced when applying this fix.

In a notebook, if I run " ! ll " this alias is not found despite being defined in the jovyan/njacobson .bashrc, is that expected?

from gpu-jupyter.

@njacobson-nci I can't help you any further as my images only use Python – and don't have Conda / Mamba installed.

from gpu-jupyter.

Understood, I appreciate the help very much!

from gpu-jupyter.

P.S.: You can always install Conda / Mamba on user level in my images.

from gpu-jupyter.