Hey,

I'm trying to use your docker container in a kubernetes cluster with kubespawner for jupyterhub.
I got everything running except tensorflow with GPU acceleration because of this error from the container log:

"Could not load dynamic library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory"

Because I'm new to kubernetes/docker, I don't know if this library has to be inside the container, mounted from the host filesystem or through c.Spawner.env_keep = ['LD_LIBRARY_PATH'], which I already tried.

With the localspawner everything is working and every GPU (6 available) visible.
The container only get access to one GPU at a time.

But that's the result:

import tensorflow as tf
tf.config.list_physical_devices() 

[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'),
 PhysicalDevice(name='/physical_device:XLA_CPU:0', device_type='XLA_CPU'),
 PhysicalDevice(name='/physical_device:XLA_GPU:0', device_type='XLA_GPU')]

So I thought, maybe anyone here can guide me in the right direction.

Thanks in advance

Hey,

thanks for putting these dockerfiles online!
I tried building and running them, but despite confirmative output at
docker run --gpus all nvidia/cuda:10.1-base-ubuntu18.04 nvidia-smi

NVIDIA-SMI 450.51.05 Driver Version: 450.51.05 CUDA Version: 11.0

cuda is not available in pylabs. "torch.cuda.is_available()" returns "False".
"!nvidia-smi" in the notebook does also not work: "/bin/bash: nvidia-smi: command not found"

Any clue what could have gone wrong?

Best,
Tristan

Further infos:

• Docker version 19.03.12, build 48a66213fe
• docker-compose version 1.17.1, build unknown
• Ubuntu 18.04

Hi, I realized I am missing access to nvcc. There are some environment variables to export to fully use the toolkit: https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#post-installation-actions
Could you consider adding them to the image, maybe by adding the export lines to the bash profile?
Thanks for building those images!

EDIT: the libraries are there but I don’t see nvcc?

PackagesNotFoundError: The following packages are not available from current channels:
#30 19.03
#30 19.03 - mkl-include
#30 19.03 - mkl
#30 19.03
#30 19.03 Current channels:

I found this repo after a lot of struggling with setting up a docker image where I could run GPU accelerated TensorFlow with all the various creature comforts I've grown used to (jupyter lab plugins, access to local storage, modify which local files are shared or not, etc). You can see my pitiful attempts to set up here, but it looks like you folks have solved most of the problems I've encountered already.

It seems like this repo contains everything one would need to get started - but I had some concerns about running the setup scripts on my local machine - there are a lot of references to a specific username "jovyan", hard-coded local directory structure, as well as modifications to the passwd file.

I'm curious if this repo has been configured for one person to run and I'm just looking at a public repo of a passion project, or if its ok to run on my local machine without clobbering the passwd file with stuff that will mess up my ability to login?

Thanks!!

Dear all, first of all thanks a lot for your efforts! I am currently facing the problem that torch as well as numba successfully detect cuda or the gpu but tensorflow does not.

I have an Ubuntu "20.04.2 LTS (Focal Fossa)" machine with a Quadro RTX 6000 (Driver 460.56 and CUDA version 11.2). I am running the cschranz/gpu-jupyter:latest image with the digest sha256:5f0f9ee643573f6255e09242afbd107d339b089c1d3a1be2a66bd1b5b59ef443.

The base setup seems to work fine as I can run nvidia-smi from the terminal inside of the image and get reasonable results:

Running the PyTorch check returns True:

import torch
torch.cuda.is_available()

Running the numba check returns True as well:

from numba import cuda
cuda.is_available()

In contrast to this the TensorFlow check returns False:

import tensorflow as tf
tf.test.is_gpu_available(
    cuda_only=False, min_cuda_compute_capability=None
)

The output of

import tensorflow as tf
from tensorflow.python.client import device_lib
print(tf.config.list_physical_devices('XLA_GPU'))
device_lib.list_local_devices()

is

[name: "/device:CPU:0"
 device_type: "CPU"
 memory_limit: 268435456
 locality {
 }
 incarnation: 10269939690289433299,
 name: "/device:XLA_CPU:0"
 device_type: "XLA_CPU"
 memory_limit: 17179869184
 locality {
 }
 incarnation: 6286114861605125503
 physical_device_desc: "device: XLA_CPU device",
 name: "/device:XLA_GPU:0"
 device_type: "XLA_GPU"
 memory_limit: 17179869184
 locality {
 }
 incarnation: 2585380967679947376
 physical_device_desc: "device: XLA_GPU device"]

Do you have any ideas with regard to that issue?

Thanks a lot!

I went through the whole process of setting everything up on following https://medium.com/@christoph.schranz/set-up-your-own-gpu-based-jupyterlab-e0d45fcacf43

Everything installs just fine and it seems to start the docker container and jupyter lab however, I cannot seem to connect to it. See log file below

gpu-jupyter_1 | Set username to: jovyan
gpu-jupyter_1 | usermod: no changes
gpu-jupyter_1 | Granting jovyan sudo access and appending /opt/conda/bin to sudo PATH
gpu-jupyter_1 | Executing the command: jupyter lab
gpu-jupyter_1 | [I 20:43:07.743 LabApp] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret
gpu-jupyter_1 | [I 20:43:08.007 LabApp] [jupyter_nbextensions_configurator] enabled 0.4.1
gpu-jupyter_1 | install dir: /opt/conda/lib/python3.7/site-packages/jupyter_tabnine
gpu-jupyter_1 | Begin to download TabNine Binary from https://update.tabnine.com/2.3.4/x86_64-unknown-linux-gnu/TabNine
gpu-jupyter_1 | Finish download TabNine Binary to /opt/conda/lib/python3.7/site-packages/jupyter_tabnine/binaries/2.3.4/x86_64-unknown-linux-gnu/TabNine
gpu-jupyter_1 | [I 20:43:08.549 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.7/site-packages/jupyterlab
gpu-jupyter_1 | [I 20:43:08.549 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
gpu-jupyter_1 | [I 20:43:08.551 LabApp] Serving notebooks from local directory: /home/jovyan
gpu-jupyter_1 | [I 20:43:08.551 LabApp] The Jupyter Notebook is running at:
gpu-jupyter_1 | [I 20:43:08.551 LabApp] http://f421fa7971ac:8888/
gpu-jupyter_1 | [I 20:43:08.551 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
gpu-jupyter_1 | [C 20:44:15.403 LabApp] received signal 15, stopping
gpu-jupyter_1 | [I 20:44:15.404 LabApp] Shutting down 0 kernels
gpu-jupyter_1 | Set username to: jovyan
gpu-jupyter_1 | usermod: no changes
gpu-jupyter_1 | Granting jovyan sudo access and appending /opt/conda/bin to sudo PATH
gpu-jupyter_1 | Executing the command: jupyter lab
gpu-jupyter_1 | [I 20:46:54.775 LabApp] [jupyter_nbextensions_configurator] enabled 0.4.1
gpu-jupyter_1 | install dir: /opt/conda/lib/python3.7/site-packages/jupyter_tabnine
gpu-jupyter_1 | TabNine binary already exists in /opt/conda/lib/python3.7/site-packages/jupyter_tabnine/binaries/2.3.4/x86_64-unknown-linux-gnu/TabNine ignore downloading
gpu-jupyter_1 | [I 20:46:55.134 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.7/site-packages/jupyterlab
gpu-jupyter_1 | [I 20:46:55.134 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
gpu-jupyter_1 | [I 20:46:55.136 LabApp] Serving notebooks from local directory: /home/jovyan
gpu-jupyter_1 | [I 20:46:55.136 LabApp] The Jupyter Notebook is running at:
gpu-jupyter_1 | [I 20:46:55.136 LabApp] http://f421fa7971ac:8888/
gpu-jupyter_1 | [I 20:46:55.136 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

Also when I try to telnet to see if the port is open showing anything I get back
telnet localhost 8888
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused

Also doesnt work if I use the AWS address either.

Is docker not passing the jupyter notebook through to the port outside the container? How would I fix this.

If I run jupyter notebook or jupyter lab without the docker it works just fine.

Hi,

Thank you for sharing this project.

I get an error when I run " ./start-local.sh -p 8848 ":

The Compose file './docker-compose.yml' is invalid because:
services.gpu-jupyter.deploy.resources.reservations value Additional properties are not allowed ('devices' was unexpected)
Could you help with this please?

today, when I want to create container of cschranz/gpu-jupyter:latest, it got error message like this:

Failure OCI runtime create failed: container_linux.go:370: starting container process caused: exec: "tini": executable file not found in $PATH: unknown

how to fix this?

also, my current container also did not responding, to status 302

This is what I ran in the "Git Bash Here" prompt on Windows 10:

$ cd gpu-jupyter
$ ./generate-Dockerfile.sh
$ bash start-local.sh -p 8889

This is what the output of docker-compose logs -f looks like:

$ docker-compose logs -f
ERROR: The Compose file '.\docker-compose.yml' is invalid because:
services.gpu-jupyter.ports contains an invalid type, it should be a number, or an object

The quickstart setup for GPU support is missing a flag for docker: --gpus all.

I had to do (modified for my system config)

docker run -d -it --gpus all -p 8848:8888 -v /home/${USER}/workspace:/home/jovyan/work -e GRANT_SUDO=yes -e JUPYTER_ENABLE_LAB=yes --user root --restart always --name gpu-jupyter_1 gpu-jupyter```

Hello,
This is a great Docker image, thanks for releasing it.
Unfortunately, I'm having issues extending the your cuda11, ubuntu 18 Python only image.
If I add the following apt packages unzip and zip and the pip packages jupyter-server-proxy, jupyter-archive, prettytable, and upgrade pip and tensorboard I end up with a 7GB image that results in timeout when using helm upgrade.
Am I doing something wrong?
Here is the Dockerfile.usefulpackages :

LABEL authors="Christoph Schranz <[email protected]>, Mathematical Michael <[email protected]>"

USER root

RUN pip install --no-cache-dir ipyleaflet "plotly>=4.14.3" "ipywidgets>=7.5"

# Install important packages and Graphviz
RUN set -ex \
 && buildDeps=' \
    graphviz==0.11 \
' \
 && apt-get update \
 && apt-get -y install \
	htop \
 	apt-utils \
 	iputils-ping \
 	graphviz \
 	libgraphviz-dev \
 	openssh-client \
	zip \
	unzip \
 && pip install --no-cache-dir $buildDeps 

# Install various extensions
RUN fix-permissions $CONDA_DIR

# RUN jupyter labextension install @jupyterlab/github
# RUN pip install jupyterlab-git
RUN pip install jupyterlab-drawio
RUN jupyter nbextension enable --py --sys-prefix ipyleaflet
RUN jupyter labextension install jupyterlab-plotly
RUN jupyter labextension install @jupyter-widgets/jupyterlab-manager plotlywidget
# RUN pip install --no-cache-dir jupyter-tabnine  --user && \
#   jupyter nbextension install --py jupyter_tabnine --user && \
#   jupyter nbextension enable --py jupyter_tabnine --user && \
#   jupyter serverextension enable --py jupyter_tabnine --user
RUN pip install --no-cache-dir jupyter_contrib_nbextensions \
  jupyter_nbextensions_configurator rise
#  jupyter nbextension enable codefolding/main

RUN jupyter labextension install @ijmbarr/jupyterlab_spellchecker
RUN pip install --no-cache-dir --upgrade pip
RUN pip install --no-cache-dir prettytable
RUN pip install --no-cache-dir jupyter-server-proxy
RUN pip install --no-cache-dir --upgrade tensorboard
RUN pip install --no-cache-dir jupyter-archive

RUN fix-permissions /home/$NB_USER

# Switch back to jovyan to avoid accidental container runs as root
USER $NB_UID

Do you think that's good idea that it expose port 22 and preset user password to allow SSH/SFTP in
So that PyCharm user could use the container as compute backend remotely no matter where they are

I'm using this excellent docker image for tests.
Installing huggingface asrecognition fails, as it requires llvmlite-0.36.0-cp39-cp39-manylinux2010_x86_64. However, even the gpu-jupyter generated as "slim" (--slim) includes llvmlite 0.38, which cannot be uninstalled with pip.

I removed it with conda. Is it safe or you can suggest other solutions?

The Registry is set on port 5001 and can be changed in the docker-compose-swarm.yml in the line " image: 127.0.0.1:5001/gpu-jupyter".

I was looking into which parts of this image that mainly used space, as I felt the 17.9GB of uncompressed image (looked at cschranz/gpu-jupyter:v1.4_cuda-11.0_ubuntu-18.04).
According to Docker desktop on my machine, these lines contributed around 6GB of space:

This differs a lot from how for instance the Jupyter Project uses Conda in their dockerfiles, such as in the base-notebook: https://github.com/jupyter/docker-stacks/blob/6d61708f1747d4fb15e3c0166805ebb5fba41ea1/base-notebook/Dockerfile#L134-L144

A better approach for the lines in gpulibs would be something like:

RUN conda install --quiet --yes \
    pyyaml mkl mkl-include setuptools cmake cffi typing && \
    pip install torch torchvision torchaudio torchviz && \
    conda clean --all -f -y && \
    fix-permissions $CONDA_DIR && \
    fix-permissions /home/$NB_USER

I tried using this diff on my machine and got a image size of 13.8GB (uncompressed) on the "v1.4_cuda-11.0_ubuntu-18.04" -branch

diff --git a/.build/Dockerfile b/.build/Dockerfile
index f20b066..596c051 100755
--- a/.build/Dockerfile
+++ b/.build/Dockerfile
@@ -307,15 +307,9 @@ RUN pip install --upgrade pip && \
 
 # Install PyTorch with dependencies
 RUN conda install --quiet --yes \
-    pyyaml mkl mkl-include setuptools cmake cffi typing
-
-# Check compatibility here:
-# https://pytorch.org/get-started/locally/
-# Installation via conda leads to errors installing cudatoolkit=10.1
-RUN pip install torch torchvision torchaudio torchviz
-
-# Clean installation
-RUN conda clean --all -f -y && \
+    pyyaml mkl mkl-include setuptools cmake cffi typing && \
+    pip install torch torchvision torchaudio torchviz && \
+    conda clean --all -f -y && \
     fix-permissions $CONDA_DIR && \
     fix-permissions /home/$NB_USER

Some similar fixes might be found for pip aswell.

If there is a reason why you have split this into multiple stages, there might be smart to add a comment about it.

Steps:

./generate_dockerfile.sh --slim

bash start-local.sh -p 8888

Error:

failed to build, step 25. getting miniconda. Perhaps a link broke?

Hello,

Thanks for your helpful work.

I followed all the installation instructions, and everything went well:

• Prerequisites with : docker run --gpus all nvidia/cuda:10.1-base-ubuntu18.04 nvidia-smi => OK
• docker image building => OK

Then, when I try to run the docker container, I get the following error:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: requirement error: unsatisfied condition: cuda>=10.1, please update your driver to a newer version, or use an earlier cuda container\\\\n\\\"\"": unknown. ERRO[0003] error waiting for container: context canceled

This is a bit confusing since, I was expecting your script to select the right cuda version + it works with nvidia-smi command. What shall I do to have this working on my host ? Many thanks for your help.

The configuration of my host:

• nvidia-390.138
• cuda9
• cuDNN7

https://github.com/jupyter/docker-stacks has updated to Jupyterlab 3.x

We should do the same but make sure that all extensions we use work well with it.

Right now I tested and the 'slim' version and works fine out of the box. But the 'useful packages' version breaks.
So before update we will have to deal with those packages or see if they still make sense, since due to the changes in 3.x they added functionality that was in some extensions, so some are no longer needed.

Another change is that now some extensions are installed using pip or conda.

We can check them here: jupyterlab/jupyterlab#9461

As the popularity of this project grows, I would like to propose a logo for this project:

It's style is inspired by the main Jupyter project and has the following characteristics:

• can be pronounced as gupyter or gpupyter as proposed by @mathematicalmichael in #27
• the orange bows and points hint the top and bottom parts of G, P, and U behind the "gupyter"

What do you say? Should the bows and points be differently aligned?
Should this also be discussed with someone from the Jupyter foundation?

I'm also available on Facebook with my clear name and twitter as IoTChris.

First of all, thank you for your repository. And may I ask you why I can not create a new notebook. I tried a lot of answer from google but nothing seem happens. I tried to pass many params to overcome that errors like -e GRANT_SUDO=yes --user root NB_UID=$(id -u) -e NB_GID=$(id -g) but I still can not create new notebook.
Here is my full command:
sudo docker run -it --rm -d -p 9012:8888 -v $(pwd):/home/jovyan/work/ -e NB_UID=$(id -u) -e NB_GID=$(id -g) -e GRANT_SUDO=yes --user root cschranz/gpu-jupyter:v1.4_cuda-11.0_ubuntu-20.04_python-only.
I go to docker container and here is the error:
PermissionError: [Errno 13] Permission denied: '/home/jovyan/work/Cong/wsi_ai/.ipynb_checkpoints/Untitled-checkpoint.ipynb'
Thanks for any helps

.

@ChristophSchranz quick Q: are you manually updating the dockerhub?
If so, we can discuss adding directives to
https://github.com/iot-salzburg/gpu-jupyter/blob/master/.github/workflows/main.yml
which perform that function automatically. If this is something you're interested in, open a new issue and I can help close it out. you'll just need to add your dockerhub token to the secrets of this repo.

Originally posted by @mathematicalmichael in #19 (comment)

• automatic build of the "latest" tag (already works)
• automatically build the smaller images

hello, maybe I am not clear gave the description in previous thread: #54

I used your image to the students in the university lab. after I pulls the image, I put the password from the student. the method is :

nano jupyter_notebook_config.py

paste the password to c.NotebookApp.password = u'student salted password'.

then I restarted the service.

I always used the above method for your previous image (last month)

but this latest image, I cannot replicate the methods.

if I using your proposed: "Another solution would be to create a new Dockerfile that pulls cschranz/gpu-jupyter:tag" it would be burden due some student like to change the password often and I used portainer so the representative only use portainer to manage the docker and image (last time we gave them the terminal access, it was catastrophe)

Hi,
Could you add nvtop to your builds:
https://github.com/Syllo/nvtop

For some reason, I cannot build a new image with this Dockerfile

FROM cschranz/gpu-jupyter:v1.4_cuda-11.0_ubuntu-18.04
RUN git clone https://github.com/Syllo/nvtop.git &&
mkdir -p nvtop/build && cd nvtop/build &&
cmake .. -DNVML_RETRIEVE_HEADER_ONLINE=True &&
make && make install &&
cd /tmp && rm -rf /tmp/nvtop

Error:
/usr/bin/ld: cannot find -lnvidia-ml

Some environment variable is missing, because once the image is running I can compile nvtop if I first deactivate conda

There should be a parameter for generate_Dockerfile.sh to select the docker stack that you with to install.

In my case I want to run 'tensorflow-notebook' instead of 'datascience-notebook', I know tensorflow is added afterwards, but the final image is too big, and I think it shouldn't be to hard to implement this.

Hello,

Thanks a lot for maintaining this Docker image.
Starting from tensorflow 2.0, there's no separate version for CPU or GPU.
See this page: https://www.tensorflow.org/install/gpu
Therefore if you do pip install tensorflow-gpu>=2.0.*, it is actually an alias of doing pip install tensorflow>=2.0.*. The latter one will also have all GPU libraries.
However in the current latest version of gpu-jupyter, if someone do pip install tensorflow, there will be two identical tensorflows in the system.
I'm not sure if this should be an issue of gpu-jupyter or tensorflow itself, but I guess it is somehow better to get rid of the name tensorflow-gpu.

Thanks very much for reading,
Arthur Guo

Hi,

We have a running ML cluster and are currently doing something similar to have custom build image of docker-stacks on top of a nvidias gpu enabled container).

As it is always a good idea to not duplicate work, we thought of moving to your codebase.

However, perhaps a rather noob question but what is the difference with your approach vs installing pytorch/tensorflow and cudatoolkit (and cudann) via conda and setting the correct ENV variables (that is used in nvidia container to detect that it is a gpu container) ?

I'm getting an error when starting the container. Logs below:

[I 2021-02-20 11:07:52.437 ServerApp] jupyterlab | extension was successfully linked.
[W 2021-02-20 11:07:52.441 NotebookApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-02-20 11:07:52.441 NotebookApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-02-20 11:07:52.441 NotebookApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-02-20 11:07:52.441 NotebookApp] 'password' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-02-20 11:07:52.441 NotebookApp] 'password' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[I 2021-02-20 11:07:52.448 ServerApp] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/jupyter_cookie_secret
[I 2021-02-20 11:07:52.619 ServerApp] jupyter_tabnine | extension was found and enabled by nbclassic. Consider moving the extension to Jupyter Server's extension paths.
[I 2021-02-20 11:07:52.620 ServerApp] jupyter_tabnine | extension was successfully linked.
[I 2021-02-20 11:07:52.620 ServerApp] nbclassic | extension was successfully linked.
install dir: /home/jovyan/.local/lib/python3.8/site-packages/jupyter_tabnine
Begin to download TabNine Binary from https://update.tabnine.com/3.2.28/x86_64-unknown-linux-gnu/TabNine
[I 2021-02-20 11:07:52.813 ServerApp] jupyter_tabnine | extension was successfully loaded.
[I 2021-02-20 11:07:52.817 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.8/site-packages/jupyterlab
[I 2021-02-20 11:07:52.818 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2021-02-20 11:07:52.831 ServerApp] jupyterlab | extension was successfully loaded.
[I 2021-02-20 11:07:52.856 ServerApp] nbclassic | extension was successfully loaded.
[I 2021-02-20 11:07:52.856 ServerApp] Serving notebooks from local directory: /home/jovyan
[I 2021-02-20 11:07:52.856 ServerApp] Jupyter Server 1.4.0 is running at:
[I 2021-02-20 11:07:52.856 ServerApp] http://7f63aa28ed3f:8888/lab
[I 2021-02-20 11:07:52.856 ServerApp] or http://127.0.0.1:8888/lab
[I 2021-02-20 11:07:52.857 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
Traceback (most recent call last):
File "/opt/conda/lib/python3.8/site-packages/jupyter_server/serverapp.py", line 2018, in launch_browser
browser = webbrowser.get(self.browser or None)
File "/opt/conda/lib/python3.8/webbrowser.py", line 65, in get
raise Error("could not locate runnable browser")
webbrowser.Error: could not locate runnable browser

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/bin/jupyter-lab", line 10, in
sys.exit(main())
File "/opt/conda/lib/python3.8/site-packages/jupyter_server/extension/application.py", line 520, in launch_instance
serverapp.start()
File "/opt/conda/lib/python3.8/site-packages/jupyter_server/serverapp.py", line 2100, in start
self.start_app()
File "/opt/conda/lib/python3.8/site-packages/jupyter_server/serverapp.py", line 2059, in start_app
self.launch_browser()
File "/opt/conda/lib/python3.8/site-packages/jupyter_server/serverapp.py", line 2020, in launch_browser
self.log.warning(('No web browser found: %s.') % e)
UnboundLocalError: local variable '' referenced before assignment
Finish download TabNine Binary to /home/jovyan/.local/lib/python3.8/site-packages/jupyter_tabnine/binaries/3.2.28/x86_64-unknown-linux-gnu/TabNine
Semantic completion enabled.

I'm not sure if the problem is due to the "could not locate runnable browser", or the subsequent unboundlocalerror when trying to log the warning ...

I have a terminal open and I'm trying to install a new Debian package through apt or apt-get.

Example:

It asks for a password for the user 'jovyan'

or

It shows this without the sudo

Looking at this issue tells that the parameter -e GRANT_SUDO=yes to the user jovyan will solve it. Looking at your Dockerfile reveals that this parameter doesn't exist. Is there a possible help on where I can add this to the Docker file or how I can fix this?

Thanks

Hi,

I know it sounds bad, but I have just realised that the pc I'm trying to set up this system on has really old gpus, with cuda capability 3.0. Tensorflow supports >3.5. But theoretically it is possible to use it with cc 3.0 too by building tensorflow or disabling xla (?). However, I would like to use these gpus with this setup (jupyter + docker + gpu), do you have any idea how to solve this/whether is it possible to solve this? I'm still getting to know using docker.
I managed to launch jupyter lab following your tutorial on medium.com and only then I realised that the gpus were not detected.

I noticed it's not working anyway due to permissions. But moreover, I'm investigating thousands of chinese-origin POST requests to /login?next=%2Ftree%3F polluting my jupyter logs and noticed this while doing so:

$ docker logs -f jupyter-2
Set username to: jovyan
usermod: no changes
Granting jovyan sudo access and appending /opt/conda/bin to sudo PATH
Executing the command: jupyter notebook
[I 04:47:57.325 NotebookApp] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret
[INFO 2021-01-21 04:47:57.457 TabNine]:  install dir: /opt/conda/lib/python3.7/site-packages/jupyter_tabnine
[INFO 2021-01-21 04:47:57.589 TabNineDownloader]: Begin to download TabNine Binary from https://update.tabnine.com/3.2.28/x86_64-unknown-linux-gnu/TabNine
[ERROR 2021-01-21 04:47:57.590 TabNineDownloader]: Download failed, error: [Errno 13] Permission denied: '/opt/conda/lib/python3.7/site-packages/jupyter_tabnine/binaries'

The reason I started investigating this was because my GPU was running 150W for an indeterminate amount of time without me having kicked off any work. When I killed the singular docker process, the GPU calmed down immediately, so I'm sure the jupyter instance was responsible for the GPU. That of course makes the aforementioned POST requests especially fun... yay security!

My suggestion is we immediately change the default password.

Since docker/compose#6691 Docker compose supports GPU support without needing to install anything else.

# confirm TensorFlow sees the GPU
import tensorflow
from tensorflow.python.client import device_lib
assert 'GPU' in str(device_lib.list_local_devices())

--> no problem

# just try
import tensorflow as tf
with tf.device('/gpu:0'):
    a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
    b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
    c = tf.matmul(a, b)

with tf.Session() as sess:
    print (sess.run(c))

-->

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-4-ff2f78fb99cd> in <module>
      6     c = tf.matmul(a, b)
      7 
----> 8 with tf.Session() as sess:
      9     print (sess.run(c))

AttributeError: module 'tensorflow' has no attribute 'Session'

a solution might be

pip install --upgrade --force-reinstall tensorflow-gpu

(but it did't worked for me in a child docker image)

from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())

[name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 891693182611328360
, name: "/device:XLA_CPU:0"
device_type: "XLA_CPU"
memory_limit: 17179869184
locality {
}
incarnation: 4840590997946823616
physical_device_desc: "device: XLA_CPU device"
, name: "/device:GPU:0"
device_type: "GPU"
memory_limit: 5618991104
locality {
  bus_id: 1
  links {
  }
}
incarnation: 11947021443223061618
physical_device_desc: "device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0, compute capability: 6.1"
, name: "/device:XLA_GPU:0"
device_type: "XLA_GPU"
memory_limit: 17179869184
locality {
}
incarnation: 16370588895216994843
physical_device_desc: "device: XLA_GPU device"
]

PyTorch on GPU via CUDA works fine

Hi,
I am using cschranz/gpu-jupyter:v1.1_cuda-10.1_ubuntu-18.04 (due to jupyterhub 1.1.0 ) and I see cudann is missing, how should I get it installed ?
Thanks.

>>> import tensorflow.compat.v1 as tf
>>> tf.test.is_gpu_available(
...     cuda_only=False, min_cuda_compute_capability=None
... )
WARNING:tensorflow:From <stdin>:2: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
2021-03-18 15:04:59.245058: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-03-18 15:04:59.271631: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2399995000 Hz
2021-03-18 15:04:59.283598: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fa194000b20 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-03-18 15:04:59.283738: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2021-03-18 15:04:59.289403: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2021-03-18 15:04:59.494806: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55730caec680 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-03-18 15:04:59.494937: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2021-03-18 15:04:59.498194: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:86:00.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
2021-03-18 15:04:59.499007: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2021-03-18 15:04:59.503812: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2021-03-18 15:04:59.508316: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2021-03-18 15:04:59.509694: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2021-03-18 15:04:59.514401: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2021-03-18 15:04:59.516827: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2021-03-18 15:04:59.517725: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory
2021-03-18 15:04:59.517783: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1598] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2021-03-18 15:04:59.517877: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-03-18 15:04:59.517931: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2021-03-18 15:04:59.517985: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
False

I am trying to add jupyterlab-git(https://github.com/jupyterlab/jupyterlab-git) into this image, but don't know what is the right way to do. Does anybody know how to ?

I'm passing

-e GRANT_SUDO=1
--user root

but unlike the base images from docker-stacks, this seems insufficient to allow sudo apt-get update to work from the jupyter terminal.

is there a lock that can be disabled?

This is a good project. But my OS is CentOS 7. So, how should I fit it in my OS if I really want to use it. May be it could fix my cuda problem.

Hi:

I am trying to build own image via the README, just test run, didn't change anything.
I have hit the wall about this errors:

#6 40.76 Unpacking openssl (1.1.1f-1ubuntu2.11) over (1.1.1f-1ubuntu2.10) ...
#6 40.87 Preparing to unpack .../libsasl2-modules-db_2.1.27+dfsg-2ubuntu0.1_amd64.deb ...
#6 40.89 Unpacking libsasl2-modules-db:amd64 (2.1.27+dfsg-2ubuntu0.1) over (2.1.27+dfsg-2) ...
#6 40.99 Preparing to unpack .../libsasl2-2_2.1.27+dfsg-2ubuntu0.1_amd64.deb ...
#6 41.01 Unpacking libsasl2-2:amd64 (2.1.27+dfsg-2ubuntu0.1) over (2.1.27+dfsg-2) ...
#6 41.08 Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.11) ...
#6 41.15 Setting up libsasl2-modules-db:amd64 (2.1.27+dfsg-2ubuntu0.1) ...
#6 41.17 Setting up libsasl2-2:amd64 (2.1.27+dfsg-2ubuntu0.1) ...
#6 41.20 Setting up libfdisk1:amd64 (2.34-0.1ubuntu9.3) ...
#6 41.22 Setting up mount (2.34-0.1ubuntu9.3) ...
#6 41.25 Setting up openssl (1.1.1f-1ubuntu2.11) ...
#6 41.29 Setting up fdisk (2.34-0.1ubuntu9.3) ...
#6 41.32 Processing triggers for libc-bin (2.31-0ubuntu9.7) ...
#6 41.38 Reading package lists...
#6 41.45 Building dependency tree...
#6 41.47 Reading state information...
#6 41.47 Package fonts-liberation is not available, but is referred to by another package.
#6 41.47 This may mean that the package is missing, has been obsoleted, or
#6 41.47 is only available from another source
#6 41.47 
#6 41.47 E: Package 'fonts-liberation' has no installation candidate
#6 41.47 E: Unable to locate package pandoc
#6 41.47 E: Unable to locate package run-one
#6 41.47 E: Unable to locate package tini
------
executor failed running [/bin/bash -o pipefail -c apt-get update --yes &&     apt-get upgrade --yes &&     apt-get install --yes --no-install-recommends     ca-certificates     fonts-liberation     locales     pandoc     run-one     sudo     tini     wget &&     apt-get clean && rm -rf /var/lib/apt/lists/* &&     echo "en_US.UTF-8 UTF-8" > /etc/locale.gen &&     locale-gen]: exit code: 100
ubuntu@DESKTOP-ER9TLLQ:~/gpu-jupyter$ 

could you please advise what's the problem???

Thanks

Thank you for such an amazing project!

I want to use JupyterHub on Kubernetes to serve JupyterLab workspaces to multiple users. All users must be able to use the NVIDIA RAPIDS suite of GPU-accelerated Python Data Science libraries out-of-the-box.

However, none of the available Jupyter Docker Stacks provide RAPIDS yet.

I had a couple of questions:

1. Does the current gpu-jupyter DockerHub image work with JupyterHub out-of-the-box? I mean, if I just provide gpu-jupyter as a base image in the JupyterHub Helm config, will it work automatically? [EDIT: I was able to do this successfully! :)]
2. Can someone help in adding RAPIDS to the existing gpu-jupyter image, or direct me as to how I can build a custom image myself?

Any help would be greatly appreciated!

Hi,

I found this error when I try to execute: docker build -t gpu-jupyter .build/

Reading package lists...
W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 Release: At least one invalid signature was encountered.
E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 Release' is not signed.
W: GPG error: https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Release: At least one invalid signature was encountered.
E: The repository 'https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Release' is not signed.
W: GPG error: http://archive.ubuntu.com/ubuntu bionic InRelease: At least one invalid signature was encountered.
E: The repository 'http://archive.ubuntu.com/ubuntu bionic InRelease' is not signed.
W: GPG error: http://security.ubuntu.com/ubuntu bionic-security InRelease: At least one invalid signature was encountered.
E: The repository 'http://security.ubuntu.com/ubuntu bionic-security InRelease' is not signed.
W: GPG error: http://archive.ubuntu.com/ubuntu bionic-updates InRelease: At least one invalid signature was encountered.
E: The repository 'http://archive.ubuntu.com/ubuntu bionic-updates InRelease' is not signed.
W: GPG error: http://archive.ubuntu.com/ubuntu bionic-backports InRelease: At least one invalid signature was encountered.
E: The repository 'http://archive.ubuntu.com/ubuntu bionic-backports InRelease' is not signed.
The command '/bin/sh -c apt-get update && apt-get install -yq --no-install-recommends wget bzip2 ca-certificates sudo locales fonts-liberation run-one && apt-get clean && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 100

My environment is:

• MacOS Catalina Version 10.15.5 (19F101)
• Docker desktop 2.3.0.3 (45519)

Thanks,

JC

Hello, thanks to your team for providing such a wonderful tool. But now I have a problem.

When I run my application code in the Terminal，I get some error messages: No such file or directory: '/usr/local/cuda/bin/nvcc'

Traceback (most recent call last):
  File "setup.py", line 44, in get_cuda_version
    version_str = subprocess.check_output(
  File "/opt/conda/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/opt/conda/lib/python3.8/subprocess.py", line 493, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/opt/conda/lib/python3.8/subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/opt/conda/lib/python3.8/subprocess.py", line 1706, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/cuda/bin/nvcc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "setup.py", line 102, in <module>
    CUDA, CUDA_VERSION = locate_cuda()
  File "setup.py", line 83, in locate_cuda
    version = get_cuda_version(cuda_home)
  File "setup.py", line 51, in get_cuda_version
    raise RuntimeError("Cannot read cuda version file")
RuntimeError: Cannot read cuda version file

I test if nvcc exists:

(base) jovyan@8382018aecb5:~$ nvcc -V
bash: nvcc: command not found

After that, I tried to add environment variables when creating the image:

FROM cschranz/gpu-jupyter:v1.4_cuda-11.0_ubuntu-20.04_python-only

ENV GRANT_SUDO yes
ENV JUPYTER_ENABLE_LAB yes

ENV PATH $PATH:/usr/local/cuda/bin 
ENV CUDA_HOME /usr/local/cuda
ENV LD_LIBRARY_PATH /usr/local/cuda/lib64:$LD_LIBRARY_PATH

RUN python3 -m pip install opencv-contrib-python

And, use conda to install 'cudatoolkit', But it still didn't work:

(base) jovyan@8382018aecb5:~$ env | grep CUDA
NVIDIA_REQUIRE_CUDA=cuda>=11.0 brand=tesla,driver>=418,driver<419 brand=tesla,driver>=440,driver<441 brand=tesla,driver>=450,driver<451
CUDA_VERSION=11.0.3
CUDA_HOME=/usr/local/cuda
(base) jovyan@8382018aecb5:~$ nvcc -V
bash: nvcc: command not found

How should I solve this problem?

By the way, It's no problem to test cuda:

Thu Jul  1 03:03:31 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.84       Driver Version: 460.84       CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro P1000        Off  | 00000000:01:00.0  On |                  N/A |
| 34%   34C    P8    N/A /  N/A |    757MiB /  4038MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

import torch
torch.cuda.is_available() 
# True

# after run "conda install cudatoolkit -y"
from numba import cuda
cuda.is_available()
# True

Hi

It would be great if you include fastai in your image.

Best regards, Jakub.

I tried your protocol of gpu-jupyter deployment using docker swarm, and I managed to deploy Jupyter on my servers. But when deploying this way, GPU is no longer supported!

Is there any suggestions?

hello, I used cschranz/gpu-jupyter:latest, and I cannot login using my new password. I always used this approach:

create password from notebook.auth import passwd > passwd()

cd /home/jovyan/.jupyter/

nano jupyter_notebook_config.py

paste the password to c.NotebookApp.password = u'salted password'.

if I delete the password, the default password is back: gpu-jupyter

Hello,

I have an issue when trying to build the Docker image. I had already built the same image about six months ago without any errors, on the same machine. Here are the commands I have used (which are basically the commands presented on the Readme) :
git clone https://github.com/iot-salzburg/gpu-jupyter/
git checkout origin/v1.4_cuda-11.0_ubuntu-20.04
./generateDockerfile.sh --python-only
docker build -t gpu-jupyter .build/
No modification was made to the original source code.

The following error appears during step 75/81 :

RuntimeError: JupyterLab failed to build
See the log file for details: [...]
The command 'bin/bash -o pipefail -c jupyter labextension install jupyterlab-plotly' returned a non-zero code: 1

The log file is unaccessible, since it is created within the container.

I have also made sure that my GPUs are accessible from within Docker containers.

Thank you in advance!

I followed these instructions:
./generate_Dockerfile.sh
docker build -t gpu-jupyter .build/

I get:

Step 1/84 : FROM nvidia/cuda:10.1-base-ubuntu18.04
[...]
Step 61/84 : RUN conda install --quiet --yes 'tensorflow=1.13*' 'keras=2.2*' && conda clean --all -f -y && fix-permissions $CONDA_DIR && fix-permissions /home/$NB_USER
---> Running in 558a5b0a3260
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.

Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
failed

UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

Specifications:

• joblib -> python[version='2.7.|3.4.|3.5.|3.6.']
• mpmath -> python[version='2.7.|3.4.|3.5.|3.6.']
• patsy=0.5 -> python[version='2.7.|3.5.|3.6.*']
• python-json-logger -> python[version='2.7.|3.5.|3.6.*']
• tblib -> python[version='2.7.|3.5.|3.6.*']
• tzlocal -> python[version='2.7.|3.4.|3.5.|3.6.']
• vincent=0.4 -> python[version='2.7.|3.4.|3.5.|3.6.']
• xlrd -> python[version='2.7.|3.5.|3.6.*']

Your python: python=3.7

If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.

The following specifications were found to be incompatible with a past
explicit spec that is not an explicit spec in this operation (scipy):

• keras=2.2 -> h5py
• keras=2.2 -> scipy[version='>=0.14']
• tensorflow=1.13 -> protobuf[version='>=3.6.1']

The following specifications were found to be incompatible with each other:

Package libgcc-ng conflicts for:
r-uuid -> libgcc-ng[version='>=4.9|>=7.3.0']
zlib -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
r-dplyr -> libgcc-ng[version='7.2.0.|>=4.9|>=7.2.0|>=7.3.0']
r-rcpp -> libgcc-ng[version='7.2.0.|>=4.9|>=7.2.0|>=7.3.0']
r-fansi -> libgcc-ng[version='>=4.9|>=7.3.0']
r-reshape2=1.4 -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
sed -> libgcc-ng[version='>=4.9|>=7.3.0']
r-matrix -> libgcc-ng[version='7.2.0.|>=4.9|>=7.2.0|>=7.3.0']
tk -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
rpy2=3.1 -> libgcc-ng[version='>=7.3.0']
libxml2 -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
gcc_impl_linux-64 -> libgcc-ng[version='>=7.3.0']
r-processx -> libgcc-ng[version='7.2.0.|>=4.9|>=7.2.0|>=7.3.0']
libuv -> libgcc-ng[version='>=4.9|>=7.3.0']
libcblas -> libgcc-ng[version='>=7.3.0']
statsmodels=0.11 -> libgcc-ng[version='>=7.3.0']
r-sys -> libgcc-ng[version='>=4.9|>=7.3.0']
gettext -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
scikit-image=0.16 -> libgcc-ng[version='>=7.3.0']
xorg-xproto -> libgcc-ng[version='>=4.9|>=7.3.0']
cryptography -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
libllvm8 -> libgcc-ng[version='>=7.3.0']
harfbuzz -> libgcc-ng[version='>=4.9|>=7.3.0']
r-readr -> libgcc-ng[version='7.2.0.|>=4.9|>=7.2.0|>=7.3.0']
r-randomforest=4.6 -> libgcc-ng[version='>=4.9|>=7.3.0']
r-bitops -> libgcc-ng[version='7.2.0.|>=4.9|>=7.2.0|>=7.3.0']
r-rcpparmadillo -> libgcc-ng[version='>=4.9|>=7.3.0']
make -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
libedit -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
r-modelmetrics -> libgcc-ng[version='>=4.9|>=7.3.0']
sqlite -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
yaml -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
r-mass -> libgcc-ng[version='7.2.0.|>=4.9|>=7.2.0|>=7.3.0']
r-lattice -> libgcc-ng[version='7.2.0.|>=4.9|>=7.2.0|>=7.3.0']
lz4-c -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
cytoolz -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
pycosat -> libgcc-ng[version='>=4.9|>=7.3.0']
libffi -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
r-mime -> libgcc-ng[version='7.2.0.|>=4.9|>=7.2.0|>=7.3.0']
libiconv -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
r-mgcv -> libgcc-ng[version='7.2.0.|>=4.9|>=7.2.0|>=7.3.0']
msgpack-python -> libgcc-ng[version='>=7.2.0|>=7.3.0']
r-covr -> libgcc-ng[version='>=4.9|>=7.3.0']
cython=0.29 -> libgcc-ng[version='>=4.9|>=7.3.0']
r-openssl -> libgcc-ng[version='7.2.0.|>=4.9|>=7.2.0|>=7.3.0']
r-readxl -> libgcc-ng[version='7.2.0.|>=4.9|>=7.2.0|>=7.3.0']
ruamel.yaml.clib -> libgcc-ng[version='>=7.3.0']
r-fs -> libgcc-ng[version='>=4.9|>=7.3.0']
r-ipred -> libgcc-ng[version='7.2.0.|>=4.9|>=7.2.0|>=7.3.0']
gmp -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
xorg-renderproto -> libgcc-ng[version='>=4.9|>=7.3.0']
r-rcurl=1.98 -> libgcc-ng[version='>=7.3.0']
libpng -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
r-callr -> libgcc-ng[version='>=4.9|>=7.2.0']
r-nnet -> libgcc-ng[version='7.2.0.|>=4.9|>=7.2.0|>=7.3.0']
r-bit -> libgcc-ng[version='>=4.9|>=7.3.0']
pywavelets -> libgcc-ng[version='>=4.9|>=7.3.0']
libssh2 -> libgcc-ng[version='>=7.2.0|>=7.3.0']
r-rsqlite=2.1 -> libgcc-ng[version='>=4.9|>=7.3.0']
r-ellipsis -> libgcc-ng[version='>=7.3.0']
r-stringi -> libgcc-ng[version='7.2.0.|>=4.9|>=7.2.0|>=7.3.0']
mistune -> libgcc-ng[version='>=4.9|>=7.3.0']
pandas=1.0 -> libgcc-ng[version='>=7.3.0']
pyyaml -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
ruamel_yaml -> libgcc-ng[version='>=4.9|>=7.3.0']
r-purrr -> libgcc-ng[version='7.2.0.|>=4.9|>=7.2.0|>=7.3.0']
r-roxygen2 -> libgcc-ng[version='>=4.9|>=7.3.0']
kiwisolver -> libgcc-ng[version='>=7.2.0|>=7.3.0']
r-ps -> libgcc-ng[version='>=4.9|>=7.3.0']
zeromq -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
r-backports -> libgcc-ng[version='7.2.0.*|>=4.9|>=7.2.0|>=7.3.0']
xorg-xextproto -> libgcc-ng[version='>=4.9|>=7.3.0']
liblapacke -> libgcc-ng[version='>=7.3.0']
`
[+many more lines like this, not only for libgcc-ng[]

I have an Ubuntu 18.04 machine with a RTX 3090 (Driver 460.32.03 and CUDA version 11.2). I made a custom build of gpu-jupyter using the following commands:

git clone https://github.com/iot-salzburg/gpu-jupyter.git
cd gpu-jupyter
./generate-Dockerfile.sh
docker build -t gpu-jupyter .build/

My GPU, drivers, and the CUDA installation seem to be working fine based on calling nvidia-smi from the terminal inside Jupyter.
Also running this PyTorch check gives me True

import torch
torch.cuda.is_available() 

However, numba doesn't seem to detect any CUDA devices as the following returns False

from numba import cuda
cuda.is_available()

Running numba -s from the terminal within Jupyter detects my GPU:

__CUDA Information__
CUDA Device Initialized                       : True
CUDA Driver Version                           : 11020
CUDA Detect Output:
Found 1 CUDA devices
id 0     b'GeForce RTX 3090'                              [SUPPORTED]
                      compute capability: 8.6
                           pci device id: 0
                              pci bus id: 129
Summary:
        1/1 devices are supported

CUDA Librairies Test Output:
None

but shows the following error at the end:

Warning (cuda): Probing CUDA failed (device and driver present, runtime problem?)
(cuda) <class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/usr/local/cuda/nvvm/lib64'

With all the shiny new GPUs coming out recently, I propose updating to use images that run on CUDA 11.1.
I will try to do that in a forked version in the next few weeks and if the maintainer(s) on here think this is a good plan, I am happy to submit a pull request once that is done, and as soon as I get mine, test it on an RTX3080 to make sure it runs as it should.
In the same breath, I also propose moving the images to the new Ubuntu LTS.

I have been trying to install the image and launching the jupyterhub, but I'm getting Redirect Loop constantly. I tried with a jupyter-stack image (without Nvidia drivers) and it works fine.
What can it be related to?

Cheers,