This is a fork of latonaio's lic-docker-cuda-ce repository over at BitBucket. Original source here: https://bitbucket.org/latonaio/lic-docker-cuda-ce/src/master/
I modified this script so it installs CUDA 10.2, and all other related libraries matching my Jetson Nano's cache, so the README is not fully accurate at the moment. The only updated section is the list of .deb
files you will need to gather.
You can find my latest build here: https://hub.docker.com/repository/docker/thebozzcl/cuda
This repository aims to provide a
CUDA
"enabled" container for NVIDIA
Jetson
devices AGX Xavier,
TX2,
and Nano.
This container contains some of NVIDIA Tegra libs and the CUDA libs. In the version 0.1.0:
- CUDA 10.0.166
- NVInfer 5.0.6-1
- Cudnn 7.3.1.28-1
- TensorRT 5.0.6.3-1
- Tegra drivers libs 32-1.0
- NVIDIA Gstreamer plugins from Tegra 32-1.0
Note: those proprietary libs need to be downloaded separately. Please refer to section 3 below.
The Dockerfile is designed to produce two flavours of the image :
a "build
" mode and a "run
" mode.
- The
build
image contains the necessary files (such as headers and-dev
packages) to build CUDA-enabled projects. - The
run
is a much lighter image which includes only files needed at runtime such as shared libs.
This allows using a builder/runner pattern with multistage dockerfiles.
To build the image, various archives need to be provided. As the CUDA libs are not accessible directly from a public URL, they need to be manually down once.
To retrieve the .deb files you can use JetPack SDK Manager. The files are extracted in a folder specified during the installation process at step 2 (see the download and install options).
Note: it is required to put those files like the following. Minor versions might be different depending on the Jetpack version.
NOTE FROM @rjbozzol: I updated this section of the original README to show what my own DEB path ended up being. Instead of using the JetPack SDK Manager, I found all these files just sitting in /var/cache/apt/archives/
! This could be automated to get all the libraries when building in a Tegra machine... but I'll leave that as a potential upgrade in the future.
./nvidia-deb/
├── cuda
│ └── cuda-repo-ubuntu1804-10-2-local-10.2.107-435.17.01_1.0-1_arm64.deb
├── cudnn
│ ├── libcudnn8_8.0.0.180-1+cuda10.2_arm64.deb
│ ├── libcudnn8-dev_8.0.0.180-1+cuda10.2_arm64.deb
│ └── libcudnn8-doc_8.0.0.180-1+cuda10.2_arm64.deb
├── libnvonnxparsers-dev_7.1.3-1+cuda10.2_arm64.deb
├── libnvparsers-dev_7.1.3-1+cuda10.2_arm64.deb
├── nvinfer
│ ├── libnvinfer7_7.1.3-1+cuda10.2_arm64.deb
│ ├── libnvinfer-bin_7.1.3-1+cuda10.2_arm64.deb
│ ├── libnvinfer-dev_7.1.3-1+cuda10.2_arm64.deb
│ ├── libnvinfer-doc_7.1.3-1+cuda10.2_all.deb
│ ├── libnvinfer-plugin7_7.1.3-1+cuda10.2_arm64.deb
│ ├── libnvinfer-plugin-dev_7.1.3-1+cuda10.2_arm64.deb
│ ├── libnvinfer-samples_7.1.3-1+cuda10.2_all.deb
│ ├── libnvonnxparsers7_7.1.3-1+cuda10.2_arm64.deb
│ ├── libnvonnxparsers-dev_7.1.3-1+cuda10.2_arm64.deb
│ ├── libnvparsers7_7.1.3-1+cuda10.2_arm64.deb
│ ├── libnvparsers-dev_7.1.3-1+cuda10.2_arm64.deb
│ ├── python3-libnvinfer_7.1.3-1+cuda10.2_arm64.deb
│ └── python3-libnvinfer-dev_7.1.3-1+cuda10.2_arm64.deb
└── tensorrt
└── tensorrt_7.1.3.0-1+cuda10.2_arm64.deb
Tegra's archives are downloaded directly from NVIDIA website within the container at build.
For Tegra 32.1: JAX-TX2-Jetson_Linux_R32.1.0_aarch64.tbz2
To build the image you should run:
-
For the
build
version:$ docker build --build-arg RUN=0 -t tegra-cuda-base:build .
-
For the
run
version:$ docker build --build-arg RUN=1 -t tegra-cuda-base:run .
The build can be configured using the following args. Here are the default values:
CUDA_VERSION_DASHED
: Version of CUDA to be installed. This must match the version of a CUDA deb file (i.e. cuda-repo-l4t-XX-Y-local_*) in your localnvidia-deb/cuda
folder. (default= "10-0")
TEGRA_VERSION
: The version of Tegra libs used. (default= 32.1.0)TEGRA_ARCHIVE
: The URL of the archive of the Tegra's drivers to download. (default= https://developer.nvidia.com/embedded/dlc/l4t-jetson-driver-package-32-1-JAX-TX2)Note: So far, the archive's name was not constant between version 28.2 and 32.1 of Tegra. Therefore the URL is not created from
TEGRA_VERSION
. Please be sure thatTEGRA_VERSION
matches the version ofTEGRA_ARCHIVE
.NV_DRIVERS
: name of the archive containing the L4T drivers inside the Tegra archive. (default= Linux_for_Tegra/nv_tegra/nvidia_drivers.tbz2)NV_GSTAPPS
: name of the archive containing the Gstreamer plugins inside the Tegra archive. (default= Linux_for_Tegra/nv_tegra/nv_sample_apps/nvgstapps.tbz2)
RUN
: Set to1
if you want to produce a "run" image (i.e. without build dependencies). To keep the build dependencies and create a "build" image, set it to0
. Please refer to section 2 for details about these two modes. (default:0
)
In this example, we will see how to use the CUDA-enable image produced with this
repository. We will build a sample provided along with CUDA dev packages. The
following dockerfile is available in Dockerfile.test
:
# GLOBAL ARGS
ARG CUDA_BASE_IMAGE=tegra-cuda-base
ARG TAG_BUILD=build
ARG TAG_RUN=run
ARG CUDA_SAMPLE_DIR=/usr/local/cuda/samples/
# BUILD STAGE -----------------------------------------------------------------
FROM ${CUDA_BASE_IMAGE}:${TAG_BUILD} AS builder
ARG CUDA_SAMPLE_DIR
ARG TARGETS="."
RUN export CUDA_VERSION=`cat /usr/local/cuda/version.txt \
| cut -d " " -f3 \
| cut -d "." -f1,2` \
&& echo "Build for CUDA $CUDA_VERSION" \
&& cd ${CUDA_SAMPLE_DIR} \
&& echo Build following projects: `\
find ${TARGETS} -name Makefile \
| sed "s/Makefile//g" \
| sed "s/\.\///g"`\
&& PROJECTS=` find ${TARGETS} -name Makefile \
| sed "s/Makefile//g" \
| sed "s/\.\///g" ` \
make -j `nproc`
# RUN STAGE -------------------------------------------------------------------
FROM ${CUDA_BASE_IMAGE}:${TAG_RUN} as tester
ARG CUDA_SAMPLE_DIR
ARG SAMPLE_DEST=/compiled-samples
WORKDIR ${SAMPLE_DEST}
COPY --from=builder ${CUDA_SAMPLE_DIR}/bin/aarch64/linux/release/ ${SAMPLE_DEST}
CMD [ "/bin/bash" ]
The build argument TARGETS
allows selecting which samples in the directory
/usr/local/cuda/samples
you want to build:
- Set it to
""
(<empty string>) or"."
to build all the samples. - To build a specific sample, set it to the relative path of the directory of
the desired sample. E.g.:
"1_Utilities/deviceQuery"
. - You can also specify a folder to build all the subprojects inside. E.g.:
"1_Utilities"
- To build multiple directories or samples, specify them in a single string.
Folders names must be separated by a space
" "
. E.g.:"1_Utilities 5_Simulations"
.
Note: You can retrieve the list of available samples by running the build container as follows:
$ docker run --rm tegra-cuda-base:build find /usr/local/cuda/samples -name Makefile
$ docker build \
--build-arg CUDA_BASE_IMAGE=tegra-cuda-base \
--build-arg TAG_BUILD=build \
--build-arg TAG_RUN=run \
--build-arg TARGETS="1_Utilities/deviceQuery" \
--build-arg SAMPLE_DEST="/compiled-samples" \
-t cuda-test \
-f Dockerfile.test .
$ docker run --rm -it --privileged cuda-test
# Once in the container, run the sample:
root@86534cd14e9b:/compiled-sample# ./deviceQuery
Or the lazy way:
$ docker run --rm -it --privileged cuda-test /compiled-samples/deviceQuery
You should obtain a similar output on a Jetson AGX Xavier
./deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 1 CUDA Capable device(s)
Device 0: "Xavier"
CUDA Driver Version / Runtime Version 10.0 / 10.0
CUDA Capability Major/Minor version number: 7.2
Total amount of global memory: 15692 MBytes (16454631424 bytes)
( 8) Multiprocessors, ( 64) CUDA Cores/MP: 512 CUDA Cores
GPU Max Clock rate: 1500 MHz (1.50 GHz)
Memory Clock rate: 1377 Mhz
Memory Bus Width: 256-bit
L2 Cache Size: 524288 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(32768, 32768), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 1 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: Yes
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device supports Compute Preemption: Yes
Supports Cooperative Kernel Launch: Yes
Supports MultiDevice Co-op Kernel Launch: Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 0 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 10.0, CUDA Runtime Version = 10.0, NumDevs = 1
Result = PASS
Note:
--privileged
is used to grant access to GPU devices on the host. It is fine to use it for testing but may not be suitable for a released app. Check Docker privileged and capabilities docs for more details.