Monorepo (single or "mono" repository) for deep learning.
Create a directory for a virtual environment:
/InServiceOfX$ python3 -m venv ./venv/
Activate it:
/InServiceOfX$ source ./venv/bin/activate
You should see the prompt have a prefix (venv)
.
Deactivate it:
deactivate
From the "base directory" of this repository, you may run the Python unit tests and integration tests as follows:
$ pytest ./ThirdParty/NeuralOperators
This is a specific example (you may change it) illustrating running the integration tests for neuraloperators
, and it seems pytest will recursively run all the tests it detects in all the subdirectories.
Requiring manual effort:
- edit
/etc/docker/daemon.json
for place to put Docker containers, in field "data-root"
docker info | grep 'Docker Root Dir'
Example /etc/docker/daemon.json
{
"data-root": "/mnt/ey/docker",
"runtimes": {
"nvidia": {
"args": [],
"path": "nvidia-container-runtime"
}
}
}
However, read about the difference between storage driver "overlay2" for Docker vs. storage driver "vfs":
https://docs.docker.com/storage/storagedriver/overlayfs-driver/ https://docs.docker.com/storage/storagedriver/vfs-driver/
vfs is very slow compared to overlayfs because vfs does a deep copy (of each layer). If the filesystem mounted is virtiofs type, then if you try to force "storage-driver": "overlay2", it will return error, for instance
Aug 01 08:03:32 ernest-yeung dockerd[18151]: time="2024-08-01T08:03:32.397154488Z" level=error msg="failed to mount overlay: invalid argument" storage-driver=overlay2
Aug 01 08:03:32 ernest-yeung dockerd[18151]: failed to start daemon: error initializing graphdriver: driver not supported: overlay2
Aug 01 08:03:32 ernest-yeung systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
When starting and stopping Docker services, don't do sudo systemctl stop docker.socket, you'll need root access to restart it again. When I inadvertently did so, what I ended up doing was deleting the virtual machine and starting over again. https://www.ibm.com/docs/en/z-logdata-analytics/5.1.0?topic=software-relocating-docker-root-directory
- Register ssh keys to github:
In SetupVirtualMachine.sh
script
- git lfs install
- manage Docker as a non-root user, so that running docker doesn't require sudo.
- Install nvidia-container-toolkit as 1 of 3 prerequisites for using NVIDIA Pytorch Docker.
- Let's follow https://docs.nvidia.com/deeplearning/frameworks/user-guide/index.html#keyconcepts and do a
docker pull
on the Docker container we'll build on top of, from NVIDIA Container Pytorch. *