A docker tool that helps put machine learning in places that empower ml developers

MLDOCK builds in conveniences and the power of docker and frames it around the core machine learning tasks related to production.

As a tool this means MLDOCK's goals are:

• Provide tooling to improve the ML development workflow. ✅
• Enable portability of ml code betwen platforms and vendors (Sagemaker, AI Platform, Kubernetes, other container services). ✅
• Lower the barrier to entry by developing containers from templates. ✅
• Be ready out the box, using templates to get you started quickly. Bring only your code. ✅
• For any ML frameworks, runs in any orchestrator and on any cloud. (as long as it integrates with docker) ✅

What it is not:

• Service orchestrator ❌
• Training Scheduler ❌
• Hyperparameter tuner ❌
• Experiment Tracking ❌

Inspired by Sagify, Sagemaker Training Toolkit and Amazon Sagemaker.

1. (Optional) Use virtual environment to manage dependencies.
2. Install dotenv easily configure environment.

pip install --user python-dotenv[cli]

note: dotenv allows configuring of environment through the .env file. MLDOCK uses ENVIRONMENT VARIABLES in the environment to find your DOCKER_HOST, DOCKERHUB credentials and even AWS/GCP credentials.

3. Create an .env with the following:

# for windows and if you are using WSL1
DOCKER_HOST=tcp://127.0.0.1

# for WSL2 and linux (this is default and should work out of the box)
# but for consistency, set this dockerhost

DOCKER_HOST=unix://var/run/docker.sock

note: Now to switch environments just use dotenv as follows:

dotenv -f "/path/to/.env" run mldock local build --dir <my-project-path>

The MLDOCK command line utility provides a set of commands to streamline the machine learning container image development process. The commands are grouped in to 3 functionality sets, namely:

1. Install MLDOCK

The pip install is the only supported package manager at present. It is recommended that you use an environment manager, either virtualenv or conda will work.

pip install mldock[cli]

2. Setup local config for the mldock cli

This command sets up mldock cli with some nice to have defaults. It may even prompt you for some set up.

mldock configure init

3. Initialize or create your first container

You will see a some of prompts to set up container.

mldock project init --dir my_ml_container

note:

• Just hit Return/Enter to accept all the defaults.

4. Build your container image locally

mldock local build --dir my_ml_container

5. Run your training locally

mldock local train --dir my_ml_container

6. Run your training locally

mldock local deploy --dir my_ml_container

1. Add the following to .env

DOCKERHUB_USERNAME=<your/user/name>
DOCKERHUB_PASSWORD=<your/dockerhub/password>
DOCKERHUB_REGISTRY=https://index.docker.io/v1/
DOCKERHUB_REPO=<your/user/repo/name>

2. Push your container to dockerhub

mldock registry push --dir my_ml_container --provider dockerhub --build

note: The flags allow you to stipulate configuration changes in the command. --build says build the image before pushing. This is required initially since the dockerhub registry will prefix your container name. --provider tells MLDOCK to authenticate to dockerhub and push the container there.

hint In addition to DockerHub, both AWS ECR & GCP GCR are also supported.

• docker compose sees my files as directories in mounted volume - USE "./path/to/file" format | https://stackoverflow.com/questions/42248198/how-to-mount-a-single-file-in-a-volume
• simlinks from my container have broken permissions in WSL2 | microsoft/WSL#1475