acm-uiuc / clearwaters-container Goto Github PK

Making a REST endpoint to create/delete new docker containers. Similar to gpu-cluster-backend.

A worker daemon will be created for monitoring the status of worker and executing tasks on the worker. The daemon should be able to complete the following tasks at a minimum.

• Create a docker container
• Shutdown a docker container
• Report which containers are running

Ideally the worker daemon has a config file located somewhere in /etc/ and runs as a service, so that if a worker crashes, it can immediately be restarted.

As stated in #2 , there will exist worker daemons for managing each worker locally, however a head-node manager must exist as well. The head node manage will do the following tasks:

• Keep a list of worker nodes
• Send message to worker daemons for creating/destroying containers
• Keep a list of container swarms

A communication protocol will need to be decided between the workers and the head-node .

Before we can implement the scheduler in code, we should decide on how the jobs will be scheduled.

We have a few options, based on current schedulers, such as round robin, CFS, or even FIFO. However, since we will most likely have a job queue and also are running on multiple systems, we will need to ensure that all tasks are run fairly and also that workers are not overloaded with tasks while some do no work at all.

Make sure the round robin algorithms labels each container formed with an id and name and adds to a log and also documents the size of f for simulation purposes before it actually goes on the cluster

What we want to do is create containers on all of our worker nodes with a single command from the head node.

The container image we are using is:
registry.gitlab.com/acm-uiuc/sigops/clearwaters-docker/ubuntu-mpich-arm64
Which can be grabbed using:
docker pull registry.gitlab.com/acm-uiuc/sigops/clearwaters-docker/ubuntu-mpich-arm64

I imagine a very simple workflow being the head-node sending a message to all client daemons to create the containers, and the client daemons then responding with the IDs of the created containers.

Its lacking at the moment