python3 /home/vagrawal/cluster-jobs/create_job.py --script /home/vagrawal/cluster-jobs/run.sh --template /home/vagrawal/cluster-jobs/template.sh --dir /home/vagrawal/ --gpu 2 --mem 64 --cpu 1 --array --pull --gpu-names V100-32 --bid 1000 --avoid g048
There are three folders of interest. (There could be more if your codebase is on multiple repos)
- ~/cluster-jobs
- ~/your_repo(s)_which_contains_your_code_to_run_and_data
- ~/jobs (if this doesn't exist, mkdir it)
- Everytime you submit a job (i.e. python3 create_job.py …..), a new subfolder is created inside ~/jobs/
- Each of those subfolders contains a few files.
- stdout-idx.txt : this contains the terminal output that you would get if you would run the "python your_main_file.py" file in local machine (used for debugging)
- “cluster-jobs”
- git pull all the latest changes (on the cluster)
- Before submitting any job, make sure you have a “~/jobs” folder on your cluster
- This contains the actual files that you need to run on cluster
- git pull all the repos that you need. This would include all the data and files that you need to run.
- (Let's say) Path where you cloned this repo in the cluster: ~/your_repo(s)_which_contains_your_code_to_run_and_data
In cluster-jobs folder, you have three files.
- run.sh contains the python commands to run ("unbuffer" is optional) → (unbuffer) python file_name.py
- template.sh contains the new and updated/modified environment variables.
- create_job.py this is the submission script. it's responsible for creating jobs and submitting them. It takes each line from run.sh and create a job for it. This also pulls all the git-repos that you mention here before submitting your jobs.
You just have to run this create_job.py file, with the required arguments to handle everything. It runs on python3.
To run this submission script, this sample command could be
python3 /home/vagrawal/cluster-jobs/create_job.py --script /home/vagrawal/cluster-jobs/run.sh --template /home/vagrawal/cluster-jobs/template.sh --dir /home/vagrawal/ --gpu 2 --mem 64 --cpu 4 --array --pull --gpu-names V100-32 --bid 1800 --avoid g048
- --dir the jobs are submitted from this folder. This becomes the "pwd" (present working directory).
For information on other tags, check out the argument parser in create_job.py
If your project needs to activate a virtual environment, it is suggested to activate it before running the submission script. (create_job.py)
To login to the cluster, $ssh [email protected]
To load cuda, do this from inside the cluster: $module load cuda/10.2
You need to load these modules every time you open a new terminal and activate your env.