d-hayashi / k8s-slurm-injector Goto Github PK

I was disappointed to find that all the jobs submitted by k8s-slurm-injector had been canceled

such as,
22 default-p+ batch root 2 CANCELLED+ 0:0
22.batch batch root 2 CANCELLED 0:0
22.0 env root 2 COMPLETED 0:0

Purpose

Feature request

Description

Support retrieving resource request from field spec.containers[].resources.limits.nvidia.com/gpu instead of labels.

Note

I think this can be achieved in the following way.

1. Get resource requests of GPUs from the specs
2. Remove the GPU requests/limitations from the specs
3. Inject a Slurm job
4. Get CUDA_VISIBLE_DEVICES from the Slurm job
5. Set the value as NVIDIA_VISIBLE_DEVICES in the requested containers