Adding partition causes the entire cluster to fail due to failures in `/slurm/scripts/setup.py` about hpc-toolkit HOT 3 CLOSED

I tried it again and it launched, so I may be missing a step, will reopen if I see this again

from hpc-toolkit.

I was unable to reproduce the behavior described.

Since the blueprint provided references a custom built image I had to make some changes to the blueprint to test.
Here is the blueprint I tested with. Changes from the blueprint provided:

• Use base debian 11 slurm image in place of custom image
• Remove two startup scripts that were not used by any modules

I have deployed the the blueprint, uncommented the new partition (including use on the controller), and redeployed the blueprint. I tried this several times and did not see any nodes on the tiny partition go down. I also tried it while a job was running on the tiny partition.

Are there any other actions that you are taking that could be causing the nodes on the existing partition to fail? Is this behavior consistently reproducible for you? Are you able to reproduce the behavior with the blueprint linked in this comment?

Note: I will note that if a node is already up and I run /slurm/scripts/setup.py I do see it fail due to attempting to remount previously mounted volumes. But this only happens when I manually call setup.py and does not occur when I re-deploy the cluster with a new partition.

from hpc-toolkit.

Thanks for testing. Two potential differences are 1. different ghpc toolkit version (I installed it a month ago) 2. This cluster has been reconfigured several times before this. I will reproducing again with new ghpc clone and new deployment to rule out those

from hpc-toolkit.