Comments (3)
I tried it again and it launched, so I may be missing a step, will reopen if I see this again
from hpc-toolkit.
I was unable to reproduce the behavior described.
Since the blueprint provided references a custom built image I had to make some changes to the blueprint to test.
Here is the blueprint I tested with. Changes from the blueprint provided:
- Use base debian 11 slurm image in place of custom image
- Remove two startup scripts that were not used by any modules
I have deployed the the blueprint, uncommented the new partition (including use on the controller), and redeployed the blueprint. I tried this several times and did not see any nodes on the tiny partition go down. I also tried it while a job was running on the tiny partition.
Are there any other actions that you are taking that could be causing the nodes on the existing partition to fail? Is this behavior consistently reproducible for you? Are you able to reproduce the behavior with the blueprint linked in this comment?
Note: I will note that if a node is already up and I run /slurm/scripts/setup.py
I do see it fail due to attempting to remount previously mounted volumes. But this only happens when I manually call setup.py
and does not occur when I re-deploy the cluster with a new partition.
from hpc-toolkit.
Thanks for testing. Two potential differences are 1. different ghpc toolkit version (I installed it a month ago) 2. This cluster has been reconfigured several times before this. I will reproducing again with new ghpc clone and new deployment to rule out those
from hpc-toolkit.
Related Issues (20)
- How to use image-builder.yaml to install a docker image to template VM HOT 31
- SLURM 1.20 deployed and having node creation error HOT 40
- Slurm setup fails in deployed blueprint - possible error getting metadata HOT 7
- HPC toolkit no longer works with a2 instances HOT 6
- Partition a208 misconfigured in hpc-interprise-slurm.yaml HOT 4
- Update ml-slurm blueprint to use updated base image for schedmd debian 11 HOT 1
- Unable to dynamically modify the number of nodes in a slurm cluster HOT 2
- Slurm nodes with hybrid controller module unable to configure correctly HOT 2
- error when use packer to build image in ml-slurm HOT 2
- Unable to configure Slurm due to failure to mount filestore HOT 5
- Feature request: support `hashicorp/google` and `hashicorp/google-beta` v5
- private_vpc_connection is not unique per cluster in slurm-sql module HOT 4
- ERROR: failed to sync instances when issuing `scontrol reboot` HOT 4
- Example of startup script with cluster without vm-instance? HOT 2
- Broken link HOT 1
- PMIx MPI support in Slurm HOT 16
- IP space of [gcp project subnet] is exhausted when deploying a GCP Slurm cluster HOT 2
- Packer custom image does not use specified service account email. HOT 3
- Upgrade to Ops Agent fails HOT 6
- HTCondor tutorial: add cloudresourcemanager.googleapis.com to the list of services to enable HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hpc-toolkit.