I would expect the cluster to be fully configured the same way that the clusters created with blueprints would be.
Terraform deploys correctly, the login and controller nodes are created but slurm fails to initialize. "By default each cluster creates two shared filesystems: one at /opt/cluster to hold installed applications and one at /home to hold job files for individual users." These default filestore instances are not created and slurm fails to initialize during initialization.
If applicable, attach or paste the blueprint YAML used to produce the bug.
If applicable, please attach or paste the expanded blueprint. The expanded blueprint can be obtained by running ghpc expand your-blueprint.yaml
.
2023-04-17 22:39:48,078 DEBUG: fetch_devel_scripts: scripts not found in project metadata, devel mode not enabled
2023-04-17 22:39:48,082 INFO: Setting up login
2023-04-17 22:39:48,088 INFO: installing custom scripts: login_4nx80qw2.d/ghpc_startup.sh
2023-04-17 22:39:48,089 DEBUG: install_custom_scripts: login_4nx80qw2.d/ghpc_startup.sh
2023-04-17 22:39:48,092 INFO: Set up network storage
2023-04-17 22:39:48,129 INFO: Setting up mount (nfs) alpha9ee24-controller:/home to /home
2023-04-17 22:39:48,130 INFO: Setting up mount (nfs) alpha9ee24-controller:/opt/apps to /opt/apps
2023-04-17 22:39:48,130 INFO: Setting up mount (nfs) alpha9ee24-controller:/opt/cluster to /opt/cluster
2023-04-17 22:39:48,172 DEBUG: <module>: disabling prometheus support
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/more_executors/_impl/metrics/__init__.py", line 18, in <module>
from .prometheus import PrometheusMetrics
File "/usr/local/lib/python3.6/site-packages/more_executors/_impl/metrics/prometheus.py", line 3, in <module>
import prometheus_client # pylint: disable=import-error
ModuleNotFoundError: No module named 'prometheus_client'
2023-04-17 22:39:48,383 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:39:48,386 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:39:48,392 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:39:48,602 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:39:48,605 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:39:48,607 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:39:49,603 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:39:49,607 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:39:49,611 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:39:49,630 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:39:49,636 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:39:49,643 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:39:51,231 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:39:51,237 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:39:51,243 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:39:51,256 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:39:51,262 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:39:51,268 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:39:53,817 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:39:53,822 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:39:53,828 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:39:53,842 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:39:53,849 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:39:53,855 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:39:57,939 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:39:57,946 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:39:57,952 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:39:57,965 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:39:58,973 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:39:58,980 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:40:04,519 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:40:04,538 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:40:05,528 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:40:05,535 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:40:05,549 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:40:05,555 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:40:15,025 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:40:15,043 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:40:16,035 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:40:16,041 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:40:16,055 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:40:16,062 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:40:31,044 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:40:31,066 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:40:32,056 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:40:32,063 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:40:32,076 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:40:32,083 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:40:47,067 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:40:47,086 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:40:48,077 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:40:48,083 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:40:48,097 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:40:48,104 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:41:03,087 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:41:03,108 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:41:04,098 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:41:04,104 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:41:04,122 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:41:04,129 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:41:19,109 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:41:19,130 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:41:20,123 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:41:20,130 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:41:20,143 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:41:20,150 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:41:35,130 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:41:35,151 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:41:36,143 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:41:36,150 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:41:36,165 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:41:36,172 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:41:51,152 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:41:51,171 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:41:52,166 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:41:52,172 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:41:52,188 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:41:52,195 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:42:07,172 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:42:07,193 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:42:08,189 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:42:08,196 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:42:08,211 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:42:08,217 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:42:23,194 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:42:23,214 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:42:24,212 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:42:24,218 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:42:24,233 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:42:24,239 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:42:39,215 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:42:39,234 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:42:40,234 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:42:40,240 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:42:40,255 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:42:40,262 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:42:55,235 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:42:55,257 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:42:56,256 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:42:56,263 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:42:56,279 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:42:56,287 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:43:11,258 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:43:11,282 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:43:12,280 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:43:12,288 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:43:12,300 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:43:12,308 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:43:27,283 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:43:27,305 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:43:28,302 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:43:28,308 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:43:28,322 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:43:28,329 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:43:43,306 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:43:43,327 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:43:44,323 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:43:44,329 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:43:44,345 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:43:44,352 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:43:59,328 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:43:59,348 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:44:00,346 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:44:00,353 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:44:00,365 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:44:00,371 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:44:15,349 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:44:15,368 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:44:16,366 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:44:16,372 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:44:16,387 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:44:16,394 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:44:31,369 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:44:31,389 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:44:32,387 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:44:32,395 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:44:32,409 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:44:32,416 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:44:47,390 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:44:47,410 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:44:48,410 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:44:48,416 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:44:48,435 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:44:48,442 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:45:03,411 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:45:03,431 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:45:04,436 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:45:04,443 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:45:04,458 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:45:04,465 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:45:19,432 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:45:19,454 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:45:20,459 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:45:20,466 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:45:20,487 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:45:20,494 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:45:35,455 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:45:35,475 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:45:36,488 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:45:36,494 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:45:36,508 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:45:36,515 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:45:51,476 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:45:51,500 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:45:52,509 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:45:52,515 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:45:52,531 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:45:52,538 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:46:07,501 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:46:07,525 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:46:08,532 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:46:08,539 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:46:08,551 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:46:08,557 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:46:23,526 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:46:23,547 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:46:24,552 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:46:24,558 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:46:24,576 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:46:24,583 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:46:39,548 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:46:39,569 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:46:40,577 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:46:40,583 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:46:40,598 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:46:40,605 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:46:55,570 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:46:55,592 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:46:56,599 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:46:56,605 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:46:56,621 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:46:56,627 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:47:11,593 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:47:11,613 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:47:12,622 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:47:12,627 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:47:12,644 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:47:12,651 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:47:27,613 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:47:27,635 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:47:28,645 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:47:28,652 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:47:28,667 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:47:28,675 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:47:43,636 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:47:43,658 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:47:44,668 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:47:44,676 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:47:44,688 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:47:44,695 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:47:59,659 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:47:59,680 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:48:00,689 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:48:00,695 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:48:00,711 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:48:00,717 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:48:15,681 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:48:15,701 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:48:16,712 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:48:16,718 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:48:16,734 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:48:16,739 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:48:31,702 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:48:31,722 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:48:32,735 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:48:32,740 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:48:32,756 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:48:32,764 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:48:47,723 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:48:47,743 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:48:48,757 INFO: Waiting for '/home' to be mounted...
2023-04-17 22:48:48,765 INFO: Waiting for '/opt/apps' to be mounted...
2023-04-17 22:48:48,777 ERROR: mount of path '/home' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/home']' returned non-zero exit status 32.
2023-04-17 22:48:48,784 ERROR: mount of path '/opt/apps' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/apps']' returned non-zero exit status 32.
2023-04-17 22:49:03,744 INFO: Waiting for '/opt/cluster' to be mounted...
2023-04-17 22:49:03,764 ERROR: mount of path '/opt/cluster' failed: <class 'subprocess.CalledProcessError'>: Command '['mount', '/opt/cluster']' returned non-zero exit status 32.
2023-04-17 22:49:03,766 ERROR: CalledProcessError:
command=['mount', '/opt/cluster']
returncode=32
stdout:
stderr:
mount.nfs: access denied by server while mounting alpha9ee24-controller:/opt/cluster
2023-04-17 22:49:03,767 ERROR: Aborting setup...