ginkgo-project / gitlab-hpc-ci-cb Goto Github PK

Currently, the environment variables which are passed to the container are managed through the hard-coded list PROPAGATED_ENV_VARIABLES in include.sh. This contains many Ginkgo-specific variables and needs to be configurable for generic use. It's not clear yet how the configuration can be best implemented. A simple way would be a configuration file the user needs to fill in the repository.

It should be possible to add Gitlab CI with a few test cases to this tool itself so that we can ensure it still does its job when updating it without manual involvement and while not making our entire Ginkgo pipelines fail during testing.

To do so, we would need to register a test/beta runner on an HPC machine. A problem is that for now, BwUni seems to only support one runner per user (from previous tests)?

Managing the runner's script update on the server-side might be a bit more tricky without relying on ssh, maybe a simple cron job would be sufficient with a trigger from the server-side when master is updated.

Hi,

there seems to be an issue with the login to a private registry. It works with public images from docker hub.

Best,
Stefan

I'm not sure if the interactive mode is always available, some time ago I remember finding clusters where that was not the case. If it is available, the waiting loops can all be removed by simply using an interactive job (and adding exit at the end of the script?). While the job is pending, there is a built-in waiting loop. One problem is there is less feedback while waiting?

At the moment run.sh just checks the output of the error log: If this isn't empty the CI task is handled as failed. This causes that all CI pipelines using slurm are failing although the status of the job is "COMPLETED" and not "FAILED".

Can run.sh check the status code of the job instead?

Hi,

at the moment it's not possible to become root inside the container or make it writable I assume?

Best,
Stefan

The use of #!/usr/bin/bash is not portable to Ubuntu systems. Can this be changed in the various scripts to use #!/usr/bin/env bash?

Thanks

For the script side, the setup should hopefully be somewhat simple:

• change CI_WS to point to LSDF,
• add a SLURM parameter for the constraint handling

Most of the work might be in the setup and documentation step.

Hi,

what do you think about using the SPANK plugin nvidia pyxis instead of calling enroot directly?

Best,
Stefan

EDIT: Obviously this can't be used for running jobs directly on the frontend (login submit whatever) node. But anyway it's probably better to run such jobs on a dedicated node with probably less resources instead.

Gitlab runner has a number of error tracking features,

• Failures in the user's job script should raise a BUILD_FAILURE_EXIT_CODE, this should already always be the case
• Failures in the script itself, such as invalid output, connection issues, ... should raise a SYSTEM_FAILURE_EXIT_CODE, see https://docs.gitlab.com/runner/executors/custom.html#system-failure

The second one is currently missing and should be done more properly in the case where something goes wrong in the script itself.