Quack,

Thanks for you nice work. I'm using a custom running to run tests that require an init system. It would be handy if we could override the command in custom_base.sh. I could prepare a PR if you're ok with this proposal.

Regards.
\_o<

Is there a way to set up "podman as a custom executor" in the image docker.io/gitlab/gitlab-runner:alpine?

Something such as: https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#use-the-docker-executor-with-the-docker-image-docker-in-docker

I'm trying to contribute a pull if-not-present policy for this runner similar to this docker runner option, but I can't stop podman from pulling.

In prepare.sh

#### Attempting to change this to enable the same as a "pull if-not-present" policy
    podman image ls   # shows that image is available
#    podman pull --authfile "$CACHE_DIR"/_authfile_"$CONTAINER_ID" "$IMAGE"
#    rm "$CACHE_DIR"/_authfile_"$CONTAI
    podman run \
        --detach \
        --name "$CONTAINER_ID" \
        --volume "$CACHE_DIR:/home/user/cache":Z \
        --pull=missing \  ## Added this line although missing is the default
        "${PODMAN_RUN_ARGS[@]}" \
        "$IMAGE"\
        sleep 999999999

In cleanup.sh:

# Try to remove all old containers, images, networks, and volumes
## this is killing the cache, need some way to maintain a cache. For now prune manually.
#podman system prune --force --volumes

But when I run another job, podman reaches out and pulls the container again. What am I missing? Is this a feasible contribution?

It looks like cache isn't working as expected with gitlab-runner custom executor to podman.

With the following .gitlab-ci.yml (that works with a plain docker executor), Job B fails because cat command cannot find hello.txt file that sould have been restored from cache.

default:
  image: debian:buster
  tags:
    - podman

stages:
  - build
  - test

job A:
  stage: build
  script:
    - mkdir -p vendor/
    - echo "build" > vendor/hello.txt
  cache:
    key: build-cache
    paths:
      - vendor/

job B:
  stage: test
  script:
    - cat vendor/hello.txt
  cache:
    key: build-cache
    paths:                                                                                                            
      - vendor/
    policy: pull

Here is (part of) my config.toml

[[runners]]
  name = "pleiades-ci podman runner"
  url = "https://[masked]"
  token = "[masked]"
  executor = "custom"
  builds_dir = "/builds"
  cache_dir = "/cache"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.custom]

    prepare_exec = "/home/me/bin2/prepare.sh"
    prepare_exec_timeout = 1800

    run_exec = "/home/me/bin2/run.sh"

    cleanup_exec = "/home/me/bin2/cleanup.sh"
    cleanup_exec_timeout = 300

    graceful_kill_timeout = 60
    force_kill_timeout = 180

Here is the output of the failling job :

Setting up git-lfs (2.7.1-1+deb10u1) ...
Preparing environment 00:00
Running on 7a72d7ad609c...
Getting source from Git repository 00:01
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/DEC/pleiades/demo-ci-cd/.git/
Created fresh repository.
Checking out 5c308f92 as test-cache-podman...
Skipping Git submodules setup
Restoring cache 00:00
Checking cache for build-cache...
Runtime platform                                    arch=amd64 os=linux pid=3635 revision=58ba2b95 version=14.2.0
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache
Executing "step_script" stage of the job script 00:00
WARNING: Starting with version 14.0 the 'build_script' stage will be replaced with 'step_script': https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26426
$ cat vendor/hello.txt
cat: vendor/hello.txt: No such file or directory
Cleaning up file based variables 00:01
ERROR: Job failed: exit status 1

Am i missing something ?

I think the install_command test could be improved as the packages listed under the dependencies don't all contain binaries that are equal to the name of the package.
The ca-certificates package for example does not contain any binary called ca-certificates (at least not on archlinux/debian/redhat) which causes the ca-certificates package to be installed in every job.
This can count for a significant amount of time, especially on yum based distros that have a lot of repos enabled.

Possible solutions:

• Adding a second argument to install_command for a file to check (or perhaps a command to run).
• Check if a package is installed instead of relying on the existence of a certain file/binary. This would need to be distro specific but it would be possible to use dpkg/rpm/etc. instead of apt/yum/dnf (using rpm is significantly faster than relying on yum/dnf).

Relevant issues for Gitlab

• https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4358
• https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1827

Podman 1.9 fixed the stdin truncation bug. This code can now be changed to be more idiomatic

podman exec "$CONTAINER_ID" /bin/bash < "$1"

Preparing the "custom" executor
Using Custom executor...
Running in runner-152-project-2683-concurrent-0-66548
Login to  with CI_REGISTRY_USER
Error: authenticating creds for "": error pinging docker registry : Get "https:///v2/": http: no Host in request URL
ERROR: Preparation failed: exit status 2

You can see that a podman login is tried with an empty registry address. I don't know why CUSTOM_ENV_CI_REGISTRY_USER and CUSTOM_ENV_CI_REGISTRY_PASSWORD are set while CUSTOM_ENV_CI_REGISTRY is not, but that's what my university's GitLab instance is dealing me.

A pull request is on the way.

Besides the currently supported DOCKER_AUTH_CONFIG there are multiple other ways to authenticate against the Gitlab registry. The ways are listed here.

The script should try all logins until one is working. The order should be:

1. DOCKER_AUTH_CONFIG This is a very specific configuration not only for the Gitlab Registry but for all, thus it should have the highest priority.
2. CI_DEPLOY_USER Deploy tokens do not exist by default, thus if they are created manually they should have a high priority.
3. CI_JOB_USER Provided automatically, thus low priority.
4. CI_REGISTRY_USER Provided automatically, outdated version of CI_JOB_USER thus lowest priority.

https://docs.gitlab.com/ee/user/packages/container_registry/index.html#authenticate-by-using-gitlab-cicd

Hi, I am using this executor but it seems to be impossible to change behaviour of exit code. The problem I am facing is that when I execute som .sh script on runner which should exit with status code 22 the final status code is always 1 anyway.

In this case i am not able to use allow_failure in my .gitlab-ci.yml file.

  allow_failure:
    exit_codes:
      - 22

I found that there is variable "$BUILD_FAILURE_EXIT_CODE" in run.sh script but where is this coming from ?

Is there any possibility to change this behaviour ?

Thank you very much.