balena-io-experimental / balena-gitlab-runner Goto Github PK

So replace the defaults in the settings, maybe people want to do some stuff differently.

Add env var to add extra flags

Clean the helper images as well, but:

• check if running image is not cleared, as I think that was an issue before, or
• use a separate docker-gc config for that, which excludes different files, and keeps at least one of the runners, which should be the latest.

Without this, software update on the devices, without garbage collection, can mean that gitlab-runner-helpers take up all the space...

Which can also be a balena device ;)

The variable for listen_address (see the gitlab-runner config docs) set from an env var?

Containers seem to be cleaned, but have leftover catching containers. Will need to try whether the docker images are cleaned.

Can use it automatically? Or describe it here how to set it up properly?

Hosted at https://hub.docker.com/r/imrehg/gitlab-runner-helper and should have some flags or other ways of adding it, not sure how did I do it before, if I didn't do it with hands.... https://docs.gitlab.com/runner/configuration/advanced-configuration.html#helper-image

Not sure what the healthcheck should be:

• if no config, then fine
• if config, than check that runners are running

... or think about what else could be checked

root@74d36fbe1180:/usr/src/app# balena-engine
Illegal instruction (core dumped)

The wrong version is installed?

Flowdock

Currently gitlab-runner reports developer verison, should figure out how to change that...

Make sure that signals are catched fine, so the services are smooth-stopped whenever possible.

Due to balena-io-library/base-images#511

Runner cache

• should not ever clear the cache containers for the currently configured runners (it will result in runner error)
• should, however clear containers when the runner has been reconfigured, and the containers are orphan, so to say.

This can be achieved by reading the relevant config.toml and parse the runner names?

Or run a cleanup before doing the runner registration? (edge cases?)

Runner helper

Connects to #22, here probably could use a cleanup which only keeps the most recent image for every repo. Thus could have a schedule of:

• normal cleaning of containers, skipping runner helpers
• once a day (?) cleaning that keeps 1 most recent images (likely that's the way? Unless the runners can be downgraded, in which case it would clear the wrong helper....) Can we parse this info from the config.toml (helper_image, when it's set) and maybe gitlab-runner output (when it's automatic)?
• Or could we do a general cleanup, that only targets the helper images, and skips everything else?

The runners can do prometheus, and that's great for devops, most likely. Does it make sense to run a prometheus on the device on the side (and some other prometheus instance gathering stuff?)

Some things to answer (not exhaustive):

• How to set up? Initial setup and expansion (extra runners created with gitlab-runner register
• Limitations (architecture? balena?)
• Full example project link and walkthrough

Not sure if caching works currently, maybe have to make /cache into a volume? Or does it even work for our setup?

In case running on an older device types, which doesn't have the balena env vars. Or wait until everyone's updated to OS version that has the right env vars?

Have some general logic (well documented!), that people can define application env vars, and new devices connected will automatically be added as runners somewhere

• default runner name (device short uuid?)
• architecture tags: yes/no
• default image: architecture mapping?
• other tags (balenaCloud? / openBalena)
• device type tags?

Document the fact that these are at initiation, not taking effect if already have configuration (ie. cannot do dynamic setup from env vars)

Also, would be nice to have a better format for boolean tags (ie. be able to use yes / no, in lower/uppercase, or 1 / 0)

A project that:

• can be pushed to Gitlab or to a Gitlab instance directly
• can use architecture tags
• by default build and test successfully (without changes, so easy quickstart)

Current examples from Gitlab to get inspired by: https://docs.gitlab.com/ee/ci/examples/README.html
Notables:

• Python: https://docs.gitlab.com/ee/ci/examples/test-and-deploy-python-application-to-heroku.html (no arm, doesn't work by default as it needs heroku)
• Debian: https://about.gitlab.com/2016/10/12/automated-debian-package-build-with-gitlab-ci/ (migth be interesting with artifacts

Running the latest code, only changes are environment variables. Things run okay on gitlab.com, but on our self-hosted instance, the runner does it's thing for about 20 minutes, then starts trying to hit 127.0.0.11:53 instead and doesn't recover. If you reboot the pi, you get another 20 minutes of normal operation before it reverts to this behavior. On the server side, there are no errors, it just stops hearing from the runner.

I've tried this on wired, wireless, changed raspis, tried the staging instance of our self-hosted gitlab. Only time it gets "fixed" is if I go back to using the gitlab.com instead of our self-hosted version.

14.08.20 11:05:58 (-0500) Feeding runners to channel builds=0
14.08.20 11:05:58 (-0500) Checking for jobs... nothing runner=FH_HhUcz
14.08.20 11:06:01 (-0500) Feeding runners to channel builds=0
14.08.20 11:06:01 (-0500) Checking for jobs... nothing runner=FH_HhUcz
14.08.20 11:06:04 (-0500) Feeding runners to channel builds=0
14.08.20 11:06:04 (-0500) Checking for jobs... nothing runner=FH_HhUcz
14.08.20 11:06:07 (-0500) Feeding runners to channel builds=0
14.08.20 11:06:07 (-0500) Checking for jobs... nothing runner=FH_HhUcz
14.08.20 11:06:10 (-0500) Feeding runners to channel builds=0
14.08.20 11:06:10 (-0500) Checking for jobs... nothing runner=FH_HhUcz
14.08.20 11:06:13 (-0500) Feeding runners to channel builds=0
14.08.20 11:06:13 (-0500) Checking for jobs... nothing runner=FH_HhUcz
14.08.20 11:06:16 (-0500) Feeding runners to channel builds=0
14.08.20 11:06:16 (-0500) Dialing: tcp redacted.com:443 ...
14.08.20 11:06:16 (-0500) WARNING: Checking for jobs... failed runner=FH_HhUcz status=couldn't execute POST against https://redacted.com/api/v4/jobs/request: Post https://redacted.com/api/v4/jobs/request: dial tcp: lookup redacted.com on 127.0.0.11:53: read udp 127.0.0.1:36144->127.0.0.11:53: read: connection refused
14.08.20 11:06:19 (-0500) Feeding runners to channel builds=0
14.08.20 11:06:19 (-0500) Dialing: tcp redacted.com:443 ...
14.08.20 11:06:19 (-0500) WARNING: Checking for jobs... failed runner=FH_HhUcz status=couldn't execute POST against https://redacted.com/api/v4/jobs/request: Post https://redacted.com/api/v4/jobs/request: dial tcp: lookup redacted.com on 127.0.0.11:53: read udp 127.0.0.1:59394->127.0.0.11:53: read: connection refused
14.08.20 11:06:22 (-0500) Feeding runners to channel builds=0
14.08.20 11:06:22 (-0500) Dialing: tcp redacted.com:443 ...
14.08.20 11:06:22 (-0500) WARNING: Checking for jobs... failed runner=FH_HhUcz status=couldn't execute POST against https://redacted.com/api/v4/jobs/request: Post https://redacted.com/api/v4/jobs/request: dial tcp: lookup redacted.com on 127.0.0.11:53: read udp 127.0.0.1:34104->127.0.0.11:53: read: connection refused
14.08.20 11:06:25 (-0500) Feeding runners to channel builds=0
14.08.20 11:06:25 (-0500) Dialing: tcp redacted.com:443 ...
14.08.20 11:06:25 (-0500) WARNING: Checking for jobs... failed runner=FH_HhUcz status=couldn't execute POST against https://redacted.com/api/v4/jobs/request: Post https://redacted.com/api/v4/jobs/request: dial tcp: lookup redacted.com on 127.0.0.11:53: read udp 127.0.0.1:58683->127.0.0.11:53: read: connection refused

Including:

• amd64
• i386 (optional?)
• i386-nlp (optional?)
• armv7
• armv6 (Pi1/PiZero)
• aarch64

Some architectures (i386*, I guess?) might need creation of gitlub-runner-helper images, FYI.

E.g. if BALENA_... is not available, don't use them

When runners are switched, there might be old runner containers left around, they should be cleaned up, but the name match depends on the current runner configuration, so needs some smarts. Also runner_ seem to match runner- as well, maybe.... Other container cleannup would be good too, to make it more hands off...

Currently compiled and released to imrehg/gitlab-runner-helper, but needs testing, and needs upstreaming.