Git Product home page Git Product logo

balena-gitlab-runner's Introduction

gitlab-runner on balenaOS devices

GitLab runners are worker nodes to run your tests in a GitLab CI/CD environment. This project aims to help you deploy these runners on physical hardware on balenaOS devices.

Features:

  • a docker executor gitlab runner
  • automatic first runner provisioning to a gitlab instance, only need to provide a token
  • can add multiple runners to a single device manually
  • automatic docker image/container garbage collection using docker-gc by Spotify.

Discuss this project in the forums!

Getting Started

If using balenaCloud, follow the getting started guide for your desired device type, to see how to register (if you haven't yet), and how to provision a device onto balenaCloud. Afterwards deploy this project either as git push balena master or using the CLI with balena push <projectname>.

If using openbalena, follow the guides there to provision a device, and run balena push <projectname> as appropriate.

Settings

There are a number of environment variables you can customise your deployment with. These help setting up your first runner automatically:

  • GITLAB_TOKEN: required, the runner registration token, you can get the value from the Settings > CI/CD > Runners page for a particular project, or from your GitLab instance's administration page.
  • GITLAB_URL: optional, the URL where the runner should connect to. The default is value https://gitlab.com/. You can see what you need to set for a particular project at the Settings > CI/CD > Runners page.
  • GITLAB_DESCRIPTION: optional, the description added to the runner on your runner page. If left empty, defaults to the device's short UUID.
  • GITLAB_DEFAULT_IMAGE:optional, the default image to use in jobs, if the job doesn't specify the image. If left empty it defaults to balenalib/<devicetype>-debian, with the device type automatically filled in.
  • TAG_ARCHITECTURE: optional, whether or not create the runner with tag showing the architecture (as given by uname -m, such as armv7l, x64_64, etc). Left empty defaults to yes, and any other value will prevent this tag being added.
  • TAG_DEVICE_TYPE: optional, whether or not create the runner with tag showing the balena device type (i.e. raspberrypi3, intel-nuc, etc). Left empty defaults to yes, and any other value will prevent this tag being added.
  • TAG_BALENA: optional, whether or not tag the runner with the balena tag, thus showing clearly what are balena devices among your runners, if there are other types as well. If left empty, defaults to yes, and any other value will prevent this tag being added.
  • GITLAB_TAGS: optional, the list of any other tags to add (comma delimited, such as office,screen).
  • GITLAB_RUN_UNTAGGED: optional, whether or not run untagged build jobs. Left empty defaults to yes (do run untagged jobs), any other value make it run only tagged jobs (and matching tags). You can also change this on your runner configuration page.
  • GITLAB_LOCKED: optional, whether or not the runner is locked to a specific project. Left empty defaults to yes (locked to a project), any other value makes it default unlocked. You can also change this on your runner configuration page.
  • GITLAB_RUNNER_HELPER_IMAGE: optional, to set a different helper image to use with the runner. For more information, check the relevant GitLab logs. This is a registration-time setting, so changing the value won't change an already registered runner For x86 32-bit runners, GitLab doesn't provide runner helper images, but we have some custom builds available at imrehg/gitlab-runner-helper:i386-${CI_RUNNER_VERSION} and that value is automatically applied for 32-bit runners (such as Intel Edison). It's custom work based on a patched version of the upstream GitLab runners (see the i386 branch of our fork), thus your milage may vary.
  • DEBUG: optional, set it to any value to enable certain debug/diagnostics functionality, see the code in more details. One current example is keeping all garbage collection logs on the device (instead of just the last one)

See more information at the Configuring GitLab Runners docs. The configuration file is also available to edit.

Adding extra runners

For subsequent runners you will have to use the terminal to set them up. Connect to the runner service either in the web dashboard, balena ssh $UUID (on balenaCloud) or balena local ssh $UUID (on openbalena), and check out gitlab-runner register --help for options.

balena-gitlab-runner's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

balena-gitlab-runner's Issues

Add a README to this repo

Some things to answer (not exhaustive):

  • How to set up? Initial setup and expansion (extra runners created with gitlab-runner register
  • Limitations (architecture? balena?)
  • Full example project link and walkthrough

Clean gitlab-runner-helper images

Clean the helper images as well, but:

  • check if running image is not cleared, as I think that was an issue before, or
  • use a separate docker-gc config for that, which excludes different files, and keeps at least one of the runners, which should be the latest.

Without this, software update on the devices, without garbage collection, can mean that gitlab-runner-helpers take up all the space...

Runner disconnects from gitlab self hosted instance

Running the latest code, only changes are environment variables. Things run okay on gitlab.com, but on our self-hosted instance, the runner does it's thing for about 20 minutes, then starts trying to hit 127.0.0.11:53 instead and doesn't recover. If you reboot the pi, you get another 20 minutes of normal operation before it reverts to this behavior. On the server side, there are no errors, it just stops hearing from the runner.

I've tried this on wired, wireless, changed raspis, tried the staging instance of our self-hosted gitlab. Only time it gets "fixed" is if I go back to using the gitlab.com instead of our self-hosted version.

14.08.20 11:05:58 (-0500) Feeding runners to channel builds=0
14.08.20 11:05:58 (-0500) Checking for jobs... nothing runner=FH_HhUcz
14.08.20 11:06:01 (-0500) Feeding runners to channel builds=0
14.08.20 11:06:01 (-0500) Checking for jobs... nothing runner=FH_HhUcz
14.08.20 11:06:04 (-0500) Feeding runners to channel builds=0
14.08.20 11:06:04 (-0500) Checking for jobs... nothing runner=FH_HhUcz
14.08.20 11:06:07 (-0500) Feeding runners to channel builds=0
14.08.20 11:06:07 (-0500) Checking for jobs... nothing runner=FH_HhUcz
14.08.20 11:06:10 (-0500) Feeding runners to channel builds=0
14.08.20 11:06:10 (-0500) Checking for jobs... nothing runner=FH_HhUcz
14.08.20 11:06:13 (-0500) Feeding runners to channel builds=0
14.08.20 11:06:13 (-0500) Checking for jobs... nothing runner=FH_HhUcz
14.08.20 11:06:16 (-0500) Feeding runners to channel builds=0
14.08.20 11:06:16 (-0500) Dialing: tcp redacted.com:443 ...
14.08.20 11:06:16 (-0500) WARNING: Checking for jobs... failed runner=FH_HhUcz status=couldn't execute POST against https://redacted.com/api/v4/jobs/request: Post https://redacted.com/api/v4/jobs/request: dial tcp: lookup redacted.com on 127.0.0.11:53: read udp 127.0.0.1:36144->127.0.0.11:53: read: connection refused
14.08.20 11:06:19 (-0500) Feeding runners to channel builds=0
14.08.20 11:06:19 (-0500) Dialing: tcp redacted.com:443 ...
14.08.20 11:06:19 (-0500) WARNING: Checking for jobs... failed runner=FH_HhUcz status=couldn't execute POST against https://redacted.com/api/v4/jobs/request: Post https://redacted.com/api/v4/jobs/request: dial tcp: lookup redacted.com on 127.0.0.11:53: read udp 127.0.0.1:59394->127.0.0.11:53: read: connection refused
14.08.20 11:06:22 (-0500) Feeding runners to channel builds=0
14.08.20 11:06:22 (-0500) Dialing: tcp redacted.com:443 ...
14.08.20 11:06:22 (-0500) WARNING: Checking for jobs... failed runner=FH_HhUcz status=couldn't execute POST against https://redacted.com/api/v4/jobs/request: Post https://redacted.com/api/v4/jobs/request: dial tcp: lookup redacted.com on 127.0.0.11:53: read udp 127.0.0.1:34104->127.0.0.11:53: read: connection refused
14.08.20 11:06:25 (-0500) Feeding runners to channel builds=0
14.08.20 11:06:25 (-0500) Dialing: tcp redacted.com:443 ...
14.08.20 11:06:25 (-0500) WARNING: Checking for jobs... failed runner=FH_HhUcz status=couldn't execute POST against https://redacted.com/api/v4/jobs/request: Post https://redacted.com/api/v4/jobs/request: dial tcp: lookup redacted.com on 127.0.0.11:53: read udp 127.0.0.1:58683->127.0.0.11:53: read: connection refused

Fix version reporting

Currently gitlab-runner reports developer verison, should figure out how to change that...

Find ways to clean up old runner containers

When runners are switched, there might be old runner containers left around, they should be cleaned up, but the name match depends on the current runner configuration, so needs some smarts. Also runner_ seem to match runner- as well, maybe.... Other container cleannup would be good too, to make it more hands off...

Add health check

Not sure what the healthcheck should be:

  • if no config, then fine
  • if config, than check that runners are running

... or think about what else could be checked

Add default runner setup from env vars

Have some general logic (well documented!), that people can define application env vars, and new devices connected will automatically be added as runners somewhere

  • default runner name (device short uuid?)
  • architecture tags: yes/no
  • default image: architecture mapping?
  • other tags (balenaCloud? / openBalena)
  • device type tags?

Document the fact that these are at initiation, not taking effect if already have configuration (ie. cannot do dynamic setup from env vars)

Also, would be nice to have a better format for boolean tags (ie. be able to use yes / no, in lower/uppercase, or 1 / 0)

Create demo project to use this application

A project that:

  • can be pushed to Gitlab or to a Gitlab instance directly
  • can use architecture tags
  • by default build and test successfully (without changes, so easy quickstart)

Current examples from Gitlab to get inspired by: https://docs.gitlab.com/ee/ci/examples/README.html
Notables:

Dynamic list of cleared containers/images

Runner cache

  • should not ever clear the cache containers for the currently configured runners (it will result in runner error)
  • should, however clear containers when the runner has been reconfigured, and the containers are orphan, so to say.

This can be achieved by reading the relevant config.toml and parse the runner names?

Or run a cleanup before doing the runner registration? (edge cases?)

Runner helper

Connects to #22, here probably could use a cleanup which only keeps the most recent image for every repo. Thus could have a schedule of:

  • normal cleaning of containers, skipping runner helpers
  • once a day (?) cleaning that keeps 1 most recent images (likely that's the way? Unless the runners can be downgraded, in which case it would clear the wrong helper....) Can we parse this info from the config.toml (helper_image, when it's set) and maybe gitlab-runner output (when it's automatic)?
  • Or could we do a general cleanup, that only targets the helper images, and skips everything else?

Check if cache volumes work

Not sure if caching works currently, maybe have to make /cache into a volume? Or does it even work for our setup?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.