From #72 (comment)

In your case it sounds like it's because we expect every job with the self-hosted label to also have a label we recognize. It sounds like you have multiple installations of cdk-github-runners in different accounts, each implementing different labels. That means some of the installations will not recognize the label and cancel the job. We can probably add a flag to disable this behavior. Let's do this in a separate ticket.

@laxgoalie392

We can create GitHub App from manifest. It doesn't include client id and secret, but are those truly required? Can we use API to generate them? It also doesn't install the app, but there is probably a way to direct the user to the right URL automatically. We can then collect the installation webhook event and save installation id from there.

There are security considerations with this. We definitely don't want to let anyone who can guess the URL attach itself to our service. But we want to keep it simple so authentication against Cognito or IdP is a bit too much. Maybe we can settle for a one time setup token that gets deleted once the installation is complete. It can be given to the user in the stack output or with another function like the status function.

For example, we should automatically retry when getting capacity errors.

Hey @kichik,

thanks for this amazing piece of software.
I`m missing the option to set a property "subnetSelection" for the subnets of the fargate runner explicitely.
Is there some intention behind ?

   const fargateRunner = new FargateRunner(this, "fargate runner", {
      label: "fargate",
      vpc: existingVpc,
      ...
      ...
      subnetSelection: vpcSubnets,

    });

background:

I used this

    const fargateRunner = new FargateRunner(this, "fargate runner", {
      label: "fargate",
      vpc: existingVpc,
      assignPublicIp: false,
      spot: true,
    });

and received this error message

(node:27784) UnhandledPromiseRejectionWarning: Error: There are no 'Private' subnet groups in this VPC. Available types: Isolated,Deprecated_Isolated

br,

flo

Hi!

When using a lambda runner I cannot install dependencies at runtime. Looks like the $HOME would need to be set to /tmp so that lambda have write privileges.

Error:

npm WARN read-shrinkwrap This version of npm is compatible with lockfileVersion@1, but package-lock.json was generated for lockfileVersion@2. I'll try to do my best with it!
npm ERR! correctMkdir failed to make directory /home/sbx_user1051/.npm/_locks
npm WARN @[email protected] No repository field.

npm ERR! code EROFS
npm ERR! syscall mkdir
npm ERR! path /home/sbx_user[10](https://github.com/REDACTED#step:3:11)51
npm ERR! errno -30
npm ERR! rofs EROFS: read-only file system, mkdir '/home/sbx_user1051'
npm ERR! rofs Often virtualized file systems, or other file systems
npm ERR! rofs that don't support symlinks, give this error.
Error: Process completed with exit code 226.

To reproduce

Action:

name: Cypress tests - Integration
on: [push]

jobs:
  cypress-tests:
    runs-on: ["self-hosted", "lambda_m1024_s1024"]
    environment:
      name: development
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Prepare environment
        run: |
          npm  install

Runner:

        self.lambda_runner = LambdaRunner(
            scope=self,
            id="LambdaRunner",
            log_retention=logs.RetentionDays.ONE_MONTH,
            labels=["lambda_m1024_s1024"],
            vpc=self.vpc,
            security_group=self.security_group,
            memory_size=1024,
            ephemeral_storage_size=cdk.Size.gibibytes(1),
            timeout=cdk.Duration.minutes(10),
            subnet_selection=ec2.SubnetSelection(subnet_group_name="PrivateSubnet"),
        )

Hey,

I use codebuild for my runners and the default linux image (no custom image) provided.
Ecr has the capability to do a vulnerability scan (https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-scanning.html).
I found a lot of critical security vulnerabilities inside of the container image.
Do you have a clue why ?

br,

flo

Starting many instances at once is error prone, and will fail. This can happen when many jobs are submitted simultaneously. For us, we have several workflows for each PR, and we hit this limit pretty easily.

{
  "resourceType": "aws-sdk:ec2",
  "resource": "runInstances.waitForTaskToken",
  "error": "Ec2.Ec2Exception",
  "cause": "Request limit exceeded. (Service: Ec2, Status Code: 503, Request ID: dde217ee-eb44-4443-98a6-5bd134a8bf0f)"
}

I think a retry needs to be added, that handles this specific error case.

See https://docs.aws.amazon.com/codebuild/latest/userguide/sample-docker-custom-image.html

Trying to use this library with my GHE resulted in the following error message:

The GitHub server does not support configuring a self-hosted runner with 'DisableUpdate' flag.

I'm using the codebuild provider and narrowed it down to the --disableupdate flag.

I'm new to this whole topic, but removing it from by buildspec manually resolved the issue. I'm guessing that the flag might even be irrelevant anyways since the runners are ephemeral?

What are the impacts of removing that flag? could we make it optional?

• Spot pricing
• Docker support for Windows
• Only way to get MacOS support?
• Many more instance size options

Similar to #161, the exports for AmiBuilder and co are missing.

I would suggest to refactor and simplify how this projects handles exports, and hopefully avoid similar issues in the future.

Scenario is: in the CodeBuild Provider, when I try to run

docker compose up -d

in my GitHub Actions script, I get an error output like this:

unknown shorthand flag: 'd' in -d

I'm using the LINUX_X64_DOCKERFILE_PATH for my docker image, I guess the "issue" is with docker-in-docker.

It's possible to run it with docker-compose up -d but since docker-compose has been deprecated, I think it's a good move to update this. What do you think?

User should be able to install additional dependencies, setup internal repositories, and in general fully customize the image to their needs.

Challenges:

• Must pay for at least 24 hours of usage of a dedicated host for each test (sponsorship?)
• Instance can take 15-40 minutes to start (keeping some runners warm is a must?)
• EC2 Image Builder doesn't support Mac OS X (packer?)

I'm using a codebuild runner and trying to run an action that relies on aws credentials.

i've tried using the following step in my workflow:

- name: Test
  run: aws sts get-caller-identity

but i end up getting the following error:

Unable to locate credentials. You can configure credentials by running "aws configure".

I also tried using the aws configure credentials action but am hitting basically the same error

- name: AWS Secure Access
  uses: aws-actions/configure-aws-credentials@v1
  with:
    aws-region: us-east-1

My first impression was that I would be able to inherit the role that the codebuild runner was using

We shouldn't expose something unless it's needed so it's easier to make changes to the internals in the future.

For example:

constructor(scope: Construct, id: string, props: LambdaRunnerProps) {

Should be:

constructor(scope: Construct, id: string, props?: LambdaRunnerProps) {

Would be nice to have the option to set multiple labels (String Array) instead of just one label for a runner.

When installing extra packages with ImageBuilder, we might come across a tzdata dependency. When that happens, codebuild will get stuck here:

Configuring tzdata
--
990 | ------------------
991 |  
992 | Please select the geographic area in which you live. Subsequent configuration
993 | questions will narrow this down by presenting a list of cities, representing
994 | the time zones in which they are located.
995 |  
996 | 1. Africa        6. Asia            11. System V timezones
997 | 2. America       7. Atlantic Ocean  12. US
998 | 3. Antarctica    8. Europe          13. None of the above
999 | 4. Australia     9. Indian Ocean
1000 | 5. Arctic Ocean  10. Pacific Ocean

I suggest setting the TZ to UTC in all docker images, like this:

ENV TZ=UTC
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

Windows, ARM, GPU, and maybe even Mac once we get EC2 instance support.

Does it make sense to launch multiple EC2 instances for each workflow?

This really slows the process down. I think ideally it should launch one instance, and then re-use it for all runs, and then when idle - shut down.

With the last version Ec2Runners have been introduced. I just wanted to give it a try and came over a missing export of Ec2Runner and Ec2RunnerProps within the index.ts

import { Ec2Runner} from "@cloudsnorkel/cdk-github-runners";

lib/github-runners-stack.ts:15:3 - error TS2305: Module '"../node_modules/@cloudsnorkel/cdk-github-runners/lib"' has no exported member 'Ec2Runner'.

Seams the export is missing.

I added the ec2 runner for test ...

cdk code:

    const cloudPlatformsEc2Runner = new Ec2Runner(
      this,
      `${props.serviceName}-${props.stage}-ec2-runner`,
      {
        labels: ["cloudplatforms", "ec2"],
        vpc: existingVpc,
        subnetSelection: vpcSubnets,
        spot: true,
        instanceType: ec2.InstanceType.of(ec2.InstanceClass.T3, ec2.InstanceSize.MICRO),
      }
    );

workflow.yml:

name: fargate example
on: workflow_dispatch
jobs:
  self-hosted:
    runs-on: [self-hosted,cloudplatforms,ec2]
    steps:
      - run: echo hello world

stepfunctions:

these labels have been transmitted:

  "labels": {
    "self-hosted": true,
    "cloudplatforms": true,
    "ec2": true
  },

result:

it`s taking the wrong label :O

I assume because of this weird parent label

Discussed in #137

Expectations

Allow the following configurable and selectable setups:

1. Default: Create Lambda URL for the function (simplest zero headache approach)
2. Disable function access
3. Create API Gateway for the function that limits access to just GitHub webhook IPs (statically defined in code with action on this project that updates it once in a while, assuming they don't change those IPs often)
4. Create API Gateway for the function that limits access to a given list of IPs/CIDRs
5. Nice-to-have: private API Gateway that can only be accessed from certain VPC where GitHub Enterprise is installed

Label distribution and failures. More?

I just tried out to use the EC2 runners. Within our accounts we do not have a default VPC which leads to the stack can not be deployed.

const ec2Provider = new Ec2Runner(this, "ec2Runner", {
      subnet: subnets[0],
      spot: true,
    })

10:28:42 AM | CREATE_FAILED        | AWS::ImageBuilder::Image                       | Dev/GitHubRunners/...mage Builder/Image
Resource handler returned message: "Error occurred during operation 'No default VPC for this user. GroupName is only supported for EC2-Classic and default VPC.'." (RequestToken: ...., Handle
rErrorCode: GeneralServiceException)

Currently the Ec2RunnerProps also do not provide the option to set a VPC.

https://github.com/CloudSnorkel/cdk-github-runners/blob/main/src/providers/ec2.ts#L126-L186

Related to #113.

I haven't looked into this one yet but i just want to jot it down.

I applied the fix in #114. Initial use of the inherited worked to get secrets but i and ran into the following in a later step:

Error saving credentials: open /home/runner/.docker/config.json2271103188: permission denied

i'm trying to use this action:

- name: Login to Amazon ECR
  uses: aws-actions/amazon-ecr-login@v1
  with:
    registries: "****"

feels like a file system issue

See https://aws.amazon.com/blogs/containers/building-container-images-on-amazon-ecs-on-aws-fargate/

There is no clean way to add dynamic data into a built image. For example, I would like to add some secret ARNs for secrets that were created by CDK.

The status function should return some information on the image builder. Latest build, image digest, recent failures, etc.

Problem

Because the images use ubuntu:18.04, the git version is 2.17.1, checkout@v3 throws this warning message:

The repository will be downloaded using the GitHub REST API
To create a local Git repository instead, add Git 2.18 or higher to the PATH

This causes some problems with committing inside the self-hosted runner, meaning you will get an error like this when you try to commit:

fatal: not a git repository (or any of the parent directories): .git

I don't know if there's a specific reason we use 18.04 but is it possible to bump the ubuntu version to 20.04 since it is LTS as well?

When I try to use the ec2 runner I do get following error message in the step-function.

"error": {
    "Error": "Ec2.Ec2Exception",
    "Cause": "The provided credentials do not have permission to create the service-linked role for EC2 Spot Instances. (Service: Ec2, Status Code: 403, Request ID: 0ed21094-6fec-467b-bc8f-777207682f73)"
  },

We have an existing build role that i would like to reuse. Would be be able to allow that to be passed in and used?

When logging into a private ECR, the runner has this error:

Error: Could not login to registry ***: WARNING! Using -*** the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /home/runner/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Error saving credentials: open /home/runner/.docker/config.json3597419489: permission denied

any idea why? this is the action I'm using

- name: login to ECR
  env:
    REGION: ${{ secrets.AWS_REGION }}
    ACCOUNT_ID: ${{ secrets.AWS_ACCOUNT_ID }}
  run: aws ecr get-login-password --region $REGION | docker login --username AWS --password-stdin $ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com

when I run the same step on github servers, I get the same warning, but without the error:

Run aws ecr get-login*** $REGION | docker login --username AWS --password-stdin $ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com
WARNING! Your password will be stored unencrypted in /home/runner/.docker/config.json.
Configure a credential helper to remove this warning. See
Login Succeeded
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

I'm suspicious of that file name, config.json3597419489, 🤔 maybe I'm missing something in my command?

Problematic scenarios

The following scenarios are not properly handled yet.

1. User cancels workflow. We currently don't recognize the cancellation and if the user cancels the workflow before the runner boots up, the runner will stay there until it times out or another job comes along. If the runner is finally assigned another job, that means it took it away from another runner that was started just for the new job. So this can cause an endless resource waste cycle where a runner is always on, even when a job is not running. For Fargate that has no time limit, this can mean the runner will basically run forever.
2. Runner failure like configuration issues, missing capacity, or any random AWS failure. If the runner fails to even get the job, the job will just sit there waiting for the next runner to boot. But as we only create one runner per job, that means the old job is stealing the runner from the new job. The old job had to wait for the new job, and the new job will have to wait for the next job to come up. That could delay jobs for no reason. Our current solution for that is cancelling the workflow so the failure is clear and no job stealing occurs. However this can lead back to scenario number 1 if it happens fast enough.

Another complication to all of this is having to remove runners. There is a limited number of self-hosted runners allowed per-repo, and if we don't clean them up, we can get stuck unable to add runners. The runner usually removes itself, but any error conditions like provider timeout (lambda runs out of time), can prevent removal. That's why we always delete the runner using API on error. But to be able to delete the runner, we have to first stop any jobs running on it. That's another reason why we sadly have to cancel the entire workflow.

Wishlist

If we can get GitHub to make some changes, the following would really help simplify the solution for these corner cases.

1. GitHub runner should support a timeout configuration that allows us to tell it to only wait a certain amount of time before giving up on getting a job.
2. GitHub runner should allow us to configure a runner for a specific job and only for that job.
3. GitHub API should allow us to mark just one job as failed instead of the whole workflow.

Other solutions

Assuming we can't get our wishlist items, here some incomplete ideas to help resolve these issues.

• Have the step function monitor the job using /repos/{owner}/{repo}/actions/runs/{run_id}/jobs or the other one that has attempt information too. If we detect the job was cancelled, we can stop the runner. This means we won't be able to use .sync variants and will have to monitor the runners in the step function. This will also not work for Lambda runners as you can't stop a Lambda execution. But Lambdas are limited to 15 minutes which is not too bad.
• Monitor the job with the method above and stop the runner if the job hasn't started in 5 minutes. That would mean the runner was stolen or the labels don't match or there is another issue. Will this work with multiple jobs running at the same time "stealing" runners from each-other? We already create the runners in the repo scope to limit cross-job stealing.
• Figure out the undocumented actions API so we can fail a single job. We have the credentials and connection information in the .runner and .credentials files. Another option is creating a special runner fork that fails a single job.
• Some kind of external to the step function monitor that makes sure all jobs are behaving right and has the power to start more runners if needed so stolen runners can be "fixed".

I deployed a simple codebuilder setup via Python and CDK, but visiting the github.setup.url just shows a blank page. The Javascript console has these errors (I've replaced sensitive IDs with <my-foo>):

Content Security Policy: The page's settings blocked the loading of a resource at inline ("default-src"). [<my-function>.lambda-url.eu-west-2.on.aws:7:1](https://<my-function>.lambda-url.eu-west-2.on.aws/?token=<my-token>)
Content Security Policy: The page's settings blocked the loading of a resource at inline ("default-src"). [<my-function>.lambda-url.eu-west-2.on.aws:28:1](https://<my-function>.lambda-url.eu-west-2.on.aws/?token=<my-token>)
Content Security Policy: The page's settings blocked the loading of a resource at https://<my-function>.lambda-url.eu-west-2.on.aws/favicon.ico ("img-src"). resource:186:19
Content Security Policy: The page's settings blocked the loading of a resource at inline ("default-src"). moz-extension:1:52727

This was in Firefox. I tried installing Google Chrome, and the page seems to work in that.

There was no label match, yet the state machine still waits to reap idle providers. No real harm here, but I think it'd be more optimized if it didn't wait in the cases where there was no match.

In fact, ideally, I think this selection should as early as possible.

I have a question about the webhook lambda that github calls(?): is there any way to avoid unnecessary invocations with this? for example, if I have the function URL, even though I'll get unauthorized message every time, I'm still invoking the function, so is there a way to limit this invocation?

I've tried creating a security group in a VPC which only allows for github action's IPs to invoke the function, but I get an error due to there being too many IPs 😄.

hey @kichik,

I would like to have the logs of the step function in cloudwatch to be able to analyze them.

It`s possible to enable the logs (described )

...directly in the state machine

br,

flo

What's the "golden path" to upgrade the version of the runner contained within the Docker image (specifically for Lambda, but across the board I suppose)? GitHub is whining about it being outdated on Action invocation...

Hey @kichik,

can`t get that image - maybe because I`m in another region .. eu-central-1
I wanted to build a windows container image for codebuild ...

code

    const windowsDefaultImage = new ContainerImageBuilder(
      this,
      `${props.serviceName}-${props.stage}-windows-image`,
      {
        architecture: Architecture.X86_64,
        os: Os.WINDOWS,
        runnerVersion: RunnerVersion.latest(),
        rebuildInterval: Duration.days(14),
        instanceType: ec2.InstanceType.of(
          ec2.InstanceClass.T3A,
          ec2.InstanceSize.LARGE
        ),
      }
    );

error:

The following required resource 'Image' cannot be found: 'arn:aws:imagebuilder:us-east-1:aws:image/windows-server-20
19-x86-core-ltsc2019-amd64/2020.12.8'