Comments (11)
I gave ECS a try and found that it starts the container in about a second. When it also has to pull the image, it takes about 20 seconds to start the container. We can probably pre-pull the image. This assumes a running instance and no need to scale-up and create more instances.
Both cases take an extra 10 seconds or so for the runner to connect after the container starts. That's just how long the runner code takes and this might be optimized in the future by GitHub.
Some code for the future (doesn't handle auto-scaling by container number yet):
new ecs.Cluster(
stack,
'ecs cluster',
{
enableFargateCapacityProviders: false,
capacity: {
instanceType: ec2.InstanceType.of(ec2.InstanceClass.M5, ec2.InstanceSize.LARGE),
minCapacity: 1,
maxCapacity: 2,
desiredCapacity: 1,
vpcSubnets: { subnetType: SubnetType.PUBLIC },
spotInstanceDraining: false, // waste of money to restart jobs as the restarted job won't have a token
},
vpc: vpc,
},
// TODO ECS_IMAGE_PULL_BEHAVIOR=prefer-cached
);
const task = new ecs.Ec2TaskDefinition(
stack,
'task',
);
task.addContainer('runner', {
image: ecs.ContainerImage.fromEcrRepository(fargateX64Builder.bind().imageRepository),
user: 'runner',
memoryLimitMiB: 2048,
command: ['./config.sh', '--unattended'],
logging: ecs.LogDriver.awsLogs({
streamPrefix: 'runner',
}),
});
from cdk-github-runners.
Maybe ECS autoscaling + a simple daemon to precache builder images?
You'd still get extra long cold starts for scale up + image pull, though you could potentially mitigate that w/ Warm Pools which let you balance the performance/price tradeoff
from cdk-github-runners.
We currently focus on ephemeral runners where one job doesn't affect the status of the next one. Leaving a runner behind means every package you installed in job 1, will be available in job 2. That can cause unexpected and unpredictable results.
In the future we may add API for a pool of prewarmed runners or even runners that stay around. But right now the focus is on-demand runners.
from cdk-github-runners.
Agreed on stateless.
Would it be possible to run an EC2 with a Docker daemon that starts ephemeral containers for jobs?
from cdk-github-runners.
How would that be different than CodeBuild/Fargate/Lambda providers?
from cdk-github-runners.
Ability to run Docker in Docker (dind)
from cdk-github-runners.
Faster start times. Lambda is fast. But CodeBuild is very slow to provision. Same for fargate.
from cdk-github-runners.
I think solving this with pre-warmed instances will be simpler. This way we don't have to create another system to manage capacity and running containers in EC2.
from cdk-github-runners.
Wouldn't pre-warmed instances eat up all of the savings then?
E.g. in our case, for every PR, we run 12 GitHub Workflows. This means 12 instances will need to be always on stand-by.
Otherwise, even if we needed to wait for one, it will delay the entire PR readiness (all tests need to pass).
from cdk-github-runners.
Yes, using pre-warmed instances, or an always-on instance, would eat up the savings of on-demand. AWS prices scale linearly with instance size. So they both solutions would eat up savings in a similar manner, unless you wish to overprovision on the shared instance. But either way you don't get the cost savings of on-demand.
This is beginning to sound like ECS. Do you know if ECS with an already active instance provisions faster than Fargate? Adding ECS support should be relatively easy and I think it will give us a shared instance without much work or orchestration code to maintain in the future.
from cdk-github-runners.
I was surprised to see the on-demand behavior of the EC2 Runner Provider. My understanding was that GitHub Actions runs in Docker containers, so I was expecting that Actions run on EC2 Runners would similarly use Docker and inherently be ephemeral. I would second the notion of using ECS to manage runners on EC2.
from cdk-github-runners.
Related Issues (20)
- feat: Allow lambda image builder to use AL2023 HOT 3
- Ec2 has not been terminated HOT 5
- Waiting for a runner to pick up this job... HOT 2
- feature request: attach custom tags to the ec2 HOT 28
- CodeBuild on Lambda support
- Missing amazon-cloudwatch-agent for Amazon Linux 2 and 2023
- Missing AL2023 support in a few places
- Runner not terminated after cancellation of job HOT 3
- Improve logging
- Error: Python cfn-response module, you may need to update your Lambda function code so that CloudFormation can attach the updated version HOT 6
- Long org/repository names can cause conflicts on the state machine naming HOT 2
- Problem Deprecated Construct CodeBuildImageBuilder HOT 6
- Add Environment Variable HOT 8
- Enable SOCI for images built with EC2 Image Builder
- Windows runner: no logs / no access HOT 5
- feat: add private registry mirror HOT 3
- windows ec2 doesn`t reach the stop state in ec2 userdata script HOT 4
- missing tools on windows runner for github cache action HOT 4
- Cancel prev jobs when new commit has arrived in PR HOT 2
- Using an existing Github App HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cdk-github-runners.