Release 5.10.2 not starting any servers about terraform-aws-github-runner HOT 10 CLOSED

Suddenly started having issues, though was running 5.9. Not sure if related, but noticed this in our scale-up lambda logs, shortly after the "Received event" log.

"Ignoring error: response.json(...).catch is not a function"

EDIT: actually, I've only been getting the above error since upgrading to 5.10.x. Reverting to see what happens. My issue on 5.9 was likely something else.

EDIT2: response.json issue gone after reverting to 5.9.0. Curious if you find similar logs to above once you're all wiped/restored.

Yes I have the same error in that log

    "level": "WARN",
    "message": "Ignoring error: response.json(...).catch is not a function",
    "service": "service_undefined",

from terraform-aws-github-runner.

I can confirm I am seeing the same issues. I upgraded both my runner code and lambdas from 5.9.0 to 5.10.2 and started seeing the problem with all of my scale-up lambdas (see below). If I downgrade just the lambdas from 5.10.2 to 5.9.0 they work. I have tried this twice (upgrade and downgrade) - the results are the same.

Scale Up Lambdas Errors

{
    "level": "WARN",
    "message": "Ignoring error: response.json(...).catch is not a function",
    "service": "service_undefined",
    "timestamp": "2024-05-01T14:03:59.036Z",
    "xray_trace_id": "1-66324bcd-b91f32230d4f1c5052a29617",
    "region": "us-east-1",
    "environment": "<REDACTED",
    "aws-request-id": "d9b95d22-1a54-59ca-9a20-038e545cb1f1",
    "function-name": "<REDACTED>-scale-up",
    "module": "lambda.ts"
}

This seems like a wide spread issue as I'm not the only one in the community experiencing this problem.

from terraform-aws-github-runner.

Suddenly started having issues, though was running 5.9. Not sure if related, but noticed this in our scale-up lambda logs, shortly after the "Received event" log.

"Ignoring error: response.json(...).catch is not a function"

EDIT: actually, I've only been getting the above error since upgrading to 5.10.x. Reverting to see what happens. My issue on 5.9 was likely something else.

EDIT2: response.json issue gone after reverting to 5.9.0. Curious if you find similar logs to above once you're all wiped/restored.

from terraform-aws-github-runner.

Thanks for reporting, we going to dig in further

from terraform-aws-github-runner.

Same issue here.

from terraform-aws-github-runner.

I have finally got things back up and working using v5.10.0 Last one I know worked fine.

I made a total mess of the system. Also the AMI user I was using within the instance images did not have all the permissions it needed and so instances were starting then shutting down again.

Once it had the permission to read from the s3 bucket download the github runner then things started to work again. When I cleared things out is where it really got messed up.

from terraform-aws-github-runner.

Ran into this after upgrading v5.10.0 -> v5.10.2 as well. Instances would not spin up anymore. Rolled back to v5.10.0 and things seem to be working again.

from terraform-aws-github-runner.

I have now got myself into such a state trying to wipe everything from aws and recreate fresh.

I keep getting errors like The specified log group already exists

EntityAlreadyExists: Instance Profile *****-gh-ci-runner-profile already exists

EntityAlreadyExists: Instance Profile *****-gh-ci-runner-profile already exists

from terraform-aws-github-runner.

My uneducated guess is maybe this change
v5.10.1...v5.10.2#diff-4eccdb723f617ab228bd897ae3b78dba1fb4a779f818efa21e55ccb75b06113dL19

I have rolled back to 5.10.1 and servers are starting again. However no jobs are picked up.

from terraform-aws-github-runner.

We were seeing the same thing, downgrading to 5.10.0 fixed the problem.

We have also been seeing this in lambdas across several versions, although things appears to be working:
"service": "service_undefined",

from terraform-aws-github-runner.