Layer Errors prevent the "main" Lambda to get executed about newrelic-lambda-extension HOT 8 CLOSED

Hi

Just to add to the above, we suffered an outage in production last night for a number of lambdas due to this as well I believe, but it was due to the cloud-collectors in EU failing, as opposed to config issues.

Is there a workaround we can put in for now to stop this happening until a proper fix is available? Reducing the timeout or whatever for the NR_EXT?

Thanks

from newrelic-lambda-extension.

@ftieben I was able to reproduce and subsequently simulate this issue. Have a pull request in the works (#75), should be merged and released early next week.

from newrelic-lambda-extension.

@ftieben Thanks for reporting this issue. Transferred it to the relevant repository.

from newrelic-lambda-extension.

@ftieben Thanks for the update. We have released a refactor of our timeout logic which should (hopefully) do the right thing in most cases. We hope you'll give it a try and report any issues you run into. In this specific use case, the extension should no longer suffer from a runaway HTTP request and will instead exit within the deadlines provided by Lambda.

from newrelic-lambda-extension.

@kolanos Thanks a lot for the work you are doing, I've tried an example Lambda with the new Layer Version

arn:aws:lambda:eu-central-1:451483290750:layer:NewRelicNodeJS12X:44

The Lambda is working fine while it can reach NR but it is still failing as soon as I put it into an VPC without allowed egress:

Response:

{
  "errorMessage": "2021-06-04T06:59:10.097Z 72f018d0-250d-4326-86e6-6e3b4b1e3c49 Task timed out after 5.00 seconds"
}

Logs:

START RequestId: 72f018d0-250d-4326-86e6-6e3b4b1e3c49 Version: $LATEST
[NR_EXT] New Relic Lambda Extension starting up
EXTENSION	Name: newrelic-lambda-extension	State: Registered	Events: [INVOKE,SHUTDOWN]
END RequestId: 72f018d0-250d-4326-86e6-6e3b4b1e3c49
REPORT RequestId: 72f018d0-250d-4326-86e6-6e3b4b1e3c49	Duration: 5004.30 ms	Billed Duration: 3000 ms	Memory Size: 128 MB	Max Memory Used: 32 MB	
2021-06-04T06:59:10.097Z 72f018d0-250d-4326-86e6-6e3b4b1e3c49 Task timed out after 5.00 seconds

from newrelic-lambda-extension.

@ftieben Thanks for the update. What is the timeout configured for the function? And are you reporting to the New Relic EU region? Going to try to reproduce this as closely as I can. Any other details that might be relevant would be useful as well.

from newrelic-lambda-extension.

@ftieben We have released the deadline tweaks in #75. Hope you'll give it a try and report any issues. Thanks again.

from newrelic-lambda-extension.

@ftieben Closing this for now. Please feel free to re-open if you continue to have issues. Am reasonably confident this case is now handled gracefully by the extension. Thanks again.

from newrelic-lambda-extension.