Asynchronous Initialization Extension Layer for NodeJs

The Problem

Asynchronous operations do not complete during the initialization phase. This is specifically a problem for provisioned concurrency. One of the major benefits is that long running operations can be initialized during provisioned concurrency, but this does not happen with AWS Lambda NodeJs operations.

A JavaScript IIFE can be used to start a promise during the initialization phase, but it does not complete until the function is actually called. Sometimes it does not complete until multiple requests later since it is running as a background task in the event loop.

The Solution

An AWS Extension layer can be used to keep the AWS Lambda container active until the AWS Lambda NodeJs function initialization completes all of its asynchronous operations. I have linked an example repository where I demonstrate this.

I would like to turn this over to the aws-lambda-extensions repository. Is it Ok to create a pull request for this? Are there any questions or modifications needed?

The example is here: https://github.com/chuckbarkertech/nodejs-example-async-init-extension

Since both the layers are running on the same port (3000) I am getting port already in use error

I tried using this extension to capture logs in Python. The problem is, when Lambda lasts less than a second, I only get messages from extension after about 5 minutes. This renders this essentially unusable, as this is nowhere near real-time logging. Is this normal? How can I fix this?

How can I access event/context object when using extensions?
https://docs.aws.amazon.com/whitepapers/latest/serverless-architectures-lambda/the-event-object.html

I am trying to publish a similar dockerized lambda container with an embedded extension similar to what you have created here.

Except instead of a python lambda extension I wanted to use golang and I am trying to attach this to a dockerzied nodejs lambda function.

But when I try and combine the two I am getting this error 👇

I have a sample repository here where you can see and deploy exactly what I have created to recreate this issue.

I am somewhat confused because the extension seems to work fine deployed on its own as a layer and if I modify the docker file for the nodejs lambda container to publish the extension code to a directory other than /opt/extensions then I am able to find the extension code when I search for it within my containerized nodejs app 🤔

We have a lambda function which connects to AppConfig but due to security reason lambda function should be inside a VPC. Since there is no VPC endpoint, we will have to route the traffic to on-prem proxy server via transit gateway from where the request will be sent to AppConfig endpoint. In order to use AppConfig, we are using AppConfig extensions layer provided by AWS. Now we would like to know how we can send traffic from AppConfig Extensions to AppConfig via the on-prem proxy server

Any plans to add examples in C#?

Hi Team,

Thank you for the samples. They were very helpful to get started.

When listening to logs from Logs API, not all logs have the equest id information within them. Only certain platform events seem to have the request id information.

Is there a way to capture request id for all events ? The lambda context usually has the request id info, is there a way to pass the lambda's context object to the extension ?

At the moment, I try to capture request id from the platform start event and add it to all messages, but the request id seems to mixup, when the collected stream of logs have logs from 2 different invocations

Regards,
Bhavya.

Enhance "cache-extension-demo" lambda extension to cache secrets via parameter store, here is an example configuration.

parameters:
  - region: us-east-1
    names:
      - /aws/reference/secretsmanager/abcsecret

Setup:

• Lambda that makes an http request to external extension.
• External Extension running an http.Server (localhost:8111)
• Incoming requests in the Extension are queued into a channel and handled async while returning status-code 201 to lambda request
• During async handling of the requests a http call is made.

Expected Behaviour:
The HTTP call works during async handling

Actual Behaviour:
Only sync calls (for example during the http.HandleFunc or just in the main-func) work. Async http call times out after shutdown event

I tried multiple ways to implement this. For example making "processEvents" async and "StartChannel" async. Or a simple go-routine call in the handlefunc. Nothing seemed to work. And it seems very strange to me.

func main(){
         fmt.Println("So far its working")
        //this works perfectly fine
	_, err := http.Get("https://jsonplaceholder.typicode.com/posts/1")

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	/**** async http.listenAndServe with handlefunc etc ****/

	sigs := make(chan os.Signal, 1)
	signal.Notify(sigs, syscall.SIGTERM, syscall.SIGINT)

	go func() {
		s := <-sigs
		fmt.Println(s)
		fmt.Println("closing now")
		eventhubHandler.CloseChannelAndCauseShutdown()
	}()
	_, err = extensionClient.Register(ctx, extensionName)
	if err != nil {
		panic(err)
	}

	go  handler.ReadChannel()
	processEvents(ctx)
}


//handler.go
func ReadChannel() {
	fmt.Println("starting channel")
	dataChannel := channelConfig.dataChannel
	for _:= range dataChannel {
	        // does not work and only times out after shutdown event
		_, err := http.Get("https://jsonplaceholder.typicode.com/posts/1")
	}

}

go.mod partial

        github.com/aws/aws-lambda-go v1.27.1

	github.com/aws/aws-sdk-go v1.42.25

	github.com/aws/aws-xray-sdk-go v1.6.0

template.yaml partial

LAMBDA:
    Type: AWS::Serverless::Function 
    Properties:
      CodeUri: bin/lambda
      Handler: bootstrap
      Runtime: provided.al2
      MemorySize: 256
      Timeout: 5
      Tracing: Active 
      Layers:
        - !Ref EXTENSION

I just want to share and couple of issues I faced today while deploying the template under 'https://github.com/aws-samples/aws-lambda-extensions/tree/main/s3-logs-extension-demo-zip-archive'

Error 1:

START RequestId: d3594b49-d2bd-49b2-9cca-4cc7beafe05b Version: $LATEST EXTENSION Name: logs_api_http_extension.py State: LaunchError Events: [] Error Type: PermissionDenied END RequestId: d3594b49-d2bd-49b2-9cca-4cc7beafe05b REPORT RequestId: d3594b49-d2bd-49b2-9cca-4cc7beafe05b Duration: 1.49 ms Billed Duration: 2 ms Memory Size: 128 MB Max Memory Used: 6 MB XRAY TraceId: 1-60425254-30ebc4cf7841f6fc52addcb3 SegmentId: 121c26332f517724 Sampled: true RequestId: d3594b49-d2bd-49b2-9cca-4cc7beafe05b Error: fork/exec /opt/extensions/logs_api_http_extension.py: permission denied Extension.LaunchError

Reason Error 1:
The reason behind the issue is that the underlying Lambda container requires permissions to execute the 'extensions/logs_api_http_extension.py' file.

Solution Error 1:
Run the below comand to add execution permissions to the python script:
$ chmod +x logs_api_http_extension.py

Error 2:

extension.extensions_api_client: Registering Extension at ExtensionsAPI address: http://127.0.0.1:9001/2020-01-01/extension extension.logs_api_client: Subscribing to Logs API on http://127.0.0.1:9001/2020-08-15/logs extension.http_listener: Running HTTP Server on 0.0.0.0:4243 LOGS Name: logs_api_http_extension.py State: Subscribed Types: [platform,function] Traceback (most recent call last): File "/opt/extensions/logs_api_http_extension.py", line 134, in <module> main() File "/opt/extensions/logs_api_http_extension.py", line 131, in main ext.run_forever() File "/opt/extensions/logs_api_http_extension.py", line 67, in run_forever s3 = boto3.resource('s3') File "/opt/extensions/lib/boto3/__init__.py", line 102, in resource return _get_default_session().resource(*args, **kwargs) File "/opt/extensions/lib/boto3/session.py", line 384, in resource client = self.client( File "/opt/extensions/lib/boto3/session.py", line 258, in client return self._session.create_client( File "/opt/extensions/lib/botocore/session.py", line 834, in create_client client = client_creator.create_client( File "/opt/extensions/lib/botocore/client.py", line 81, in create_client cls = self._create_client_class(service_name, service_model) File "/opt/extensions/lib/botocore/client.py", line 111, in _create_client_class self._event_emitter.emit( File "/opt/extensions/lib/botocore/hooks.py", line 356, in emit return self._emitter.emit(aliased_event_name, **kwargs) File "/opt/extensions/lib/botocore/hooks.py", line 228, in emit return self._emit(event_name, kwargs) File "/opt/extensions/lib/botocore/hooks.py", line 211, in _emit response = handler(**kwargs) File "/opt/extensions/lib/boto3/utils.py", line 61, in _handler module = import_module(module) File "/opt/extensions/lib/boto3/utils.py", line 52, in import_module __import__(name) File "/opt/extensions/lib/boto3/s3/inject.py", line 15, in <module> from boto3.s3.transfer import create_transfer_manager File "/opt/extensions/lib/boto3/s3/transfer.py", line 127, in <module> from s3transfer.exceptions import RetriesExceededError as \ File "/opt/extensions/lib/s3transfer/__init__.py", line 134, in <module> import concurrent.futures File "/opt/extensions/lib/concurrent/futures/__init__.py", line 8, in <module> from concurrent.futures._base import (FIRST_COMPLETED, File "/opt/extensions/lib/concurrent/futures/_base.py", line 414 raise exception_type, self._exception, self._traceback ^ SyntaxError: invalid syntax
Reason Error 1:
This is an issue generated in the build step, when python installs the boto3 dependencies.

See below link for more details:
boto/boto3#1399 (comment)

Solution Error 2:
The straight forward solution in my case was to build the SAM application using docker containers.
$ sam build --use-container $ sam deploy

Did anyone try to implement this for caching RDS database table data using python. If yes can you please share sample code

The following part.

I think the following is correct.

$ cd go-example-telemetry-api-extension
$ GOOS=linux GOARCH=amd64 go build -o bin/extensions/go-example-telemetry-api-extension main.go
$ chmod +x bin/extensions/go-example-telemetry-api-extension

I'm analyzing (and playing with) code in the custom-runtime-extension-demo and can't understand when the runtime gets SIGTERM.
So far I understand that in line 41 code starts waiting for curl to finish. That is start of new invocation. There we also trap SIGTERM. But I can't got that _term function to be executed ever.
It seams that the runtimes gets KILL instead of TERM signal. So this example may be a bit misleading.

@julianwood can you please help me understand that runtime behavior a little bit more.
Thanks.

It would be great to see an official Logs API example for Node.js.
I've ported one based on Python, in case anyone wants to give it a try. Comments and improvements welcome.

Fork: https://github.com/stawecki/aws-lambda-extensions/tree/nodejs-example-logs-api-extension

Is it possible for internal extensions to be able to intercept a lambda invocation event or response? I want to build essentially a wrapper function that is meant to validate certain properties of both incoming event payloads and responses.

I saw the example here for a javaagent to intercept the lambda invocations, and wasn't sure if this was possible with node in any way? https://docs.aws.amazon.com/lambda/latest/dg/runtimes-modify.html

Hey,
Followed the examples custom-runtime-extension-demo, nodejs-example-extension, and for both of them received:
{ "errorMessage": "RequestId: XXX Error: fork/exec /opt/extensions/{THE_EXTENSION_NAME}: no such file or directory", "errorType": "Extension.LaunchError" } while testing with the basic testing configuration in the UI.

From what I read in your docs everything inside the extensions dir suppose to be copied to /opt/extensions/, but it doesn't seem to happen here.
Have you had this problem before?
Also which part declares for the lambda function, the name of the extension to read from /opt/extensions/?

Thanks for the help:)

Lambda extension to send logs directly to Kinesis firehose in order to facilitate centralized logging. The logs should only be streamed to the firehose service rather than sending it to CloudWatch and creating a subscription around this.

Hey folks, is there a way to bundle node runtime with the nodejs-example-logs-api-extension repo.
I would like this layer to be totally self sufficient, so that it can be used by any lambdas that may have different runtimes.
I know that is possible according to the docs, but I'm unable to do so.
Any pointers would be greatly appreciated.

If I try and deploy a node layer with say Java runtime lambda, I get these errors:
nodejs-log-api-extension launching extension
/usr/bin/env: node: No such file or directoty

Just report issue with sam deployment, noticed
for GoExampleExtensionLayer in the template.yaml

ContentUri: go-example-extension/

should be:
ContentUri: go-example-extension/bin

otherwise the extension won't run.

James

Lambda extension to send logs directly to Kinesis Data Streams in order to facilitate centralized logging. The logs should only be streamed to the Kinesis Data Streams service rather than sending it to CloudWatch and creating a subscription around this.

SAM Template to support the Cache Extensions demo application. Adding additional AWS resources like Parameter Store would be an advantage.

I noticed that logs for the first Lambda's invoke are missing after a cold start but the extension picks up subsequent logs from additional invokes. I was only able to get the first invoke logs if I added an additional delay of 5 seconds here: https://github.com/aws-samples/aws-lambda-extensions/blob/main/python-example-logs-api-extension/extensions/logs_api_http_extension.py#L56 or if I enabled provisioned concurrency. Is there another work around to get the first invokes logs?

Steps to reproduce:
git clone https://github.com/aws-samples/aws-lambda-extensions
cd aws-lambda-extensions/s3-logs-extension-demo-zip-archive
sam deploy --stack-name s3-logs-extension-demo --guided
Accept defaults, region us-west2 used.
aws lambda invoke --function-name "logs-extension-demo-function" --payload '{"payload": "hello"}' /tmp/invoke-result --cli-binary-format raw-in-base64-out --log-type Tail --region us-west-2

See attached CW log error: ModuleNotFoundError: No module named 'boto3'

We have built a Java Lambda extension and subscribing to telemetry api to push logs to splunk. An explicit exception is thrown in lambda to check telemetry api show status as failure for Lambda exception in the platform.runTimeDone event.

AWS lambda Telemetry api msgs:-

{
"time": "2023-06-11T17:03:14.843Z",
"type": "function",
"record": "exception: .infrastructure.exception.ServerlessException\ncom.infrastructure.exception.ServerlessException: exception\ ....
}
,
{
"time": "2023-06-11T17:03:14.843Z",
"type": "platform.runtimeDone",
"record": {
"requestId": "777ff55f-e84c-41f8-ae9f-53f541f01a60",
"status": "success",
"metrics": {
"durationMs": 78.656,
"producedBytes": 0
}
}
}

Please advise, why platform.runtimeDone event status is success instead of failure for explicit exception in lambda or please advise in. which cases of invocation failure we will get status as failure in platform.runtimeDone event.

When trying to use the example in https://github.com/aws-samples/aws-lambda-extensions/tree/main/nodejs-example-logs-api-extension at commit a3a540b I am getting the following issue come up in my cloudwatch:

nodejs-example-logs-api-extension  launching extension
--
internal/modules/cjs/loader.js:1015
throw new ERR_REQUIRE_ESM(filename, parentPath, packageJsonPath);
^
 
Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: /opt/nodejs-example-logs-api-extension/node_modules/node-fetch/src/index.js
require() of ES modules is not supported.
require() of /opt/nodejs-example-logs-api-extension/node_modules/node-fetch/src/index.js from /opt/nodejs-example-logs-api-extension/extensions-api.js is an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which defines all .js files in that package scope as ES modules.
Instead rename index.js to end in .cjs, change the requiring code to use import(), or remove "type": "module" from /opt/nodejs-example-logs-api-extension/node_modules/node-fetch/package.json.
 
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1015:13)
at Module.load (internal/modules/cjs/loader.js:863:32)
at Function.Module._load (internal/modules/cjs/loader.js:708:14)
at Module.require (internal/modules/cjs/loader.js:887:19)
at require (internal/modules/cjs/helpers.js:74:18)
at Object.<anonymous> (/opt/nodejs-example-logs-api-extension/extensions-api.js:1:15)
at Module._compile (internal/modules/cjs/loader.js:999:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
at Module.load (internal/modules/cjs/loader.js:863:32)
at Function.Module._load (internal/modules/cjs/loader.js:708:14) {
code: 'ERR_REQUIRE_ESM'
}
EXTENSION	Name: nodejs-example-logs-api-extension	State: Started	Events: []
nodejs-example-logs-api-extension  launching extension
internal/modules/cjs/loader.js:1015
throw new ERR_REQUIRE_ESM(filename, parentPath, packageJsonPath);
^
 
Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: /opt/nodejs-example-logs-api-extension/node_modules/node-fetch/src/index.js
require() of ES modules is not supported.
require() of /opt/nodejs-example-logs-api-extension/node_modules/node-fetch/src/index.js from /opt/nodejs-example-logs-api-extension/extensions-api.js is an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which defines all .js files in that package scope as ES modules.
Instead rename index.js to end in .cjs, change the requiring code to use import(), or remove "type": "module" from /opt/nodejs-example-logs-api-extension/node_modules/node-fetch/package.json.
 
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1015:13)
at Module.load (internal/modules/cjs/loader.js:863:32)
at Function.Module._load (internal/modules/cjs/loader.js:708:14)
at Module.require (internal/modules/cjs/loader.js:887:19)
at require (internal/modules/cjs/helpers.js:74:18)
at Object.<anonymous> (/opt/nodejs-example-logs-api-extension/extensions-api.js:1:15)
at Module._compile (internal/modules/cjs/loader.js:999:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
at Module.load (internal/modules/cjs/loader.js:863:32)
at Function.Module._load (internal/modules/cjs/loader.js:708:14) {
code: 'ERR_REQUIRE_ESM'
}
EXTENSION	Name: nodejs-example-logs-api-extension	State: Started	Events: []
START RequestId: b2cad615-2bfa-4b6a-a2e7-6990c17a250d Version: $LATEST
END RequestId: b2cad615-2bfa-4b6a-a2e7-6990c17a250d
REPORT RequestId: b2cad615-2bfa-4b6a-a2e7-6990c17a250d	Duration: 667.06 ms	Billed Duration: 668 ms	Memory Size: 128 MB	Max Memory Used: 9 MB
RequestId: b2cad615-2bfa-4b6a-a2e7-6990c17a250d Error: exit status 1Extension.Crash

My System:
_Node Version:_12.22.12
_OS:_macOS Monterey 12.6
_AWS CLI Version:_2.7.31
_Python Version:_3.10.6

Investigation

It appears that as per node-fetch/node-fetch#1533 node-fetch doesn't support CommonJS and as such when dependabot did a bump in c65b424 broke the example

Is this variable customizable?

LAMBDA_AGENT_NAME_HEADER_KEY = "Lambda-Extension-Name"

https://github.com/aws-samples/aws-lambda-extensions/blob/main/s3-logs-extension-demo-zip-archive/extensionssrc/extensions/logs_api_http_extension/extensions_api_client.py#L11

or

https://github.com/aws-samples/aws-lambda-extensions/blob/main/python-example-logs-api-extension/extensions/logs_api_http_extension/extensions_api_client.py#L12

When i change this value for example to this,
LAMBDA_AGENT_NAME_HEADER_KEY = "Custom-Lambda-Extension-Name"
I am getting "Failed to register extensionsAPI" errors

START RequestId: 9f34a0b2-dd6c-4894-9c96-0f58351354f5 Version: $LATEST
extension.extensions_api_client: Registering Extension at ExtensionsAPI address: http://127.0.0.1:9001/2020-01-01/extension
Traceback (most recent call last):
  File "/opt/extensions/logs_api_http_extension/extensions_api_client.py", line 32, in register
    resp = urllib.request.urlopen(req)
  File "/var/lang/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/var/lang/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/var/lang/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/var/lang/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/var/lang/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/var/lang/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/extensions/logs_api_http_extension.py", line 134, in <module>
    main()
  File "/opt/extensions/logs_api_http_extension.py", line 130, in main
    ext = LogsAPIHTTPExtension(os.path.basename(__file__), _REGISTRATION_BODY, _SUBSCRIPTION_BODY)
  File "/opt/extensions/logs_api_http_extension.py", line 57, in __init__
    self.agent_id = self.extensions_api_client.register(self.agent_name, registration_body)
  File "/opt/extensions/logs_api_http_extension/extensions_api_client.py", line 41, in register
    raise Exception(f"Failed to register to ExtensionsAPI: on {self.runtime_api_base_url}/register \
Exception: Failed to register to ExtensionsAPI: on http://127.0.0.1:9001/2020-01-01/extension/register                 with agent_unique_name:logs_api_http_extension.py                  and registration_body:{'events': ['INVOKE', 'SHUTDOWN']}
Error: HTTP Error 403: Forbidden
EXTENSION	Name: logs_api_http_extension.py	State: Started	Events: []
END RequestId: 9f34a0b2-dd6c-4894-9c96-0f58351354f5
REPORT RequestId: 9f34a0b2-dd6c-4894-9c96-0f58351354f5	Duration: 7617.86 ms	Billed Duration: 7618 ms	Memory Size: 128 MB	Max Memory Used: 26 MB	
RequestId: 9f34a0b2-dd6c-4894-9c96-0f58351354f5 Error: exit status 1
Extension.Crash

Not sure if it is the limitation, but the logs are uploaded to S3 only during the INVOKE and SHUTDOWN event which makes me wait to receive logs of current invocation in the next one or during the shut down event.

Also tried uploading the batch to S3 as soon as the request is received at the local server, could able to upload them except the very last log in the lambda function.

Is there any possible solution to this ?

I got the following error when I use Go 1.20:

/var/task/main: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /var/task/main)
/var/task/main: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /var/task/main)
exit status 1

If I change it to Go 1.19 it works

Problem
When using the new Node14.x Runtime, the request to subscribe to logs (PUT - 2020-08-15/logs) returns a 500 Internal Server Error.
I have tried this on the following extensions with the same result:

• go-example-logs-api-extension
• woodchuck (my personal extension)
• Honey Comb Logs extension

Switching back to Node12.x fixes this issue for all extensions above.

Steps to reproduce

• Follow the readme to deploy the go-example-logs-api-extension layer.
• Use the layer on any Node14.x lambda.
• Invoke the Lambda

Cloudwatch Output (go-example-logs-api-extension)
time="2021-02-11T17:09:48Z" level=fatal msg="http://127.0.0.1:9001/2020-08-15/logs failed: 500[500 Internal Server Error] {\"errorMessage\":\"Internal Server Error\",\"errorType\":\"InternalServerError\"}\n" agent=go-example-logs-api-extension

Side Note
I do not think this is an issue with the examples in this repository, but might be able to find its way to the right people.

Good morning,
I'm playing with the "bash-example-wrapper" example.
In the README.md you suggest to use the "importtime" option that is supported by Python from the version 3.7.
It may be useful to specify, in the same file, which runtime version to choose in order to not waste time looking for a non-existent error in the code.
Thank you very much!

Ability to cache data from Parameter store and Dynamodb through AWS Lambda extensions. It would also be great if we can support cache expiry settings and list of secrets to be cached part of the Lambda function

Taking the python extension example as an example, I wanted to have an extension that uses ebpf bcc tools python frontend. But, just adding bcc to the requirements.txt and ship it with the extension code is not enough. It needs the BCC tools to be installed as well. In the case of lambda, assuming it's running on Amazon Linux 2, even though the BPF kernel config flags are set, one still would need to run sudo amazon-linux-extras install BCC to install it, and then the extension python code will be able to run bcc frontend such as BPF() function.

I tried to run the install command as part of extension initialization, but seems it doesn't recognize the command.

I was wondering if there's a way to do that or not?

I've followed the instructions to build and deploy the c# external lambda extension based on the latest code in this repo (https://github.com/aws-samples/aws-lambda-extensions/tree/main/csharp-example-extension). I added the layer to a simple 'Hello World' lambda function created via the AWS console. Both the extension and the lambda function use dotnetcore3.1 runtime.

Although the extension has successfully registered for shutdown events (as shown by the log output below, i.e. i.e. "EXTENSION Name: csharp-example-extension State: Ready Events: [INVOKE,SHUTDOWN]")), the shutdown event is never received (ie. not logged).

I created a Python lambda extension (source: https://github.com/aws-samples/aws-lambda-extensions/tree/main/python-example-extension ) and the SHUTDOWN event was received as expected.

Is there something wrong with the example code?

1. Invoke function twice - first invoke shows successful extension registration

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|   timestamp   |                                                                                                                                                                                            message                                                                                                                                                                                            |
|---------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1619414590165 | START RequestId: 767e8f57-8ebf-467f-8683-cad16e313c0b Version: $LATEST                                                                                                                                                                                                                                                                                                                        |
| 1619414590405 | Launching csharp-example-extension extension ...                                                                                                                                                                                                                                                                                                                                              |
| 1619414590546 | [csharp-example-extension] Registered extension with id = dab76136-6606-4ebb-9ced-187f5e8c2c46                                                                                                                                                                                                                                                                                                |
| 1619414590546 | EXTENSION Name: csharp-example-extension State: Ready Events: [INVOKE,SHUTDOWN]                                                                                                                                                                                                                                                                                                               |
| 1619414590583 | [csharp-example-extension] Handling invoke from extension: {"eventType":"INVOKE","deadlineMs":1619414620547,"requestId":"767e8f57-8ebf-467f-8683-cad16e313c0b","invokedFunctionArn":"arn:aws:lambda:ap-southeast-2:713834996596:function:lbd-as2-ws-tabjul-temp6","tracing":{"type":"X-Amzn-Trace-Id","value":"Root=1-60864e3d-64fb387b79370c27015d0cfc;Parent=02b3e12935c9b8cf;Sampled=0"}}  |
| 1619414590882 | END RequestId: 767e8f57-8ebf-467f-8683-cad16e313c0b                                                                                                                                                                                                                                                                                                                                           |
| 1619414590882 | REPORT RequestId: 767e8f57-8ebf-467f-8683-cad16e313c0b Duration: 335.24 ms Billed Duration: 336 ms Memory Size: 634 MB Max Memory Used: 83 MB Init Duration: 391.82 ms                                                                                                                                                                                                                        |
| 1619414621622 | START RequestId: ac43b4c9-f01e-47fb-80b8-01f73c710c44 Version: $LATEST                                                                                                                                                                                                                                                                                                                        |
| 1619414621626 | [csharp-example-extension] Handling invoke from extension: {"eventType":"INVOKE","deadlineMs":1619414651625,"requestId":"ac43b4c9-f01e-47fb-80b8-01f73c710c44","invokedFunctionArn":"arn:aws:lambda:ap-southeast-2:713834996596:function:lbd-as2-ws-tabjul-temp6","tracing":{"type":"X-Amzn-Trace-Id","value":"Root=1-60864e5d-5cd7adb566640b7d6e2500c8;Parent=6f09d8f861b0ebd8;Sampled=0"}}  |
| 1619414621627 | END RequestId: ac43b4c9-f01e-47fb-80b8-01f73c710c44                                                                                                                                                                                                                                                                                                                                           |
| 1619414621627 | REPORT RequestId: ac43b4c9-f01e-47fb-80b8-01f73c710c44 Duration: 1.97 ms Billed Duration: 2 ms Memory Size: 634 MB Max Memory Used: 84 MB                                                                                                                                                                                                                                                     |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

2. Change environment variable to force runtime restart

3. Invoke function twice again

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|   timestamp   |                                                                                                                             message                                                                                                                              |
|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1619415301083 | START RequestId: a5462cf3-df2e-4f0c-bdac-07824fc2adba Version: $LATEST                                                                                                                                                                                           |
| 1619415301322 | Launching csharp-example-extension extension ...                                                                                                                                                                                                                 |
| 1619415301467 | [csharp-example-extension] Registered extension with id = 1b6ad717-f7aa-4e6e-b84b-43af1816da59                                                                                                                                                                   |
| 1619415301467 | EXTENSION Name: csharp-example-extension State: Ready Events: [SHUTDOWN,INVOKE]                                                                                                                                                                                  |
| 1619415301521 | [csharp-example-extension] Handling invoke from extension: {"eventType":"INVOKE","deadlineMs":1619415331467,"requestId":"a5462cf3-df2e-4f0c-bdac-07824fc2adba","invokedFunctionArn":"arn:aws:lambda:ap-southeast-2:713834996596:function:lbd-as2-ws-tabjul-temp6 |
| 1619415301521 | ","tracing":{"type":"X-Amzn-Trace-Id","value":"Root=1-60865104-0ad6b7656e2529c2252f6a4c;Parent=1668c80d7786400a;Sampled=0"}}                                                                                                                                     |
| 1619415301820 | END RequestId: a5462cf3-df2e-4f0c-bdac-07824fc2adba                                                                                                                                                                                                              |
| 1619415301820 | REPORT RequestId: a5462cf3-df2e-4f0c-bdac-07824fc2adba Duration: 352.11 ms Billed Duration: 353 ms Memory Size: 634 MB Max Memory Used: 84 MB Init Duration: 393.78 ms                                                                                           |
| 1619415303860 | START RequestId: d4c41e46-230c-494f-b176-ac763e806883 Version: $LATEST                                                                                                                                                                                           |
| 1619415303866 | [csharp-example-extension] Handling invoke from extension: {"eventType":"INVOKE","deadlineMs":1619415333863,"requestId":"d4c41e46-230c-494f-b176-ac763e806883","invokedFunctionArn":"arn:aws:lambda:ap-southeast-2:713834996596:function:lbd-as2-ws-tabjul-temp6 |
| 1619415303866 | ","tracing":{"type":"X-Amzn-Trace-Id","value":"Root=1-60865107-14eaa89c06b4f92746c1bb88;Parent=29c5ab637bad14bb;Sampled=0"}}                                                                                                                                     |
| 1619415303866 | END RequestId: d4c41e46-230c-494f-b176-ac763e806883                                                                                                                                                                                                              |
| 1619415303866 | REPORT RequestId: d4c41e46-230c-494f-b176-ac763e806883 Duration: 2.43 ms Billed Duration: 3 ms Memory Size: 634 MB Max Memory Used: 84 MB                                                                                                                        |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

As can be seen from the logs, INVOKE events are being received and logged as expected - which makes me think the code is okay. No runtime SHUTDOWN event is logged, even after some hours.

The log I'm expecting to see is generated here:

I raised a Support Case 8257034271 for this issue, but they suggested to raise it here for a more targeted response. The case is still open.

We are trying to use external extension to send our logs directly to S3 from our lambdas to step away from the pain of managing log groups when trying to send logs to a third party vendor.

I tried the python zip file approach first ( [s3-logs-extension-demo-zip-archive] ) but I soon realized that we need to package up the python interpreter along with this extension since our main function runs on .net 6. When i tried following the docs and I uploaded the lambda layer to our .net 6 functions, i would get an error indication "python not found"

then i tried the go-example-logs-api-extension since its cross platform binary and should run on any lambda runtime. The logs indicate the extension started up successfully, registered correctly, created the lambda bucket but no logs were ever written to s3. I checked the cloudtrail logs and permissions look good but it seems to me like the "logs data buffer might be empty even though there are logs." Any suggestion here?

are there any working extensions that i can take from here and run it on my lambdas that have different runtimes?

Hi,

It looks like this extension works only for string data type. If "Data" has an object it returns none.

DDB entry:
{
"pKey": "pKey2",
"sKey": "sKey2",
"Data": {
"Admission": "12345"
}
}

Query:
const options = {
hostname: 'localhost',
port: 4000,
path: '/dynamodb?name=DynamoDbTable-pKey2-sKey2',
method: 'GET'
};

output : {"pKey":"pKey2"}

Hi team, I am encountering below output during subsequent calls to my lambda function. The output from the console is:

Test Event Name
TestEvent

Response
Calling the invoke API action failed with this message: Unexpected non-whitespace character after JSON at position 7 (line 1 column 8)

Function Logs

Request ID

I have followed the instructions in that directory and using the layer published following them.
Tried on runtimes: Python 3.10, Python 3.9
The invocation does not fail for the first time i.e invocation after shutdown event but fails for the subsequent invoke calls.

Thanks

Am I right in saying that the lambda extension layer won't work on ARM64/graviton processors?

Thanks

Hi all,
in nodejs-example-extension example for Node.js platform something during registration phase brakes the debugger when runs locally on VS Code using sam local invoke --port 6767 ... on a Lambda that includes this layer.
The following is a snap of the debug logs when launched from VS Code:

START RequestId: 05da599b-f76d-494e-905b-bead9860f186 Version: $LATEST
nodejs-example-extension  launching extension
Debugger listening on ws://0.0.0.0:6767/95282789-af6d-4c40-b544-b1d70e0650c2
For help, see: https://nodejs.org/en/docs/inspector
Debugger attached.
hello from extension
register
extensionId 0e16eae5-5e84-4b22-a204-c54424793f69
next
Starting inspector on 0.0.0.0:6767 failed: address already in use
invoke
next
...

notice the Starting inspector on 0.0.0.0:6767 failed: address already in use error line in logs, after that all set breakpoints are ignored and it's impossible to debug every Lambdas.

this is the VS Code debug task setup:

{
        "name": "lambda_with_example_layer",
        "type": "pwa-node",
        "request": "attach",
        "address": "localhost",
        "port": 6767,
        "localRoot": "${workspaceFolder}/lambda_with_example_layer",
        "sourceMaps": true,
        "remoteRoot": "/var/task",
        "protocol": "inspector",
        "stopOnEntry": false,
        "skipFiles": [
          "<node_internals>/**"
        ]
}

Is there any debugger/SAM/Docker configuration that i missed? Or is something else deeper in the layer execution flows that broke debugging in a container with SAM local?

Please add the requestId to https://docs.aws.amazon.com/lambda/latest/dg/telemetry-schema-reference.html#telemetry-api-function so we can easily correlate logs with requests.

The Java example integrates with the lifecycle of Lambda. The example is missing the S3 logging similar to other runtimes. Could you please provide a Java example for the S3 logging as well ?

I was trying to follow README steps and deploy java extensions example and noticed a couple of small issues with it:

1. target folder, as well as .jar and .zip files and VSCode .settings folder - all are missing in .gitignore;
2. zip.sh and run.sh scripts have incorrect command ordering, which resulted in the following error:

chmod: extensions/java-example-extension: No such file or directory

Hey team!

We're experiencing an issue while dealing with the Telemetry API when the function has 1k logs.

At some point, we're receiving a platform.logsDropped, likely because we're not succeeding to process the payload sent in a timely fashion (this is what I've understand from the doc)

To make it easier I've attached a PR with a full reproducer. If you invoke the function 10-30 times, you'll see a [dispatcher:Dispatch] LOG DROPPED! meaning that we have indeed received a platform.logsDropped

I've also tried different buffering options (playing with timeoutMs , maxBytes and maxItems ) without any luck.

Thanks!
Max

Hello we are currently working with the corretto 8 runtime and would benefit from using an extension.

Right now our function starts a external process with each invocation that is stopped at the end, given that this process just needs to be closed when the environment receives the shutdown signal.

Our current flow looks like this:

1. Initialize sdk clients and internal variables
2. get objects from s3
3. start process
4. do some work
5. terminate process

given that the work done in step 4 just needs to assume that the process is already running, this gives us that moving steps 3 and 5 to an extension would greatly simplify our code while retaining all retry and fallback logic intact.

you guys have plans to add a sample for java extensions?

I would like to see examples where the lambda extension is written in different language than the actual lambda function. Eg. Lambda extension written in Python and Lambda function in Java or vice versa. Is it possible? Are there any examples like that?

For S3-Logs Extension Demo using .zip archive the makefile name should be corrected to Makefile instead of makefile, otherwise the commmand sam build will fail with error
Makefile not found at /../aws-lambda-extensions/s3-logs-extension-demo-zip-archive/extensionssrc/Makefile