in a single View of AWS X-RAY , i should see an end-to-end tracing path of Lambda functions , API - Gateway and Dynamodb about aws-otel-lambda HOT 9 CLOSED

Hi @niko-achilles - this seems like it should basically work. Could you clarify

"i) HTTP-Client" - is this an instrumented client? Where does it run / does this app use opentelemetry?

Lambda Function "--calls-->" API-Gateway endpoint - to confirm, these calls use axios?

Does your lambda function define any opentelemetry variables, such as OTEL_PROPAGATORS?

from aws-otel-lambda.

Hi @anuraaga ,

here is my code base of lambda aws functions: https://github.com/niko-achilles/serverless-otel-lambda/tree/with-aws-observability

In summary:
i) http client is NOT an instrumented client. It is postman which invokes simple the API Gateway:
HTTP-Client --calls --> API-Gateway endpoint "/dev/views"

ii) Lambda functions which call the API Gateway use axios.

The Lambda functions define the following env. variables:
AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler

What do you mean with OTEL_PROPAGATORS ? is there anything that i should provide as variables in order to get a single tracing path for my invocations ?

from aws-otel-lambda.

Nope you don't need to set any variables - on the flip side if OTEL_PROPAGATORS had been set it would take priority and possibly break the propagation but if unset it should be OK.

Do you happen to be able to view the headers received by the second API gateway? It should have an X-Amzn-Trace-Id header but if that isn't being populated (and instead traceparent is) it would indicate an issue in our propagator initialization code. We'll also try to reproduce with your example.

from aws-otel-lambda.

X-Amzn-Trace-Id is in the response headers. With an invocation from Postman i get the following
X-Amzn-Trace-Id: Root=1-60bf08d2-44087a7b434eafaa085b7447

from aws-otel-lambda.

But on the second API Gateway call from lambda i see in the cloudwatch logs the following

the X-Amzn-Trace-Id has adifferent value, X-Amzn-Trace-Id: Root=1-60bf08d4-4f0f330571cd87a368f722cd;Parent=50db2dae58582ecb;Sampled=1

Registering OpenTelemetry
--
EXTENSION	Name: collector	State: Ready	Events: [INVOKE,SHUTDOWN]
2021/06/08 06:06:14 [collector] Received event: {
"eventType": "INVOKE",
"deadlineMs": 1623132380973,
"requestId": "b569ae98-e326-41e3-b2f0-2b1650a339d4",
"invokedFunctionArn": "arn:aws:lambda:eu-west-2:372677612161:function:open-telemetry-story-views-dev-getView",
"tracing": {
"type": "X-Amzn-Trace-Id",
"value": "Root=1-60bf08d4-4f0f330571cd87a368f722cd;Parent=50db2dae58582ecb;Sampled=1"
}
}

from aws-otel-lambda.

Hi @anuraaga ,

any update on this ?
i am attaching screenshot of the traces i see in my X-RAY console views, but as mentioned on the first message above, i was expecting to have as a result a single tracing context with following spans:
HTTP-Client --calls --> API-Gateway endpoint "/dev/views" --calls--> Lambda Function --calls--> API-Gateway endpoint "/dev/views/{id}"--calls--> Lambda Function --calls--> Dynamodb

But i get the following as result of tracing context(s)

Traces View of X-Ray

HTTP-Client --calls --> API-Gateway endpoint "/dev/views" --calls--> Lambda Function --calls--> API-Gateway endpoint "/dev/views/{id}"

API-Gateway endpoint "/dev/views/{id}" --calls--> Lambda Function --calls--> Dynamodb

Service Map View of X-Ray:

from aws-otel-lambda.

I need to try to reproduce this on my end I think as otherwise things seem set up fine. Give me a bit to try out your repository, thanks for the repro.

from aws-otel-lambda.

hi @anuraaga .

I changed the Layer version to arn:aws:lambda:\<region>:901920570463:layer:aws-otel-nodejs-ver-0-19-0:1
and it works, which means i get a single Context with Spans related to the single Context.

Before with my issue described above i used the Layer version arn:aws:lambda:\<region>:901920570463:layer:aws-otel-nodejs-ver-0-18-0:1

A snapshot of my Traces Map view in AWS

A snapshot of Traces Waterfall view in AWS

also for reference the code on github is updated with the changes.
https://github.com/niko-achilles/serverless-otel-lambda/tree/with-aws-observability

This issue can be closed i think.

from aws-otel-lambda.

@niko-achilles Thanks a lot for keeping at this and confirming the new version works ok, that's great news!

from aws-otel-lambda.