Comments (10)
michaelhyatt
commented on July 24, 2024
1
Tested the ADOT OTel collector traces in Jaeger and it is reporting missing parent traces.
from aws-otel-lambda.
lupengamzn
commented on July 24, 2024
Hi @michaelhyatt ,
Thanks for the feature request.
The Lambda runtime will generate X-Ray segments (Initialization and Overhead) and export them to X-Ray backend using X-Ray Daemon. This process is integrated in Lambda and doesn't provide us any possibility to modify. As a result of that, the tracing SDKs are not able to be instrumented in the area that is outside the Lambda function handler (invocation). Hence, what we can do so far is to apply the trace header from the X-Ray segments, propagate it to the spans created by OTel SDK within invocation and further export them to X-Ray backend using OTel collector so that you're able to see a complete and connected trace graph. I would say changing the default X-Ray Daemon to OTel collector in Lambda would address your issue but currently we're really not confident to say that this will be done in near future. Any specific reason why you would like Initialization and Overhead segments to be exported by OTel collector?
from aws-otel-lambda.
michaelhyatt
commented on July 24, 2024
Hi @lupengamzn
Thanks for getting back to me.
One such use case is to be able to track lambda cold starts and how long they take. I know I can get it through Lambda insights, but this requires ingesting a Cloudwatch logs feed which is not yet supported by ADOT and I would like to use ADOT to receive the data.
Another case is the spans reported by ADOT have parent.id that points at non-existing spans/segments. There are parent segments in X-Ray that represent Lambda runtime and API-GW, so I suspect these missing segments are the ones reported by either the Lambda runtime or API-GW.
This is the trace between 2 lambdas calling each other through the API-GW:
Timing and the 3 spans ADOT is reporting with unlinked (missing parents):
ADOT debug log with producer Span 0 being a child of Span 1, but Span 1 having a non-existent parent e4559134ec0f2a45
There are older events to load. Load more.
--
2021-06-29T00:28:51.050Z info builder/exporters_builder.go:97 Exporter started. { "kind": "exporter", "name": "awsxray" }
2021-06-29T00:28:51.050Z info builder/exporters_builder.go:92 Exporter is starting... { "kind": "exporter", "name": "logging" }
2021-06-29T00:28:51.050Z info builder/exporters_builder.go:97 Exporter started. { "kind": "exporter", "name": "logging" }
2021-06-29T00:28:51.050Z info service/service.go:205 Starting processors...
2021-06-29T00:28:51.050Z info builder/pipelines_builder.go:51 Pipeline is starting... { "pipeline_name": "traces", "pipeline_datatype": "traces" }
2021-06-29T00:28:51.050Z info builder/pipelines_builder.go:62 Pipeline is started. { "pipeline_name": "traces", "pipeline_datatype": "traces" }
2021-06-29T00:28:51.050Z info builder/pipelines_builder.go:51 Pipeline is starting... { "pipeline_name": "metrics", "pipeline_datatype": "metrics" }
2021-06-29T00:28:51.050Z info builder/pipelines_builder.go:62 Pipeline is started. { "pipeline_name": "metrics", "pipeline_datatype": "metrics" }
2021-06-29T00:28:51.050Z info service/service.go:210 Starting receivers...
2021-06-29T00:28:51.050Z info builder/receivers_builder.go:70 Receiver is starting... { "kind": "receiver", "name": "otlp" }
2021-06-29T00:28:51.050Z info otlpreceiver/otlp.go:83 Starting GRPC server on endpoint 0.0.0.0:4317 { "kind": "receiver", "name": "otlp" }
2021-06-29T00:28:51.050Z info otlpreceiver/otlp.go:145 Setting up a second GRPC listener on legacy endpoint 0.0.0.0:55680 { "kind": "receiver", "name": "otlp" }
2021-06-29T00:28:51.050Z info otlpreceiver/otlp.go:83 Starting GRPC server on endpoint 0.0.0.0:55680 { "kind": "receiver", "name": "otlp" }
2021-06-29T00:28:51.050Z info otlpreceiver/otlp.go:101 Starting HTTP server on endpoint 0.0.0.0:55681 { "kind": "receiver", "name": "otlp" }
2021-06-29T00:28:51.050Z info builder/receivers_builder.go:75 Receiver started. { "kind": "receiver", "name": "otlp" }
2021-06-29T00:28:51.050Z info service/application.go:197 Everything is ready. Begin running and processing data.
2021/06/29 00:28:51 Registered extension ID: "c292077b-dbed-4a4a-a787-18d28522fe9c"
2021/06/29 00:28:51 [collector] Register response: {
"functionName": "aws-python-api-worker-project-dev-producer",
"functionVersion": "$LATEST",
"handler": "handler.producer"
}
2021/06/29 00:28:51 [collector] Waiting for event...
START RequestId: c505bd1d-ccfa-435a-b013-1b6847f4abec Version: $LATEST
EXTENSION Name: cloudwatch_lambda_agent State: Ready Events: [INVOKE,SHUTDOWN]
EXTENSION Name: collector State: Ready Events: [INVOKE,SHUTDOWN]
2021/06/29 00:28:52 [collector] Received event: {
"eventType": "INVOKE",
"deadlineMs": 1624926538710,
"requestId": "c505bd1d-ccfa-435a-b013-1b6847f4abec",
"invokedFunctionArn": "arn:aws:lambda:ap-southeast-2:401722391821:function:aws-python-api-worker-project-dev-producer",
"tracing": {
"type": "X-Amzn-Trace-Id",
"value": "Root=1-60da6942-51511f142c4eb98a6271f0b0;Parent=14b8b85f0f6dcb45;Sampled=1"
}
}
2021/06/29 00:28:52 [collector] Waiting for event...
[INFO] 2021-06-29T00:28:52.714Z c505bd1d-ccfa-435a-b013-1b6847f4abec Message: {'resource': '/produce', 'path': '/produce', 'httpMethod': 'POST', 'headers': {'Accept': '*/*', 'CloudFront-Forwarded-Proto': 'https', 'CloudFront-Is-Desktop-Viewer': 'true', 'CloudFront-Is-Mobile-Viewer': 'false', 'CloudFront-Is-SmartTV-Viewer': 'false', 'CloudFront-Is-Tablet-Viewer': 'false', 'CloudFront-Viewer-Country': 'AU', 'content-type': 'application/json', 'Host': 'a76ahpomn5.execute-api.ap-southeast-2.amazonaws.com', 'User-Agent': 'curl/7.64.1', 'Via': '2.0 f64d112a663bbc2a59417646645d304f.cloudfront.net (CloudFront)', 'X-Amz-Cf-Id': '0tcPs_yXqo4W-pyPoHYmMIjEAJi1BIDRa0FpZluMhtNKAR3sE8K1UQ==', 'X-Amzn-Trace-Id': 'Root=1-60da6942-51511f142c4eb98a6271f0b0', 'X-Forwarded-For': '49.177.65.219, 130.176.108.67', 'X-Forwarded-Port': '443', 'X-Forwarded-Proto': 'https'}, 'multiValueHeaders': {'Accept': ['*/*'], 'CloudFront-Forwarded-Proto': ['https'], 'CloudFront-Is-Desktop-Viewer': ['true'], 'CloudFront-Is-Mobile-Viewer': ['false'], 'CloudFront-Is-SmartTV-Viewer': ['false'], 'CloudFront-Is-Tablet-Viewer': ['false'], 'CloudFront-Viewer-Country': ['AU'], 'content-type': ['application/json'], 'Host': ['a76ahpomn5.execute-api.ap-southeast-2.amazonaws.com'], 'User-Agent': ['curl/7.64.1'], 'Via': ['2.0 f64d112a663bbc2a59417646645d304f.cloudfront.net (CloudFront)'], 'X-Amz-Cf-Id': ['0tcPs_yXqo4W-pyPoHYmMIjEAJi1BIDRa0FpZluMhtNKAR3sE8K1UQ=='], 'X-Amzn-Trace-Id': ['Root=1-60da6942-51511f142c4eb98a6271f0b0'], 'X-Forwarded-For': ['49.177.65.219, 130.176.108.67'], 'X-Forwarded-Port': ['443'], 'X-Forwarded-Proto': ['https']}, 'queryStringParameters': None, 'multiValueQueryStringParameters': None, 'pathParameters': None, 'stageVariables': None, 'requestContext': {'resourceId': 'aiuted', 'resourcePath': '/produce', 'httpMethod': 'POST', 'extendedRequestId': 'BqViXEghywMFyrw=', 'requestTime': '29/Jun/2021:00:28:50 +0000', 'path': '/dev/produce', 'accountId': '401722391821', 'protocol': 'HTTP/1.1', 'stage': 'dev', 'domainPrefix': 'a76ahpomn5', 'requestTimeEpoch': 1624926530160, 'requestId': '55ae8ecf-b571-44e3-91eb-53713a069234', 'identity': {'cognitoIdentityPoolId': None, 'accountId': None, 'cognitoIdentityId': None, 'caller': None, 'sourceIp': '49.177.65.219', 'principalOrgId': None, 'accessKey': None, 'cognitoAuthenticationType': None, 'cognitoAuthenticationProvider': None, 'userArn': None, 'userAgent': 'curl/7.64.1', 'user': None}, 'domainName': 'a76ahpomn5.execute-api.ap-southeast-2.amazonaws.com', 'apiId': 'a76ahpomn5'}, 'body': '{"name": "John"}', 'isBase64Encoded': False}
[DEBUG] 2021-06-29T00:28:52.717Z c505bd1d-ccfa-435a-b013-1b6847f4abec Starting new HTTPS connection (1): a76ahpomn5.execute-api.ap-southeast-2.amazonaws.com:443
[DEBUG] 2021-06-29T00:28:55.225Z c505bd1d-ccfa-435a-b013-1b6847f4abec https://a76ahpomn5.execute-api.ap-southeast-2.amazonaws.com:443 "GET /dev/consume HTTP/1.1" 200 0
[DEBUG] 2021-06-29T00:28:55.226Z c505bd1d-ccfa-435a-b013-1b6847f4abec streaming subsegments...
[DEBUG] 2021-06-29T00:28:55.226Z c505bd1d-ccfa-435a-b013-1b6847f4abec sending: { "format": "json", "version": 1 }{ "id": "aa52e2561a104893", "name": "a76ahpomn5.execute-api.ap-southeast-2.amazonaws.com", "start_time": 1624926532.714969, "parent_id": "e4559134ec0f2a45", "in_progress": false, "http": { "request": { "method": "get", "url": "https://a76ahpomn5.execute-api.ap-southeast-2.amazonaws.com/dev/consume" }, "response": { "status": 200 } }, "trace_id": "1-60da6942-51511f142c4eb98a6271f0b0", "type": "subsegment", "namespace": "remote", "end_time": 1624926535.2263606 } to 169.254.79.2:2000.
2021-06-29T00:28:55.236Z INFO loggingexporter/logging_exporter.go:42 TracesExporter { "#spans": 2 }
2021-06-29T00:28:55.236Z DEBUG loggingexporter/logging_exporter.go:48 ResourceSpans #0
Resource labels:
-> telemetry.sdk.language: STRING(python)
-> telemetry.sdk.name: STRING(opentelemetry)
-> telemetry.sdk.version: STRING(1.2.0)
-> cloud.region: STRING(ap-southeast-2)
-> cloud.provider: STRING(aws)
-> faas.name: STRING(aws-python-api-worker-project-dev-producer)
-> faas.version: STRING($LATEST)
-> service.name: STRING(aws-python-api-worker-project-dev-producer)
InstrumentationLibrarySpans #0
InstrumentationLibrary opentelemetry.instrumentation.requests 0.21b0
Span #0
Trace ID : 60da694251511f142c4eb98a6271f0b0
Parent ID : 8b4abf4054e4a63e
ID : a05b87fbb86fafed
Name : HTTP GET
Kind : SPAN_KIND_CLIENT
Start time : 2021-06-29 00:28:52.715105133 +0000 UTC
End time : 2021-06-29 00:28:55.226321929 +0000 UTC
Status code : STATUS_CODE_UNSET
Status message :
Attributes:
-> http.method: STRING(GET)
-> http.url: STRING(https://a76ahpomn5.execute-api.ap-southeast-2.amazonaws.com/dev/consume)
-> http.status_code: INT(200)
Span #1
Trace ID : 60da694251511f142c4eb98a6271f0b0
Parent ID : e4559134ec0f2a45
ID : 8b4abf4054e4a63e
Name : handler.producer
Kind : SPAN_KIND_SERVER
Start time : 2021-06-29 00:28:52.71392873 +0000 UTC
End time : 2021-06-29 00:28:55.228214816 +0000 UTC
Status code : STATUS_CODE_UNSET
Status message :
Attributes:
-> faas.execution: STRING(c505bd1d-ccfa-435a-b013-1b6847f4abec)
-> faas.id: STRING(arn:aws:lambda:ap-southeast-2:401722391821:function:aws-python-api-worker-project-dev-producer)
-> faas.name: STRING(aws-python-api-worker-project-dev-producer)
-> faas.version: STRING($LATEST)
END RequestId: c505bd1d-ccfa-435a-b013-1b6847f4abec
REPORT RequestId: c505bd1d-ccfa-435a-b013-1b6847f4abec Duration: 2567.60 ms Billed Duration: 2568 ms Memory Size: 1024 MB Max Memory Used: 176 MB Init Duration: 1978.96 ms XRAY TraceId: 1-60da6942-51511f142c4eb98a6271f0b0 SegmentId: 14b8b85f0f6dcb45 Sampled: true
2021/06/29 00:34:50 [collector] Received event: {
"eventType": "SHUTDOWN",
"deadlineMs": 1624926892349,
"requestId": "",
"invokedFunctionArn": "",
"tracing": {
"type": "",
"value": ""
}
}
2021-06-29T00:34:50.950Z info service/application.go:212 Received stop test request
2021-06-29T00:34:50.950Z info service/application.go:307 Starting shutdown...
2021-06-29T00:34:50.950Z info service/service.go:225 Stopping receivers...
I also expected the consumer span 0 to be connected to the producer span 0 referring it as it's parent, but it is referring to yet another non-existent parent span.
Consumer ADOT debug log
No older events at this moment. Retry
--
START RequestId: f11c9820-cf23-4805-9ab2-a9b7dbb74ecc Version: $LATEST
LOGS Name: cloudwatch_lambda_agent State: Subscribed Types: [platform]
2021/06/29 00:28:53 [collector] Launching OpenTelemetry Lambda extension, version: v0.1.0
2021-06-29T00:28:53.487Z info service/application.go:277 Starting otelcol... { "Version": "v0.1.0", "NumCPU": 2 }
2021-06-29T00:28:53.487Z info service/application.go:185 Setting up own telemetry...
2021-06-29T00:28:53.487Z info service/application.go:220 Loading configuration...
2021-06-29T00:28:53.490Z info service/application.go:236 Applying configuration...
2021-06-29T00:28:53.493Z info builder/exporters_builder.go:274 Exporter was built. { "kind": "exporter", "exporter": "awsxray" }
2021-06-29T00:28:53.494Z info builder/exporters_builder.go:274 Exporter was built. { "kind": "exporter", "exporter": "logging" }
2021-06-29T00:28:53.495Z info builder/exporters_builder.go:274 Exporter was built. { "kind": "exporter", "exporter": "otlp" }
2021-06-29T00:28:53.496Z info builder/pipelines_builder.go:204 Pipeline was built. { "pipeline_name": "traces", "pipeline_datatype": "traces" }
2021-06-29T00:28:53.496Z info builder/pipelines_builder.go:204 Pipeline was built. { "pipeline_name": "metrics", "pipeline_datatype": "metrics" }
2021-06-29T00:28:53.496Z info builder/receivers_builder.go:230 Receiver was built. { "kind": "receiver", "name": "otlp", "datatype": "traces" }
2021-06-29T00:28:53.496Z info builder/receivers_builder.go:230 Receiver was built. { "kind": "receiver", "name": "otlp", "datatype": "metrics" }
2021-06-29T00:28:53.496Z info service/service.go:155 Starting extensions...
2021-06-29T00:28:53.496Z info service/service.go:200 Starting exporters...
2021-06-29T00:28:53.496Z info builder/exporters_builder.go:92 Exporter is starting... { "kind": "exporter", "name": "awsxray" }
2021-06-29T00:28:53.496Z info builder/exporters_builder.go:97 Exporter started. { "kind": "exporter", "name": "awsxray" }
2021-06-29T00:28:53.496Z info builder/exporters_builder.go:92 Exporter is starting... { "kind": "exporter", "name": "logging" }
2021-06-29T00:28:53.496Z info builder/exporters_builder.go:97 Exporter started. { "kind": "exporter", "name": "logging" }
2021-06-29T00:28:53.496Z info builder/exporters_builder.go:92 Exporter is starting... { "kind": "exporter", "name": "otlp" }
2021-06-29T00:28:53.496Z info builder/exporters_builder.go:97 Exporter started. { "kind": "exporter", "name": "otlp" }
2021-06-29T00:28:53.496Z info service/service.go:205 Starting processors...
2021-06-29T00:28:53.496Z info builder/pipelines_builder.go:51 Pipeline is starting... { "pipeline_name": "traces", "pipeline_datatype": "traces" }
2021-06-29T00:28:53.496Z info builder/pipelines_builder.go:62 Pipeline is started. { "pipeline_name": "traces", "pipeline_datatype": "traces" }
2021-06-29T00:28:53.496Z info builder/pipelines_builder.go:51 Pipeline is starting... { "pipeline_name": "metrics", "pipeline_datatype": "metrics" }
2021-06-29T00:28:53.496Z info builder/pipelines_builder.go:62 Pipeline is started. { "pipeline_name": "metrics", "pipeline_datatype": "metrics" }
2021-06-29T00:28:53.496Z info service/service.go:210 Starting receivers...
2021-06-29T00:28:53.496Z info builder/receivers_builder.go:70 Receiver is starting... { "kind": "receiver", "name": "otlp" }
2021-06-29T00:28:53.496Z info otlpreceiver/otlp.go:83 Starting GRPC server on endpoint 0.0.0.0:4317 { "kind": "receiver", "name": "otlp" }
2021-06-29T00:28:53.496Z info otlpreceiver/otlp.go:145 Setting up a second GRPC listener on legacy endpoint 0.0.0.0:55680 { "kind": "receiver", "name": "otlp" }
2021-06-29T00:28:53.496Z info otlpreceiver/otlp.go:83 Starting GRPC server on endpoint 0.0.0.0:55680 { "kind": "receiver", "name": "otlp" }
2021-06-29T00:28:53.496Z info otlpreceiver/otlp.go:101 Starting HTTP server on endpoint 0.0.0.0:55681 { "kind": "receiver", "name": "otlp" }
2021-06-29T00:28:53.496Z info builder/receivers_builder.go:75 Receiver started. { "kind": "receiver", "name": "otlp" }
2021-06-29T00:28:53.496Z info service/application.go:197 Everything is ready. Begin running and processing data.
2021/06/29 00:28:53 Registered extension ID: "146f9fde-727c-4b6f-820e-d46c417b9c69"
2021/06/29 00:28:53 [collector] Register response: {
"functionName": "aws-python-api-worker-project-dev-consumer",
"functionVersion": "$LATEST",
"handler": "handler.consumer"
}
2021/06/29 00:28:53 [collector] Waiting for event...
EXTENSION Name: cloudwatch_lambda_agent State: Ready Events: [INVOKE,SHUTDOWN]
EXTENSION Name: collector State: Ready Events: [INVOKE,SHUTDOWN]
2021/06/29 00:28:55 [collector] Received event: {
"eventType": "INVOKE",
"deadlineMs": 1624926541168,
"requestId": "f11c9820-cf23-4805-9ab2-a9b7dbb74ecc",
"invokedFunctionArn": "arn:aws:lambda:ap-southeast-2:401722391821:function:aws-python-api-worker-project-dev-consumer",
"tracing": {
"type": "X-Amzn-Trace-Id",
"value": "Root=1-60da6942-51511f142c4eb98a6271f0b0;Parent=45e7cdc610535622;Sampled=1"
}
}
2021/06/29 00:28:55 [collector] Waiting for event...
[INFO] 2021-06-29T00:28:55.171Z f11c9820-cf23-4805-9ab2-a9b7dbb74ecc Message: {'resource': '/consume', 'path': '/consume', 'httpMethod': 'GET', 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'CloudFront-Forwarded-Proto': 'https', 'CloudFront-Is-Desktop-Viewer': 'true', 'CloudFront-Is-Mobile-Viewer': 'false', 'CloudFront-Is-SmartTV-Viewer': 'false', 'CloudFront-Is-Tablet-Viewer': 'false', 'CloudFront-Viewer-Country': 'AU', 'Content-Type': 'application/json', 'Host': 'a76ahpomn5.execute-api.ap-southeast-2.amazonaws.com', 'User-Agent': 'python-requests/2.25.1', 'Via': '1.1 56e387dd6408b297d863ce745cdb7d9e.cloudfront.net (CloudFront)', 'X-Amz-Cf-Id': 'O-uxGB_OVVkIjqUTAB4eM9iQaa1-2qv_Dy-0ab78nKao6waOKT3Bpg==', 'X-Amzn-Trace-Id': 'Self=1-60da6944-62cbb03f69fdc0de67dc8ea4;Root=1-60da6942-51511f142c4eb98a6271f0b0;Parent=aa52e2561a104893;Sampled=1', 'X-Forwarded-For': '13.211.71.78, 64.252.174.85', 'X-Forwarded-Port': '443', 'X-Forwarded-Proto': 'https'}, 'multiValueHeaders': {'Accept': ['*/*'], 'Accept-Encoding': ['gzip, deflate'], 'CloudFront-Forwarded-Proto': ['https'], 'CloudFront-Is-Desktop-Viewer': ['true'], 'CloudFront-Is-Mobile-Viewer': ['false'], 'CloudFront-Is-SmartTV-Viewer': ['false'], 'CloudFront-Is-Tablet-Viewer': ['false'], 'CloudFront-Viewer-Country': ['AU'], 'Content-Type': ['application/json'], 'Host': ['a76ahpomn5.execute-api.ap-southeast-2.amazonaws.com'], 'User-Agent': ['python-requests/2.25.1'], 'Via': ['1.1 56e387dd6408b297d863ce745cdb7d9e.cloudfront.net (CloudFront)'], 'X-Amz-Cf-Id': ['O-uxGB_OVVkIjqUTAB4eM9iQaa1-2qv_Dy-0ab78nKao6waOKT3Bpg=='], 'X-Amzn-Trace-Id': ['Self=1-60da6944-62cbb03f69fdc0de67dc8ea4;Root=1-60da6942-51511f142c4eb98a6271f0b0;Parent=aa52e2561a104893;Sampled=1'], 'X-Forwarded-For': ['13.211.71.78, 64.252.174.85'], 'X-Forwarded-Port': ['443'], 'X-Forwarded-Proto': ['https']}, 'queryStringParameters': None, 'multiValueQueryStringParameters': None, 'pathParameters': None, 'stageVariables': None, 'requestContext': {'resourceId': 's30841', 'resourcePath': '/consume', 'httpMethod': 'GET', 'extendedRequestId': 'BqVixGU7SwMFXSw=', 'requestTime': '29/Jun/2021:00:28:52 +0000', 'path': '/dev/consume', 'accountId': '401722391821', 'protocol': 'HTTP/1.1', 'stage': 'dev', 'domainPrefix': 'a76ahpomn5', 'requestTimeEpoch': 1624926532747, 'requestId': '1f9382a0-227b-499e-a286-1d95fdf25237', 'identity': {'cognitoIdentityPoolId': None, 'accountId': None, 'cognitoIdentityId': None, 'caller': None, 'sourceIp': '13.211.71.78', 'principalOrgId': None, 'accessKey': None, 'cognitoAuthenticationType': None, 'cognitoAuthenticationProvider': None, 'userArn': None, 'userAgent': 'python-requests/2.25.1', 'user': None}, 'domainName': 'a76ahpomn5.execute-api.ap-southeast-2.amazonaws.com', 'apiId': 'a76ahpomn5'}, 'body': None, 'isBase64Encoded': False}
2021-06-29T00:28:55.182Z INFO loggingexporter/logging_exporter.go:42 TracesExporter { "#spans": 1 }
2021-06-29T00:28:55.182Z DEBUG loggingexporter/logging_exporter.go:48 ResourceSpans #0
Resource labels:
-> telemetry.sdk.language: STRING(python)
-> telemetry.sdk.name: STRING(opentelemetry)
-> telemetry.sdk.version: STRING(1.2.0)
-> cloud.region: STRING(ap-southeast-2)
-> cloud.provider: STRING(aws)
-> faas.name: STRING(aws-python-api-worker-project-dev-consumer)
-> faas.version: STRING($LATEST)
-> service.name: STRING(aws-python-api-worker-project-dev-consumer)
InstrumentationLibrarySpans #0
InstrumentationLibrary opentelemetry.instrumentation.aws_lambda 0.16.dev0
Span #0
Trace ID : 60da694251511f142c4eb98a6271f0b0
Parent ID : f1f4a7d678442d55
ID : c632bd0f9ee111ce
Name : handler.consumer
Kind : SPAN_KIND_SERVER
Start time : 2021-06-29 00:28:55.171514198 +0000 UTC
End time : 2021-06-29 00:28:55.17204406 +0000 UTC
Status code : STATUS_CODE_UNSET
Status message :
Attributes:
-> faas.execution: STRING(f11c9820-cf23-4805-9ab2-a9b7dbb74ecc)
-> faas.id: STRING(arn:aws:lambda:ap-southeast-2:401722391821:function:aws-python-api-worker-project-dev-consumer)
-> faas.name: STRING(aws-python-api-worker-project-dev-consumer)
-> faas.version: STRING($LATEST)
END RequestId: f11c9820-cf23-4805-9ab2-a9b7dbb74ecc
REPORT RequestId: f11c9820-cf23-4805-9ab2-a9b7dbb74ecc Duration: 52.70 ms Billed Duration: 53 ms Memory Size: 1024 MB Max Memory Used: 169 MB Init Duration: 1997.45 ms XRAY TraceId: 1-60da6942-51511f142c4eb98a6271f0b0 SegmentId: 45e7cdc610535622 Sampled: true
2021/06/29 00:34:53 [collector] Received event: {
"eventType": "SHUTDOWN",
"deadlineMs": 1624926894919,
"requestId": "",
"invokedFunctionArn": "",
"tracing": {
"type": "",
"value": ""
}
}
2021-06-29T00:34:53.520Z info service/application.go:212 Received stop test request
2021-06-29T00:34:53.520Z info service/application.go:307 Starting shutdown...
2021-06-29T00:34:53.520Z info service/service.go:225 Stopping receivers...
from aws-otel-lambda.
michaelhyatt
commented on July 24, 2024
For the record, leaving the Python code I used to create these two lambdas:
import logging
import os
import requests
from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.core import patch_all
patch_all()
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
URL = os.getenv('CONSUMER_API')
def producer(event, context):
logger.info(f'Message: {event}')
res = requests.get(url=URL, headers={
"Content-Type": "application/json"
})
return {'statusCode': res.status_code}
def consumer(event, context):
logger.info(f'Message: {event}')
return {'statusCode': 200}from aws-otel-lambda.
lupengamzn
commented on July 24, 2024
@michaelhyatt Thanks for providing the context!
The parents for Lambda server spans do exist. They're exported by Daemon, not collector, and that's the reason why you're not seeing them in collector's log. You can open the raw trace context in X-Ray console to see that it should have a segment with Id(e4559134ec0f2a45) there, otherwise you'll not be able to see a complete and connected service graph like the one you provided at all. Can you confirm this on your end? Or does the service graph you provided fail to represent the actual workflow of your service?
from aws-otel-lambda.
michaelhyatt
commented on July 24, 2024
Thanks, @lupengamzn. I can confirm that I can see the parent of the traces I am receiving from the OTel collector being the "Invocation" segments that are visible in X-Ray. Is there a way for these to be reported by the OTel collector as well?
My problem is not having these Invocation and API-GW segments reported by the OTel collector.
from aws-otel-lambda.
lupengamzn
commented on July 24, 2024
@michaelhyatt Thanks for the confirmation.
We can leave this thread open for more observability. At the same time, we'll wait to see if there is any update on Lambda side that we can adopt to get this addressed.
from aws-otel-lambda.
lupengamzn
commented on July 24, 2024
@michaelhyatt, Thanks for bringing this up. We're aware of this situation, and currently using AWS OTel managed Lambda layer on both X-Ray and other tracing service is not fully supported. Sorry for the inconvenience and we'll try to include that in future updates.
from aws-otel-lambda.
github-actions
commented on July 24, 2024
This issue is stale because it has been open 90 days with no activity. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled
from aws-otel-lambda.
github-actions
commented on July 24, 2024
This issue was closed because it has been marked as stale for 30 days with no activity.
from aws-otel-lambda.
Related Issues (20)
- Steps to build and download a private Python lambda layer as a local .zip or .tar file? HOT 1
- Trace is broken in Lambda Java17 Runtime HOT 2
- Traces are not flushed to the server before the lambda terminates HOT 6
- Unable to access Lambda layer for Python with x86_64 architecture HOT 5
- docs: document of supported python version is old HOT 1
- 53612: bind: address already in use HOT 3
- [nodejs lambda layer] no metrics is exported to cloudwatch logs HOT 2
- Decouple processor needs to be added HOT 9
- Does not work with custom metrics HOT 2
- Request to include the awskinesisexporter HOT 2
- awsemf exporter won't create any log group, nor generating any metrics HOT 4
- Public ARN of aws-otel-nodejs-amd64 v1.18.1 is too large HOT 2
- Support for ES6 imports and exports when using aws-otel-nodejs lambda layer HOT 3
- Which version of the collector image contains https://github.com/open-telemetry/opentelemetry-lambda/pull/310? HOT 1
- ADOT Collector Dropping Exports in Lambda Environment HOT 3
- Question - ADOT Lambda Layer Nodejs - SQS AWSTraceHeader is not propogated to Lambda HOT 1
- mongodb spans not generated HOT 1
- Spans are broken if not using X-RAY
- ADOT Lambda Layer local testing HOT 1
- Unable to get tracecontext or trace-id using SDK API | NodeJs
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aws-otel-lambda.