[Python] `Tracecontext` OTel propagator config generates an additional X-Ray span about aws-otel-lambda HOT 7 CLOSED

Hey @michaelhyatt! Thanks for providing logs and steps to reproduce.

I don't think that extra span has anything to do with X-Ray, but if you think it does can you let me know?

I also think that what you are seeing is the expected behavior, if you are following the public AWS OTel docs on Getting Started with Lambda for Python.

This is because the docs include this step:

Add the environment variable AWS_LAMBDA_EXEC_WRAPPER and set to /opt/otel-instrument.

This environment variable is used to run the /opt/otel-instrument script before your def consumer(event, lambda_context) function. That script in opentelemetry-instrument will in turn call a wrapper otel-instrument Python script which will automatically instrument Lambda by creating a trace.

Given the faas namespace in the attributes of Span#2, and its timestamp which both starts and ends before/after Spans#0 and Spans#1 I suspect that this is the span created by the otel_wrapper.py file to encompass them as the overarching parent span.

I was able to see only your 2 spans (I used the SDK exporter not the Collector exporter though) by not setting the AWS_LAMBDA_EXEC_WRAPPER environment variable and configuring the opentelmetry package on Lambda on my own without the help of the otel-instrument script. From what I understand, that script is meant to initialize much of opentelmetry for you automatically and give you a top "Lambda" span (Span#2) from which any subsequent spans will follow so you can see where they descend from on the Service Map. But if you have another use case, the layer is still useful because it provides all the opentelemetry package code for you to be able to configure it how you want 🙂

I hope that provides a solution for you, please let me know if I can answer anything else!

from aws-otel-lambda.

Hi @NathanielRN, thanks for looking into it.

I can see that the span is indeed created by the otel_wrapper.py file. It automatically assumes X-Ray trace context propagation here and I can see that the generated span (no 2 in my log) has parent.id taken from the X-Ray trace ID header.

In my case, I am using a different trace context propagation method. Instead of X-Ray, I am relying on W3C traceparent header, hence is the setting OTEL_PROPAGATORS: "tracecontext". The problem is this code is hardwired to propagate the X-Ray trace ID instead of the one I am sending in the traceparent HTTP header.

In my previous example you can see the mismatch between the parent.ids of the spans:

• Span1.parent.id is injected from traceparent header (not shown there).
• Span0.parent.id is Span1.ID <= this is done by my instrumentation.
• Span2.parent.id is pointing to some other external span.id that is presumably coming from some X-Ray external trace.

So, to summarise, I am perfectly happy with the auto-instrumentation of my lambda code, as long as the trace context is propagated properly respecting the OTEL_PROPAGATORS setting instead of being hard-wired to AWS X-Ray.

from aws-otel-lambda.

@michaelhyatt sure thing! I spoke with my colleague @garrettwegan and I hope I can give you an answer 🙂

Thanks for pointing me to that code! You're absolutely right that there is an assumption that the _X_AMZN_TRACE_ID environment variable contains the context. (Instead of looking at HTTP headers).

I should point out that the code you linked to is in an upstream OpenTelemetry repository and not in this downstream aws-otel-lambda repository (although this repo does use that code from the upstream). The code you see there was contributed as part of our team's initial release to get OpenTelemetry working with Python (and other SDKs) and the initial scope included making sure it worked with AWS X-Ray, which is why you see that it is hard-coded.

Some OTel Lambda implementations in other languages have already evolved to allow for a wider amount of user use-cases like yours. For example, the upstream OTel Lambda JavaScript now gives users the option to disables AWS X-Ray Propagation and use custom code to extract the context out of the Lambda event. Then you can use that custom context form the Lambda event instead of having it look for an X-Ray context from the _X_AMZN_TRACE_ID env variable by default.

For your issue, my suggestion would be to open an issue upstream to update the OTel Python instrumentation to have an option similar to JavaScript. The repo is still in early stages of development so it is taking help to make all the languages better!

As another note, you said this:

instead of the one I am sending in the traceparent HTTP header.

I think you may run into problems here. I have not verified this myself, but from what I understand, the design of Lambda is that it will never have HTTP headers because Lambda cannot be directly invoked by an HTTP request. HTTP headers are replaced by _X_AMZN_TRACE_ID for X-Ray (as a special case) or by "Lambda Events" (in any other case). So you must get your context from a Lambda Event if you have a custom context you want to use. (i.e. you would invoke your Lambda function and insert the HTTP headers into the event that is to be received at the Lambda side).

The ideal future would be one where upstream supports both the X-Ray method of using _X_AMZN_TRACE_ID and the poweruser method way like yourself that uses Lambda Events and an "eventContextExtractor" (like OTel JS) to get the context.

Please let me know if that explains the issue! Unfortunately it's just a limitation of where the project is for now 😞

from aws-otel-lambda.

Thanks, @NathanielRN and @garrettwegan. I created this issue in the upstream opentelemetry-lambda repo.

I wanted to clarify the trace context propagation with Lambdas.

1. Lambda function receives HTTP headers through event["headers"] and using OTel tracecontext extract/inject with the right context propagator setting should take care of using the right getter and setter to extract the parent trace context. A very similar setup can work with SQS message headers and any other protocol supporting metadata attached to the event. So, AWS X-Ray context propagation shouldn't be the only way to propagate trace context between services.
2. I had a similar issue with JS lambdas, but now it has been fixed with v0.23 of JS layer and tracecontext propagation appears to work as expected.
3. I really like all the initialisation that is done within otel-instrument and would like to continue to rely on it without having to initialise OTel myself. And relying on auto instrumentation would be super helpful instead of creating the spans myself, however, I really want the spans to be linked to the external context rather than AWS X-Ray.

Hope it makes sense, happy to move the discussion into the other upstream issue.

from aws-otel-lambda.

Thanks for creating that issue! I appreciate it! 😄

I'll do my best to answer your questions:

Regarding 1, I don't think you can count on headers being in event["headers"]. It seems like you can configure that for API Gateway but it's not defined? It would help to continue this thread on my PR to fix the problem if you know otherwise!

Regarding 2, I'm glad you got a solution! However I looked at the issue you posted and I'm not sure what fix was done to address that? But it's probably best to continue the conversation there if you have any additional questions.

Regarding 3, I think we may encounter difficulties again as pointed out in the thread on my PR but hopefully we do find a solution upstream that works!

from aws-otel-lambda.

This issue is stale because it has been open 90 days with no activity. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled

from aws-otel-lambda.

This issue was closed because it has been marked as stale for 30 days with no activity.

from aws-otel-lambda.