[nodejs] tracer.startSpan() occasionally returns non-recording span with aws-otel-nodejs 1.0 layer about aws-otel-lambda HOT 8 CLOSED

Hi @NathanielRN, there are examples expanded in the pdf, does that have the info you need?

from aws-otel-lambda.

I should've mentioned I have enabled debug logging in collector.yaml.
Attached is the effect when the otel span created via the layer tracer is of the non-recording vs recording type.

As you can see, logging from the layer is missing when the otel span created via the layer tracer is of the non-recording type.

Extra logging appears in CloudWatch when the otel span created via the layer tracer is of the recording type.

I guess this isn't a surprise, but I should've mentioned it!

from aws-otel-lambda.

Hi thanks for your issue! In the follow up to this issue here open-telemetry/opentelemetry-lambda#201, the poster said these are the same issues. I thought it best to consolidate the issues and move the discussion to that one if that's okay with you?

Thanks for your through deep dive! The PDF is excellent for understanding your issue. I will be referencing them in that issue above.

from aws-otel-lambda.

Hi @NathanielRN - yes we're part of the same team, and I'm the one who raised the same issue with aws support that I believe you're also aware of, so it's fine to consolidate them.

from aws-otel-lambda.

@simonmarshall In the last 2 pictures you posted, could you expand the logs for the isRecording= lines? I was hoping to see what the spanContext was when those logs were sent.

from aws-otel-lambda.

@NathanielRN One thing I noticed today (based off what you said in my github issue) is that sometimes the amazon x-ray trace header is coming through with sampled 0. An example of this is in @simonmarshall's screenshot above the first image. Where sampled=0 is giving a isRecording=false span whereas sampled=1 is giving a isRecording=true. I haven't been able to check through all the instances that we've seen but I can definitely say I've seen the same thing multiple times.

I'm not entirely sure what sets this sampled field on the aws trace header as all of the events are coming from the same source. Would this have an impact?

from aws-otel-lambda.

Sorry for the delay in response @Ancient-Dragon, but in short yes, I believe this ;Sampled=0 would have a large effect as I mentioned in the other issue open-telemetry/opentelemetry-lambda#201. Posted my findings and suggested solution there.

Thanks @simonmarshall I found the information I was looking for!

from aws-otel-lambda.

Closing because we solved it in open-telemetry/opentelemetry-lambda#201.

Lambda has a default sampling rate of 5% which cannot be changed.

Instead, we can change the OTel Sampler to ALWAYS sample so we get traces.

from aws-otel-lambda.