Tracer should handle traceIdArr greater than 3 and should not be throwing unhandled errors about lumigo-node HOT 13 CLOSED

Hi, @StampixSMO, we are likely issuing a further hardening, see #433. What Dori linked you is likely going to help, and we keep working to make sure we are getting this bug nailed down for good :-)

from lumigo-node.

Hey, @builtbylane Thanks for reporting, we are checking the issue

from lumigo-node.

Hey, @Nxtra, we are still investigating the root cause and the impact. In the meantime, we released a hotfix in the tracer. We see these errors in the eu-west-1 AWS region for a yet-unknown reason.

Attaching a link to the fix:

• https://github.com/lumigo-io/lumigo-node/releases/tag/v1.80.1

I will update here ASAP when we release a full fix, which also taking care of @builtbylane good comments

from lumigo-node.

This fix will get us back up, however this code on a whole does not feel production-grade:

1. inconsistent patterns for catching errors make it hard to reason with
2. Unhandled errors that would end up blowing up the entire lambda
3. lots of typescript ignores that could easily lead to runtime issues
4. object mutations should not be allowed. Should consider enforcing strict object immutability to prevent unintended mutations

Ultimately, it's the 2nd issue that concerns me the most. There's too many ways in which unexpected data from AWS would not be handled gracefully and end up blowing up our lambda again.

from lumigo-node.

@doriaviram , thanks! Patching it now.

Extra info:
Rolling back to aws-sdk 2.1301 as I first thought did not resolve it.
Only saw it on our dev accounts were we had recently deployed new function versions.
No impact on our test and prod account which were untouched.

from lumigo-node.

@builtbylane Yep, we are going to refactor this part, be sure about that

from lumigo-node.

Without a full refactor, this would make the code safer.

  if (traceIdArr.length < 3) {
    throw new Error('Expected at least 3 semi-colon separated parts in _X_AMZN_TRACE_ID.');
  }

from lumigo-node.

this is a redacted but real _X_AMZN_TRACE_ID that breaks lumigo: Root=1-640f97b4-12345;Parent=1234;Sampled=0;Lineage=1234:0

as a first step, the code should not strictly be looking for exactly 3 parts.

As a followup, there are too many areas where an unexpectedly formed traceId could blow up a lambda. For instance, what if the Root had fewer than 2 hyphens? Is there a reason an error like that should block the execution of a lambda?

from lumigo-node.

@builtbylane , does this break all currently deployed lambdas that are wrapped with the lumigo tracer?

from lumigo-node.

FWIW, I have multiple environments live and the once using "aws-sdk": "2.1301.0" are still executing fine.
Where I upgraded to "aws-sdk": "2.1333.0" I encounter the error above.

from lumigo-node.

This started breaking all our production lambdas all of a sudden that have the tracer enabled ...

stack.0 | Error: Expected 3 semi-colon separated parts in _X_AMZN_TRACE_ID.
-- | --
stack.1 | at Object.getTraceId (/opt/nodejs/node_modules/@lumigo/tracer/dist/utils.js:76:15)
stack.2 | at Object.getSpanInfo (/opt/nodejs/node_modules/@lumigo/tracer/dist/spans/awsSpan.js:35:27)
stack.3 | at Object.getCurrentTransactionId (/opt/nodejs/node_modules/@lumigo/tracer/dist/spans/awsSpan.js:40:20)
stack.4 | at Object.getFunctionSpan (/opt/nodejs/node_modules/@lumigo/tracer/dist/spans/awsSpan.js:90:33)
stack.5 | at /opt/nodejs/node_modules/@lumigo/tracer/dist/tracer.js:298:50
stack.6 | at step (/opt/nodejs/node_modules/@lumigo/tracer/dist/tracer.js:33:23)
stack.7 | at Object.next (/opt/nodejs/node_modules/@lumigo/tracer/dist/tracer.js:14:53)
stack.8 | at /opt/nodejs/node_modules/@lumigo/tracer/dist/tracer.js:8:71
stack.9 | at new Promise (<anonymous>)
stack.10 | at __awaiter (/opt/nodejs/node_modules/@lumigo/tracer/dist/tracer.js:4:12)
stack.11 | at /opt/nodejs/node_modules/@lumigo/tracer/dist/tracer.js:263:61
stack.12 | at Runtime.handler (/opt/nodejs/node_modules/lumigo-auto-instrument/index.js:42:35)
stack.13 | at Runtime.handleOnceNonStreaming (/var/runtime/Runtime.js:74:25)

from lumigo-node.

Hey @StampixSMO can you update to:
https://github.com/lumigo-io/lumigo-node/releases/tag/v1.80.1

from lumigo-node.

The issue was fixed on the PR #432, and another refactor we did to improve the handling of _X_AMZN_TRACE_ID is #433 that closing this issue.

From version 1.80.1, the issue was fixed.
Thanks for reporting @builtbylane @StampixSMO @Nxtra

from lumigo-node.