Git Product home page Git Product logo

lumigo-node's Introduction

lumigo-node ๐ŸŒ 

CircleCI codecov npm version License semantic-release

This is @lumigo/tracer, Lumigo's Node.js agent for distributed tracing and performance monitoring.

Supported NodeJS runtimes: 12.x, 14.x, 16.x, and 18.x

Usage

The @lumigo/tracer package allows you to pursue automated metric gathering through Lambda Layers, automated metric gathering and instrumentation through the Serverless framework, or manual metric creation and implementation.

With Lambda Layers

  • When configuring your Lambda functions, include the appropriate Lambda Layer ARN from these tables

Note - Lambda Layers are an optional feature. If you decide to use this capability, the list of Lambda layers available is available here.

With Serverless framework

Manually

To manually configure Lumigo in your Lambda functions:

  • First, install the @lumigo/tracer package using your preferred package manager:
$ npm i @lumigo/tracer
# or
$ yarn add @lumigo/tracer
  • Next, wrap your handler in Lumigo's trace function (note: replace YOUR-TOKEN-HERE with your Lumigo API token):
// javascript
const lumigo = require('@lumigo/tracer')({ token: 'YOUR-TOKEN-HERE' })

const myHandler = async (event, context, callback) => { ... }

exports.handler = lumigo.trace(myHandler)
// typescript
import lumigo from '@lumigo/tracer';

const tracer = lumigo({ token: 'YOUR-TOKEN-HERE' });

export const handler = tracer.trace(async (event, context) => {
  ...
});

Note

For Typescript users, you must add the following to your tsconfig.json file:

{
  ...,
  "esModuleInterop": true,
}

You can read more about it here

  • Your function is now fully instrumented

Configuration

@lumigo/tracer offers several different configuration options. Pass these to the Lambda function as environment variables:

  • LUMIGO_DEBUG=TRUE - Enables debug logging
  • LUMIGO_SECRET_MASKING_REGEX='["regex1", "regex2"]' - Prevents Lumigo from sending keys that match the supplied regular expressions. All regular expressions are case-insensitive. By default, Lumigo applies the following regular expressions: [".*pass.*", ".*key.*", ".*secret.*", ".*credential.*", ".*passphrase.*"].
    • We support more granular masking using the following parameters. If not given, the above configuration is the fallback: LUMIGO_SECRET_MASKING_REGEX_HTTP_REQUEST_BODIES, LUMIGO_SECRET_MASKING_REGEX_HTTP_REQUEST_HEADERS, LUMIGO_SECRET_MASKING_REGEX_HTTP_RESPONSE_BODIES, LUMIGO_SECRET_MASKING_REGEX_HTTP_RESPONSE_HEADERS, LUMIGO_SECRET_MASKING_REGEX_HTTP_QUERY_PARAMS.
  • LUMIGO_SECRET_MASKING_EXACT_PATH='["key1.key2", "key3.key4"]' - Prevents Lumigo from sending keys that match the supplied path (we support nested fields). All paths are case-insensitive.
  • LUMIGO_DOMAINS_SCRUBBER='[".*secret.*"]' - Prevents Lumigo from collecting both request and response details from a list of domains. This accepts a comma-separated list of regular expressions that is JSON-formatted. By default, the tracer uses ["secretsmanager\..*\.amazonaws\.com", "ssm\..*\.amazonaws\.com", "kms\..*\.amazonaws\.com"]. Note - These defaults are overridden when you define a different list of regular expressions.
  • LUMIGO_PROPAGATE_W3C=TRUE - Add W3C TraceContext headers to outgoing HTTP requests. This enables uninterrupted transactions with applications traced with OpenTelemetry.
  • LUMIGO_SWITCH_OFF=TRUE - In the event a critical issue arises, this turns off all actions that Lumigo takes in response to your code. This happens without a deployment, and is picked up on the next function run once the environment variable is present.
  • LUMIGO_AUTO_TAG=key1.key2,key3 - Configure execution tags that will be driven directly from the event for the supplied key (we support nested fields).

Step Functions

If your function is part of a set of step functions, you can add the flag step_function: true to the Lumigo tracer import. Alternatively, you can configure the step function using an environment variable LUMIGO_STEP_FUNCTION=True. When this is active, Lumigo tracks all states in the step function in a single transaction, easing debugging and observability.

const lumigo = require('@lumigo/tracer')({ token: 'DEADBEEF', step_function: true })

Note: the tracer adds the key "_lumigo" to the return value of the function.

If you override the "Parameters" configuration, add "_lumigo.$": "$._lumigo" to ensure this value is still present.

Below is an example configuration for a Lambda function that is part of a step function that has overridden its parameters:

"States": {
    "state1": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-west-2:ACCOUNT:function:FUNCTION_NAME",
      "Parameters": {
          "Changed": "parameters",
          "_lumigo.$": "$._lumigo"
        },
      "Next": "state2"
    },
    "state2": {
      "Type": "pass",
      "End": true
    }
}

Logging Programmatic Errors

With the tracer configured, simply call

console.log("[LUMIGO_LOG] <YOUR_MESSAGE>");

to create custom errors that are visible throughout the platform. This can be used anywhere in your Lambda code, and is included with the @lumigo/tracer package.

Adding Execution Tags

You can add execution tags to a function with dynamic values using the parameter addExecutionTag.

These tags will be searchable from within the Lumigo platform.

Adding tags for Manual tracing

To add a tag to a manual trace statement:

  • Add the following to your code:

    const lumigo = require('@lumigo/tracer')({ token: 'YOUR-TOKEN-HERE' })
  • Add execution tags by using

    lumigo.addExecutionTag('<key>', '<value>');

Adding tags for Auto tracing

To add a tag to an automatically-traced function:

  • Add the following to the top of your handler's .js file:

    const lumigo = require('@lumigo/tracer')
  • Use

    lumigo.addExecutionTag('<key>', '<value>');

    anywhere in your lambda code.

Execution Tag Limitations

Execution tags are subject to the following limitations:

  • The maximum number of tags is 50.
  • Key length must be between 1 and 50.
  • Value length must be between 1 and 70.

Scrubbing Limitations

Secrets scrubbing are subject to the following limitations:

  • Only JSON data secrets scrubbing is supported

lumigo-node's People

Contributors

aisofer avatar aviadmor avatar bboure avatar dependabot[bot] avatar dori-admin avatar doriaviram avatar efimk-lu avatar guymoses avatar harelmo-lumigo avatar mattbillock avatar moshe-shaham-lumigo avatar nadav3396 avatar naorpeled avatar nirbon avatar nirlumigo avatar omrilumigo avatar orrlumigo avatar prr19776 avatar saartochner-lumigo avatar sagi avatar sagivoululumigo avatar semantic-release-bot avatar shaikegross avatar shaileshkumar007 avatar shanishiri avatar therightstuff avatar uri-admin avatar uri-p avatar vladllumigo avatar yonatanhunger avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

lumigo-node's Issues

Add PR template

Is your feature request related to a problem? Please describe.
There are issues templates in this project but there is no PR template. I think it'll be good if we can add a PR template in the project.

Describe the solution you'd like
We can add a PR template.

Describe alternatives you've considered
NA

Additional context
NA

EMFILE: too many files open

Describe the bug

Runtime.UnhandledPromiseRejection: SystemError [ERR_SYSTEM_ERROR]: A system error occurred: uv_os_ho
    at /opt/nodejs/node_modules/@lumigo/tracer/dist/tracer/tracer.js:269:31
    at step (/opt/nodejs/node_modules/@lumigo/tracer/dist/tracer/tracer.js:33:23)
    at Object.next (/opt/nodejs/node_modules/@lumigo/tracer/dist/tracer/tracer.js:14:53)
    at /opt/nodejs/node_modules/@lumigo/tracer/dist/tracer/tracer.js:8:71

This is not the first time we've seen this issue with Lumigo tracer. It happens very occasionally during extreme bursts of traffic.

In this case, our lambda does not make any filesystem calls. We suspect it's Lumigo tracer making too many FS calls.

To Reproduce

Very difficult to reproduce. Happens under extreme burst

Runtime details
AWS_Lambda_nodejs18.x (eu-central-1_

Screenshots
CleanShot 2023-12-07 at 12 26 45

Last Occurrence
11:02:10 PM EST

Questions

  • Tracer makes http calls and filesystem calls. What happens if for example, Lumigo connection fails and retries?
  • How many files from FS would tracer load for a normal stacktrace / log execution?

Security vulnerabilities in axios 0.24.0

The @lumigo/tracer library is using an old version of axios which includes 2 security vulnerabilities.

Axios Cross-Site Request Forgery Vulnerability - GHSA-wf5p-g6vw-rhxx
Follow Redirects improperly handles URLs in the url.parse() function - GHSA-jchw-25xp-jwwc

Can we request that this be brought up to date to the latest version of Axios please?

Neo4j Tracer support

Hi, is there any plan for supporting automatic tracing for neo4j ?

I'd be happy to help and work on a PR if that is something you consider and I can help.

Adding build.yml to run tests in GitHub Actions

Is your feature request related to a problem? Please describe.
npm tests not running on GitHub Actions

Describe the solution you'd like
Adding tests in workflow folder

Describe alternatives you've considered
NA

Additional context
NA

Test Issue

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See an error

Expected behavior
A clear and concise description of what you expected to happen.

Runtime details

  • runtime version (node8, 10x, 12)..

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

TEST

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See an error

Expected behavior
A clear and concise description of what you expected to happen.

Runtime details

  • runtime version (node8, 10x, 12)..

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Integrate in TypeScript Project

Are there docs for using the tracer in a TypeScript project? I can't find any types and thus importing in TypeScript does not work.

"exports is not defined in ES module scope"

Describe the bug
Trying to use Lumigo/Tracer as module in a nodejs14 setup with Lambda. Since my Lambda function is 'module enabled', it can't use "require"- hence the Lumigo-layer throws me errors about using 'require'.

To Reproduce
Steps to reproduce the behavior:

  1. Use Lambda layer or install the package locally
  2. Try to import the package in your function
    import Lumigo from "@lumigo/tracer";
  3. Run your function

Expected behavior
Being able to use the module and Lumigo application

Runtime details
Node14

Errors

{
  "errorType": "ReferenceError",
  "errorMessage": "exports is not defined in ES module scope\nThis file is being treated as an ES module because it has a '.js' file extension and '/var/task/package.json' contains \"type\": \"module\". To treat it as a CommonJS script, rename it to use the '.cjs' file extension.",
  "trace": [
    "ReferenceError: exports is not defined in ES module scope",
    "This file is being treated as an ES module because it has a '.js' file extension and '/var/task/package.json' contains \"type\": \"module\". To treat it as a CommonJS script, rename it to use the '.cjs' file extension.",
    "    at file:///var/task/index.js:180:1",
    "    at ModuleJob.run (internal/modules/esm/module_job.js:183:25)",
    "    at process.runNextTicks [as _tickCallback] (internal/process/task_queues.js:60:5)",
    "    at /var/runtime/deasync.js:23:15",
    "    at _tryAwaitImport (/var/runtime/UserFunction.js:74:12)",
    "    at _tryRequire (/var/runtime/UserFunction.js:162:21)",
    "    at _loadUserApp (/var/runtime/UserFunction.js:197:12)",
    "    at Object.module.exports.load (/var/runtime/UserFunction.js:242:17)",
    "    at Object.<anonymous> (/var/runtime/index.js:43:30)",
    "    at Module._compile (internal/modules/cjs/loader.js:1085:14)"
  ]
}

Tracer should handle traceIdArr greater than 3 and should not be throwing unhandled errors

export const getTraceId = (awsXAmznTraceId) => {

export const getTraceId = (awsXAmznTraceId) => {
  if (!awsXAmznTraceId) {
    throw new Error('Missing _X_AMZN_TRACE_ID in Lambda Env Vars.');
  }

  const traceIdArr = awsXAmznTraceId.split(';');
  if (traceIdArr.length !== 3) {
    throw new Error('Expected 3 semi-colon separated parts in _X_AMZN_TRACE_ID.');
  }

  const traceId = {};
  // XXX Populates Root, Parent and Sampled keys.
  traceIdArr.forEach((item) => {
    const [key, value] = item.split('=');
    traceId[key] = value;
  });

  if (!traceId['Root'] || !traceId['Parent'] || !traceId['Sampled']) {
    throw new Error(`Either Root, Parent or Sampled weren't found in traceId.`);
  }

  // @ts-ignore
  const transactionId = traceId.Root.split('-')[2];

  // @ts-ignore
  traceId.transactionId = transactionId;

  return traceId;
};

Can this be written without ts-ignores?
what does the // XXX comment mean?
Why check that the traceIdArr is exactly 3, if those values are being checked lower down?

Note: this code will break systems that every pass in a awsXAmznTraceId that has extra information attached.

Please Add Support for NodeJS 16

Describe the solution you'd like

Support nodejs v16 based lambdas

Additional context

As AWS lambda now started to support Nodejs 16 runtime, we have migrated our lambdas to Nodejs 16 but we can't get the lumigo setup for those lambdas. Needed to setup lumigo for prod account.

Not handling event['requestContext'] is undefined

(event && event['httpMethod'] && event['requestContext']['stage']) ||

We are getting an error thrown, Cannot read property 'stage' of undefined, when we are manually triggering the lambda via the test tab of the lambda page.

We solved it with a workaround of adding the following property to our lambda event:

"requestContext": {}

There should be a check that event['requestContext'] is defined before accessing event['requestContext']['stage'] and if it's not defined continue normally (return 'invocation')

Require Function Warning During Webpack Build

Describe the bug
We are integrating our node-based AWS Lambda functions with Lumigo and noticed the following warnings in our pipeline logs. Our functions are written in TypeScript and we are able to reproduce the warnings locally running a simple webpack build. I don't believe we are seeing any adverse behavior due to these warnings, but wanted to surface them so they could be addressed just in case. Our code is for an internal closed-source application, so I'm likely unable to provide specific code or configuration examples, but I am happy to consider any further questions and facilitate any troubleshooting as necessary.

Expected behavior
No warnings during a build.

Runtime details
Node: 19.9.0
Webpack: 5.75.0
Lumigo-node: 1.84.2
Typescript: 4.9.4

Logs

WARNING in ./node_modules/@lumigo/tracer/dist/utils/requireUtils.js 18:83-90
Critical dependency: require function is used in a way in which dependencies cannot be statically extracted
    at CommonJsRequireContextDependency.getWarnings (/opt/atlassian/pipelines/agent/build/node_modules/webpack/lib/dependencies/ContextDependency.js:102:18)
    at Compilation.reportDependencyErrorsAndWarnings (/opt/atlassian/pipelines/agent/build/node_modules/webpack/lib/Compilation.js:3132:24)
    at /opt/atlassian/pipelines/agent/build/node_modules/webpack/lib/Compilation.js:2729:28
    at _next2 (eval at create (/opt/atlassian/pipelines/agent/build/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:34:1)
    at eval (eval at create (/opt/atlassian/pipelines/agent/build/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:62:1)
    at /opt/atlassian/pipelines/agent/build/node_modules/webpack/lib/FlagDependencyExportsPlugin.js:385:11
    at /opt/atlassian/pipelines/agent/build/node_modules/neo-async/async.js:2830:7
    at Object.each (/opt/atlassian/pipelines/agent/build/node_modules/neo-async/async.js:2850:39)
    at /opt/atlassian/pipelines/agent/build/node_modules/webpack/lib/FlagDependencyExportsPlugin.js:361:18
    at /opt/atlassian/pipelines/agent/build/node_modules/neo-async/async.js:2830:7
 @ ./node_modules/@lumigo/tracer/dist/hooks/mongodb.js 5:21-53
 @ ./node_modules/@lumigo/tracer/dist/hooks/index.js 5:16-36
 @ ./node_modules/@lumigo/tracer/dist/index.js 18:14-32

WARNING in ./node_modules/@lumigo/tracer/dist/utils/requireUtils.js 25:87-94
Critical dependency: require function is used in a way in which dependencies cannot be statically extracted
    at CommonJsRequireContextDependency.getWarnings (/opt/atlassian/pipelines/agent/build/node_modules/webpack/lib/dependencies/ContextDependency.js:102:18)
    at Compilation.reportDependencyErrorsAndWarnings (/opt/atlassian/pipelines/agent/build/node_modules/webpack/lib/Compilation.js:3132:24)
    at /opt/atlassian/pipelines/agent/build/node_modules/webpack/lib/Compilation.js:2729:28
    at _next2 (eval at create (/opt/atlassian/pipelines/agent/build/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:34:1)
    at eval (eval at create (/opt/atlassian/pipelines/agent/build/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:62:1)
    at /opt/atlassian/pipelines/agent/build/node_modules/webpack/lib/FlagDependencyExportsPlugin.js:385:11
    at /opt/atlassian/pipelines/agent/build/node_modules/neo-async/async.js:2830:7
    at Object.each (/opt/atlassian/pipelines/agent/build/node_modules/neo-async/async.js:2850:39)
    at /opt/atlassian/pipelines/agent/build/node_modules/webpack/lib/FlagDependencyExportsPlugin.js:361:18
    at /opt/atlassian/pipelines/agent/build/node_modules/neo-async/async.js:2830:7
 @ ./node_modules/@lumigo/tracer/dist/hooks/mongodb.js 5:21-53
 @ ./node_modules/@lumigo/tracer/dist/hooks/index.js 5:16-36
 @ ./node_modules/@lumigo/tracer/dist/index.js 18:14-32

Additional context
We do have esModuleInterop: true in our tsconfig.json and tsc does not surface any warnings or errors. The above errors only surface when running webpack.

Execution tags globals list not cleared unless tracing

Describe the bug
I've followed the docs here to add execution tags to our functions. Note that we are not currently tracing these functions, just monitoring their invocations with Lumigo.

It looks like when a busy lambda function is kept alive, the list of execution tags are stored in a global variable that will continue to grow and never be cleared (clearing this list appears to only be performed at the end of tracing, which again - we're not using)

To Reproduce
Steps to reproduce the behavior:

  1. Enable Lumigo integration with your AWS environment (don't enable tracing at this stage)
  2. Follow the docs here to add the node-tracer lib to your lambda function
  3. On invocation, attempt to write an execution tag with lumigo.addExecutionTag('<key>', '<value>');
  4. Repeatedly invoke this function, and once the list of tags has grown to 50, you will see the following message:.

INFO Lumigo Warning: Skipping addExecutionTag: Unable to add tag: maximum number of tags is 50: <key> - <value>

Expected behavior

  • Execution tags should be cleared at the end of each invocation, or a utility function should be exposed to the caller to perform manual clear up (i.e expose clearGlobals)
  • ExecutionTags should be useable without tracing

Runtime details

  • node v16

Screenshots
N/A

Additional context
N/A

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.