Instead of node-statsd, consider using hot-shots, a fork of the former that is maintained.

/cc @elbuo8

Related to https://auth0.slack.com/archives/C04SSNEKT/p1507125019000044

jose [Today at 7:50 AM]
in #logs_uncaught

so @U45GBFLKZ, sentry is properly logging uncaugh exceptions but as you mention maybe we can do a better job tagging:
https://github.com/auth0/auth0-server/blob/f2b455007901ec0e13cadfda385a2de6b6b1eb2e/lib/logs/logger.js#L6
https://github.com/auth0/auth0-instrumentation/blob/master/lib/logger.js#L69-L74

2 replies
half-ogre [5 hours ago]

Yeah, the tagging is the issue. Otherwise I can't send just the uncaughts to PD.

half-ogre [5 hours ago]

I haven't found a good extensibility point in Raven for this. I am thinking about making a custom stream that dynamically sets the error level and pushes the tags.

Replaces https://github.com/auth0/dx/issues/198

In order to know what methods area added/removed/deprecated/no longer required to be called/required to be called please add a changelog or keep the readme updated so we always send data to DD when we update and don't realize there is a breaking change (or there is more than one)

Remove these two env vars AWS_ACCESS_KEY_SECRET and AWS_ACCESS_KEY_ID from everywhere since this is not needed when using the AWS sdk properly.

The SDK can take this from either process.env, role, etc.

Currently three errors have been flagged in the customer API repo caused by insecure dependencies. Namely:

• string in swagger-tools
• debug
• moment
• ms

Instead of using pidusage to just get a memory RSS value, use something like this:

setInterval(function() {
  var memUsage = process.memoryUsage();
  agent.metrics.gauge('memory.rss', memUsage.rss);
  agent.metrics.gauge('memory.heapTotal', memUsage.heapTotal);
  agent.metrics.gauge('memory.heapUsed', memUsage.heapUsed);
}, 5000);

https://help.datadoghq.com/hc/en-us/articles/203567579-Why-am-I-seeing-inconsistencies-for-metrics-submitted-to-DogStatsD-

From @elbuo8:
https://auth0.slack.com/archives/C44EZEWCT/p1510164392000301

Extend auth0-instrumentation to support STS Assume role for Kinesis Logs given that there will be more services in different accounts which may want to use auth0-logs pipelines.

We don't longer use this kind of stream and is not useful.

Currently only an error reporting plugin is available to Hapi. We could provide something like metrics.js as well.

Related to this discussion https://auth0.slack.com/archives/C6TSSQD0U/p1507064265000103

Yenkel is suggesting using https://www.npmjs.com/package/debounce

From Verlic:

We do not want to waste time in limitd and debounce must be correct, e.g. if we have 25 exceptions in a second and they are all different is ok. but…. if the exception A is “different” than exception B just because the tenant name changed, then that should be “debounced”. We only need 1 exception. For these we need to understand the scenarios that happened in the past few weeks that greatly increase the usage of Sentry (#sentry_plan_changes )

Replaces https://github.com/auth0/dx/issues/197

We should benchmark the use of all these multiple logging streams at the same time, since this is known to be a resource hog.

I did some initial testing with pino and it seems promising (although didn't collect numbers in a controlled manner, etc). There are a few caveats:

• The data changes slightly: we don't send ProcessInfo or pkg.name anymore.
• No Sentry support (not out-of-the-box with an npm package, at least).
• No serializer support: from what I've seen, this is the only deal breaker. If we can implement it, we're good.

The pino version can be found here.

I like clear interfaces, env or process.env is too much information. Consider using a params like object with camel cased settings as any other node.js module.

We have had problems in the past on modules that receive env like auth0/ext, tenant-audit etc. That code is hard to read, think and the interfaces are not clear.

🚨 The automated release from the master branch failed. 🚨

I recommend you give this issue a high priority, so other packages depending on you could benefit from your bug fixes and new features.

You can find below the list of errors reported by semantic-release. Each one of them has to be resolved in order to automatically publish your package. I’m sure you can resolve this 💪.

Errors are usually caused by a misconfiguration or an authentication problem. With each error reported below you will find explanation and guidance to help you to resolve it.

Once all the errors are resolved, semantic-release will release your package the next time you push a commit to the master branch. You can also manually restart the failed CI job that runs semantic-release.

If you are not sure how to resolve this, here is some links that can help you:

• Usage documentation
• Frequently Asked Questions
• Support channels

If those don’t help, or if this issue is reporting something you think isn’t right, you can always ask the humans behind semantic-release.

No npm token specified.

An npm token must be created and set in the NPM_TOKEN environment variable on your CI environment.

Please make sure to create an npm token and to set it in the NPM_TOKEN environment variable on your CI environment. The token must allow to publish to the registry https://a0us.jfrog.io/a0us/api/npm/npm/.

Good luck with your project ✨

Your semantic-release bot 📦🚀

Replaces https://github.com/auth0/dx/issues/199

• In a Hapi context, init() is a bit of a problem. A consumer must call agent.init() before any plugins actually become available to register, which is contrary to common Hapi plugin conventions.
  • Furthermore, calling init() doesn't allow a la carte usage; it'll try to init logging, error reporting, and metrics at once.
  • init() also attempts to make agent a singleton (though makes no such guarantees). This can make more "granular" usage difficult, if not impossible. Is this necessary?
• By expecting package.json, this module's functionality is tightly coupled to a Node.js context. Is this necessary?

Some ideas:

• initialize things when the Hapi plugin's "register" function is called; not before
• each chunk of functionality should allow explicit initialization--whether this is three (3) separate function calls, or a single function call with appropriate options
• extract the Hapi plugin (and Express middleware for that matter) into their own modules
• expect a single configuration object, not pkg and env
• return a new object every time init() (or whatever) is called. If this is implemented, don't make consumers use new

UPDATE: This module doesn't expect a package.json; just rather something that looks like it.

Reproduction steps

On Node 10:

1. Create a new project mkdir example
2. Initialize NPM cd example; npm init
3. Install auth0-instrumentation npm i auth0-instrumentation

Expected

Package installs correctly.

Received

2 warnings generated.
  CXX(target) Release/obj.target/profiler/src/heap_output_stream.o
sed: ./Release/.deps/Release/obj.target/profiler/src/heap_output_stream.o.d.raw: No such file or directory
  CXX(target) Release/obj.target/profiler/src/heap_graph_node.o
rm: ./Release/.deps/Release/obj.target/profiler/src/heap_output_stream.o.d.raw: No such file or directory
make: *** [Release/obj.target/profiler/src/heap_output_stream.o] Error 1
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:262:23)
gyp ERR! stack     at ChildProcess.emit (events.js:197:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:254:12)
gyp ERR! System Darwin 18.2.0
gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "build" "--fallback-to-build" "--module=/Users/mike/Code/example/node_modules/v8-profiler-node8/build/profiler/v5.7.8/node-v67-darwin-x64/profiler.node" "--module_name=profiler" "--module_path=/Users/mike/Code/example/node_modules/v8-profiler-node8/build/profiler/v5.7.8/node-v67-darwin-x64"
gyp ERR! cwd /Users/mike/Code/example/node_modules/v8-profiler-node8
gyp ERR! node -v v11.10.1
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok
node-pre-gyp ERR! build error
node-pre-gyp ERR! stack Error: Failed to execute '/usr/local/bin/node /usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js build --fallback-to-build --module=/Users/mike/Code/example/node_modules/v8-profiler-node8/build/profiler/v5.7.8/node-v67-darwin-x64/profiler.node --module_name=profiler --module_path=/Users/mike/Code/example/node_modules/v8-profiler-node8/build/profiler/v5.7.8/node-v67-darwin-x64' (1)
node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (/Users/mike/Code/example/node_modules/node-pre-gyp/lib/util/compile.js:83:29)
node-pre-gyp ERR! stack     at ChildProcess.emit (events.js:197:13)
node-pre-gyp ERR! stack     at maybeClose (internal/child_process.js:984:16)
node-pre-gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:265:5)
node-pre-gyp ERR! System Darwin 18.2.0
node-pre-gyp ERR! command "/usr/local/bin/node" "/Users/mike/Code/example/node_modules/.bin/node-pre-gyp" "install" "--fallback-to-build"
node-pre-gyp ERR! cwd /Users/mike/Code/example/node_modules/v8-profiler-node8
node-pre-gyp ERR! node -v v11.10.1
node-pre-gyp ERR! node-pre-gyp -v v0.6.39
node-pre-gyp ERR! not ok
Failed to execute '/usr/local/bin/node /usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js build --fallback-to-build --module=/Users/mike/Code/example/node_modules/v8-profiler-node8/build/profiler/v5.7.8/node-v67-darwin-x64/profiler.node --module_name=profiler --module_path=/Users/mike/Code/example/node_modules/v8-profiler-node8/build/profiler/v5.7.8/node-v67-darwin-x64' (1)
npm WARN [email protected] No description
npm WARN [email protected] No repository field.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-pre-gyp install --fallback-to-build`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/mike/.npm/_logs/2019-03-05T18_47_05_429Z-debug.log
C02VD5H0HTD8:example mike$ ../src/heap_graph_node.cc:2:10: fatal error: 'heap_graph_edge.h' file not found
#include "heap_graph_edge.h"
         ^~~~~~~~~~~~~~~~~~~
1 error generated.
make: *** [Release/obj.target/profiler/src/heap_graph_node.o] Error 1
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:262:23)
gyp ERR! stack     at ChildProcess.emit (events.js:197:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:254:12)
gyp ERR! System Darwin 18.2.0
gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "build" "--fallback-to-build" "--module=/Users/mike/Code/example/node_modules/v8-profiler-node8/build/profiler/v5.7.8/node-v67-darwin-x64/profiler.node" "--module_name=profiler" "--module_path=/Users/mike/Code/example/node_modules/v8-profiler-node8/build/profiler/v5.7.8/node-v67-darwin-x64"
gyp ERR! cwd /Users/mike/Code/example/node_modules/v8-profiler-node8
gyp ERR! node -v v11.10.1
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok

A change was made in #65 to support Node version 8.x. It was a forked version of the standard v8-profiler which had been modified to support Node 8.

So it looks like hyj1991 created his own fork and applied a patch to create the new NPM release. node-inspector/v8-profiler#113

Once this gets merged back to the base and there is a mainline version supporting Node 8, we will probably want to switch this dep back to https://www.npmjs.com/package/v8-profiler.