kkamkou / node-gelf-pro Goto Github PK
View Code? Open in Web Editor NEWGraylog2 client library for Node.js
Home Page: https://docs.graylog.org/docs/gelf
License: MIT License
Graylog2 client library for Node.js
Home Page: https://docs.graylog.org/docs/gelf
License: MIT License
The length must be changed to 32765
The puzzle 37-7e52116e
from #37 has to be resolved:
node-gelf-pro/lib/adapter/tcp.js
Line 42 in 895eb1d
The puzzle was created by Kanstantsin Kamkou on 03-Apr-18.
Estimate: 60 minutes, role: DEV.
If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.
It is possible to send an empty message like this: log.error(null);
. We should ignore such a message.
Is it possible to have multiple logger instances? Each one would connect to a different Graylog server and have its own configuration.
From my reading of the source code it looks like the logger object is a singleton.
I have message with id field and that was not being transmitted. Looking at code it indeed is deleted from message with comment that it is not possible to send id. Seemed kind of arbitrary restriction so I checked the GELF spec and it has no special meaning to field named id.
Tested modifying code to not drop the id and it worked perfectly so is there any real reason to drop the id field?
I'm using TCP output to send messages to Graylog 2.
Just wanted to my share my experience and suggest an improvement. It would be nice if the "message" parameter had type validation to make sure an object isn't passed in but only a string or error object. Because it's possible to pass in an error - I misunderstood the API documentation and thought it was possible to pass in an object. Since there's no type validation I ended up with this error (in case anyone googles this "bug") in Graylog but got no error from the node-gelf-pro library:
java.lang.IllegalArgumentException: GELF message <message_id> (received from <ip_address>) has invalid "short_message"
Type validation is also nice in case somone accidentally passes in an object (or different type) when they meant to pass in a string. How the codebase currently works that would cause a bug which would be very hard to detect since it would be "silent".
node.js v10:
(node:3170) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
Add a new functionality to send multiple messages at once.
Update repose: lodash
, mocha
, coveralls
Not confirmed. Requires verification. Example:
log.error('An error message', {when: new Date()});
Hi,
I've come to an issue about the maximum number of chunks using UDP adapter.
The code limits it to 20 but the documentation suggests a maximum of 128.
Is there any reason for that limit not following the specs ?
It is possible to call it like this:
glog.info('Example message', 'Exception')
Which is transformed to an array of chars. We should notify in this case.
Having custom levels, aliases are not reliable. We need to move them to default levels.
We have a very busy system and use a TCP connection to graylog. We're starting to see errors relating to open file handles and it seems that we have several thousand sockets open to our graylog server. I was wondering why this TCP adapter doesn't reuse the socket? Why would you need more than one?
Is it possible to use node-gelf-pro
with Angular? I am getting the following errors when I compile the project after following the installation guide.
ERROR in ./node_modules/gelf-pro/lib/adapter/udp.js
Module not found: Error: Can't resolve 'dgram' in '.../node_modules/gelf-pro/lib/adapter'
ERROR in ./node_modules/gelf-pro/lib/adapter/tcp.js
Module not found: Error: Can't resolve 'net' in '.../node_modules/gelf-pro/lib/adapter'
ERROR in ./node_modules/gelf-pro/lib/adapter/tcp-tls.js
Module not found: Error: Can't resolve 'tls' in '.../node_modules/gelf-pro/lib/adapter'
I tried manually installing e.g. dgram
but the error still persists.
Apologies for posting doubts here instead of any bug/feature recommendations. Doing mainly because the support is quick.
I was wondering if the library supports any authentication mechanism to allow only authorised logging requests?
If the timeout has a low value and many logs are generated the node application shuts down with the following exception:
.../gelfTest/node_modules/async/dist/async.js:844
if (fn === null) throw new Error("Callback was already called.");
^
Error: Callback was already called.
at .../gelfTest/node_modules/async/dist/async.js:844:36
at Socket.<anonymous> (.../gelfTest/node_modules/gelf-pro/lib/adapter/tcp.js:37:11)
at emitOne (events.js:96:13)
at Socket.emit (events.js:188:7)
at Socket.<anonymous> (.../gelfTest/node_modules/gelf-pro/lib/adapter/tcp.js:33:18)
at Socket.g (events.js:291:16)
at emitNone (events.js:86:13)
at Socket.emit (events.js:185:7)
at Socket._onTimeout (net.js:334:8)
at tryOnTimeout (timers.js:232:11)
This can be recreated with the following code, if the greylog server is running.
var log = require('gelf-pro');
log.setConfig({
adapterName: 'tcp', // optional; currently supported "udp" and "tcp"; default: udp
adapterOptions: {
family: 4, // tcp only; optional; version of IP stack; default: 4
timeout: 100, // tcp only; optional; default: 10000 (10 sec)
host: '127.0.0.1', // optional; default: 127.0.0.1
port: 12201, // optional; default: 12201
},
});
for (var i = 0; i< 1000; i++) {
log.error("Test");
}
Drop these versions:
Release | Status | Codename | Initial Release | Active LTS Start | Maintenance LTS Start | End-of-life |
---|---|---|---|---|---|---|
v0.10.x | End-of-Life | - | 2013-03-11 | - | 2015-10-01 | 2016-10-31 |
v0.12.x | End-of-Life | - | 2015-02-06 | - | 2016-04-01 | 2016-12-31 |
4.x | End-of-Life | [Argon][] | 2015-09-08 | 2015-10-01 | 2017-04-01 | 2018-04-30 |
5.x | End-of-Life | 2015-10-29 | 2016-06-30 | |||
7.x | End-of-Life | 2016-10-25 | 2017-06-30 | |||
9.x | End-of-Life | 2017-10-01 | 2018-06-30 |
#8 converts all values to strings. Which is not correct
It'll be nice if we can broadcast messages toconsole
alike methods.
I discovered this because of a very chatty service using this library running my server out of available ports to send messages to Graylog.
I'm not sure of a good way to reproduce this issue or test for it, but the errors that I run into are:
null: 'uncaughtException called: Error [ERR_SOCKET_CANNOT_SEND]: Unable to send data'
{{my error callback}}
{ Error: getaddrinfo EAI_AGAIN {{my graylog host}}
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:57:26)
errno: 'EAI_AGAIN',
code: 'EAI_AGAIN',
syscall: 'getaddrinfo',
hostname: '{{my graylog host}}' }
I believe that it may be the result of the callback to async.waterfall
calling send
's callback not as a result of client.close()
.
PR incoming.
It'll be nice to have a native support of the level filtering for the library itself.
filter: function (message, extra) {
return true;
});
IllegalArgumentException[Document contains at least one immense term in field="error_stack" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[83, 121, 110, 116, 97, 120, 69, 114, 114, 111, 114, 58, 32, 47, 104, 111, 109, 101, 47, 107, 105, 114, 105, 108, 108, 47, 119, 111, 114, 107]...', original message: bytes can be at most 32766 in length; got 84418]; nested: MaxBytesLengthExceededException[bytes can be at most 32766 in length; got 84418];
Hi,
Is there a way to turn off this capability to convert a plain object into readable format?
I mean this one:
log.info( 'a new msg goes here', {me: {fname: 'k', lname: 'k', bdate: new Date(2000, 01, 01)}} ); // ^-- extra becomes: {_me_fname: 'k', _me_lname: 'k', _me_bdate: 'Tue Feb 01 2000 00:00:00 GMT+0100 (CET)'}
The minus in the sub-key is allowed, we should replace the minus sign with underscore.
I am writing a good plugin using gelf pro.
I was wondering if you would be open to an option to manually pass the log level. (I am happy to submit a PR)
As hapi logs don't map directly to syslog error levels I want to give the user an option to configure based on the log 'event' and 'tags' doc what level should be used.
Some functionality is not covered.
Let's say you add an additional level like this:
const log = require('gelf-pro');
log.setConfig({
emergency: 0,
alert: 1,
critical: 2,
error: 3,
warning: 4,
notice: 5,
info: 6,
debug: 7,
trace: 8,
});
log.trace()
will be undefined. This is because these methods are defined while requiring the module, with no chances to update the levels or aliases before.
Maybe the methods should be redefined while calling setConfig()
.
IllegalArgumentException[Document contains at least one immense term in field="error_stack" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[82, 97, 110, 103, 101, 69, 114, 114, 111, 114, 58, 32, 77, 97, 120, 105, 109, 117, 109, 32, 99, 97, 108, 108, 32, 115, 116, 97, 99, 107]...', original message: bytes can be at most 32766 in length; got 1028169]; nested: MaxBytesLengthExceededException[bytes can be at most 32766 in length; got 1028169];
Does the UDP Adapter support passing in options that the underlying drgam.createSocket
supports? The reason we ask is that we would like to pass a custom dns lookup function in order to support DNS Caching. If not, is it possible to add the feature?
Dev dependency node-coveralls
is marked as suspicious because of hoek
.
we should update the code
Right now we store only the message, the stack is also needed
Hi! I'm using gelf-pro as a tool to send my app's logs through UDP. Recently I started to get ERR_SOCKET_DGRAM_NOT_RUNNING exception from time to time.
Here's an error trace:
Error [ERR_SOCKET_DGRAM_NOT_RUNNING]: Not running
at new NodeError (node:internal/errors:363:5)
at healthCheck (node:dgram:908:11)
at Socket.close (node:dgram:740:3)
at cbResults (path/to/my/app/node_modules/gelf-pro/lib/adapter/udp.js:75:14)
at /path/to/my/app/node_modules/gelf-pro/lib/adapter/udp.js:118:13
at processTicksAndRejections (node:internal/process/task_queues:83:21)
As specified in the stack trace - the error happens when Socket.close is called.
After investigation I found that this happens only when request body is more than 1388 bytes and gelf-pro divides body in chunks.
The bug is in this code(from /lib/adapter/udp.js:103):
for (var idx in chunks) {
if (isInterrupted) {
break;
}
var chunk = chunks[idx],
packet = buffer
.from(self.specification.magicBytes.concat(packetId, idx, chunksCount, chunk));
client.send(
packet, 0, packet.length, self.options.port, self.options.host,
function (err, bytesSent) {
if (err) { return cbResults(err); }
bytesSentTotal += bytesSent;
/* istanbul ignore else */
if (idx >= chunksCount - 1) {
cbResults(err, bytesSentTotal);
}
}
);
}
More specifically in this closure:
if (idx >= chunksCount - 1) {
cbResults(err, bytesSentTotal);
}
Here idx
is specified to check whether we arrived at the end of chunks array or not, but because of how JavaScript variable scope works what actually happens is - idx is always equal to chunksCount - 1
and cbResults
is called on every loop, not only on the last one (hence ERR_SOCKET_DGRAM_NOT_RUNNING error because we try to close udp connection multiple times). Great example of how and why it works like this is provided here.
So here's a fix I came up with. What's important to note - in order for this fix to work you must use const(or let), not var(because var variable are not scope bound and by using var we actually do the same thing).
diff --git a/node_modules/gelf-pro/lib/adapter/udp.js b/node_modules/gelf-pro/lib/adapter/udp.js
index 58e452d..431b38c 100644
--- a/node_modules/gelf-pro/lib/adapter/udp.js
+++ b/node_modules/gelf-pro/lib/adapter/udp.js
@@ -101,20 +101,21 @@ adapter.send = function (message, callback) {
var packetId = Array.prototype.slice.call(crypto.randomBytes(8));
for (var idx in chunks) {
+ const capturedIndex = idx;
if (isInterrupted) {
break;
}
var chunk = chunks[idx],
packet = buffer
.from(self.specification.magicBytes.concat(packetId, idx, chunksCount, chunk));
client.send(
packet, 0, packet.length, self.options.port, self.options.host,
function (err, bytesSent) {
if (err) { return cbResults(err); }
bytesSentTotal += bytesSent;
/* istanbul ignore else */
- if (idx >= chunksCount - 1) {
+ if (capturedIndex >= chunksCount - 1) {
cbResults(err, bytesSentTotal);
}
}
This issue body was partially generated by patch-package.
It will support react/next.js/gatsby
Does this module support TLS encryption of TCP connection to graylog?
Example (refs #32):
const data = { label: "label", message: "DONE", duration: 123 };
grayLog.info(JSON.stringify(data) + "\x00");
We need to remove the null byte from tcp
and tcp-tls
adapters. Udp works as expected.
This package looks like a fit for something I'm trying to do. But I can't get past the error below. I'm trying to follow the example configuration from the README.
Error: Bad socket type specified. Valid types are: udp4, udp6
at newHandle (dgram.js:59:9)
at new Socket (dgram.js:89:16)
at Object.exports.createSocket (dgram.js:109:10)
at udp._createSocket (/Users/dave/devel/integration-tools/node_modules/gelf-pro/lib/adapter/udp.js:42:16)
at udp.send (/Users/dave/devel/integration-tools/node_modules/gelf-pro/lib/adapter/udp.js:66:21)
at gelf.send (/Users/dave/devel/integration-tools/node_modules/gelf-pro/lib/gelf-pro.js:110:21)
at /Users/dave/devel/integration-tools/node_modules/gelf-pro/lib/gelf-pro.js:152:14
at doNTCallback0 (node.js:419:9)
at process._tickCallback (node.js:348:13)
Hello,
I'm experiencing an issue with your library when I'm using the method setConfig() :
node_modules\gelf-pro\lib\gelf-pro.js:57
this.config = _.merge({}, this.config, conf);
^
TypeError: Cannot set property config of #<Object> which has only a getter
Do you have any fix ?
Sincerly
var grayLog = require("gelf-pro");
grayLog.setConfig({
adapterName: "tcp",
timeout: 1000,
host: "192.168.0.102",
port: 12203,
family: 4
});
// ... later
const data = { label: "label", message: "DONE", duration: 123 };
grayLog.info(JSON.stringify(data) + "\x00");
I know my server and configuration is working because I call bellow from shell and it works since I can see log added to the input.
echo -e '{"label":"get_user","message":"DONE","duration":123.2}\x00' | netcat -w 1 localhost 12203
TIA
Some issues are pretty questionable, nevertheless people need the badge to be green
We must have aliases which are compatible with console
Right now it is not possible to pass an new Error()
instance to the log function
log.emergency(new Error('Test'));
log.emergency('Example', new Error('Test'));
Hi,
I'm trying to use the module with the TCP adapter but seems it doesn't work properly. Every message sent to the graylog server it's has the level set as -1, the short_message
as the whole extra field.
I use the .message
function passing message, level extra and a callback function. I tried to use the al other level function like info
warn
but nothing changes. I also tried to use a transform
function and set those parameters as the extra parameter (extra is a JSON in my case) but nothing change.
Let me know if you need some other info from me.
the specified lodash version has a newly discovered (low severity - https://npmjs.com/advisories/1523) security vulnerability. I've just forked gelf-pro and I'm testing with 4.17.19 which has a fix for the vulnerability.
Hi,
When the udp adapter logs a message which is bigger than one chunk and an error occurs it still tries to send the other chunks. If another chunk fails you will get the following error:
Error: Callback was already called.
at <...>\node_modules\async\dist\async.js:844:36
at doSend (dgram.js:368:7)
at afterDns (dgram.js:358:5)
at GetAddrInfoReqWrap.asyncCallback [as callback] (dns.js:62:16)
This error caused some of your applications to crash.
You can easily reproduce this behavior when you try to send a log message to a non existing / resolvable address (which was the case in our system). See the following test code:
const gelf = require('gelf-pro');
gelf.setConfig({
adapterOptions: {
host: 'aUnresolvableAddressMaybeBecauseOfATypo.com',
},
});
// some randomly generated 3000 byte string to get more than one chunk
const randomLongContent = 'zlWP9hmxjtjWmCxblq7Ivl8rz8wXIkScj5gcprlb1iFoadcxUGmKzRBfO7stNaOxypf6LFNjOEUsar8nkR84WMRj4G5kTbqG3PfvtEEOzJbEniRZRIrOi9laZIjHSAtKfbpe9056M6O3uKAEMtFJgKhcIoO8Zh3LI75NS1yOXPX3JrmL1mvcTSWsyn7bDBwBc42cKdYQQebkbhYZo1jRpivy5ADvA8nZG14qBo7vMX9BKRxkMq6vpKWmZAkrQRNDixMQLMlCM4jNnWxRfQZ0FyVbHrUt8zu0Vza9YWEgevUrX1orBKMqu9QDlISOLSEGuCIj8NejGM99FjM52QGN59Dq1lMctMxPIJEqbRs0hevITfEZcQZo8JbeKCj91mXQK0I1IldmXyZM7RGatSxQm1cc8Sx95mblOpSmHtGjmKgwYYJqgDvEU6bRmOhuPGWT0kFN5sP0Joyb5CutBzkmzRlaaBx3w5GuuD5cx0tj89BA1EwGG8FAWoxbJbXdwZmReHi8T1kfTI9ru2J8mEQMktRfawX00PiyBODxCO5QOJphq1bQX1oUNl0eSztHACe8VxuHCgXkDJzjWMNxREgRdftlEmCh4BbZa4h6t44PzV5H07tNXrATjgfX7ce2XvmxnQ4J6kXNqSonQPF06aIEw9OgsHtzLGbYHILENiQIOsdhTePabXKqTewxsccqXejWKf6KPdtJ4EaDNKqvsaYuTVzRfrGml5HxAunBZYxl8gulbapI2SUpcBrsWAL9bs1VNzKKHSqbEBp7B6JmwxOL39Vpx64CfXr0jdmdNQ965DAcYyVcoSnl4yhQ5FRv3HpEvRGsO7LOFadziLSIfsB9TBr83WPaeLFGEGg3dC2m5OeUs2jjnPZIeTuz2AwakhObyqQFsqaGDmPscl7kNiMBJH0AXilCKlIKjrPyDXDBhQHqQHgqu9qIkdX47iSo8LOP2yY4yyqDhyJXC1OMCXH6C2FPaVtPaq4VuISghv7MIWTitk52g21aMsYziVxzFv6ryjRJfcld9096U6xKpMbGwWrgsCsH2XUpFWHhTSdWQByGs33nkhqTmIxnB2geRL32n3ycDbxqdEZHajSOZZ7SRVNuNiS3619B8txGrQGwfqsG8yy8m9hp8bmKAHLnFnHiZwuK1FxDARNXA8L4WbvAwFVwWFMbfOCvnXlNx5amr1APDj0t3DxF5UzBCJYNQ48S56ljT3dSDJDoBvHROfR9cPKzcdWFSnJ2d8vCoBTMJeE3W8cil93Rfr993ywKEhnOzo32qlFD3RuTJHVSXkMow5J7jwePejUA3SdGuf380PzGq4rtpYx7PqZgFlbENbvEhUud9cqcaG80aVV2iWSuqjhjezGaaEoZoWFsTH9n7oK4X4sI5lHmRhYE94CpU8rROOCUJHWnFraFNqwi0GaiHI4yoOYEZyKiU1JjF5Bfr6OFUpckg3lhne33Y4N5SoUgO0HrLRJg8fOvDajwi5zUnIaKWSEHQH8pzOcvdLSNgyO56EJeFcRLLseCvFKl5RjFGCXKkxPbTRkNx6IwpUNiy5rZZyy8MW1whQHYkOBcyKeynxfRoBLiRmnHIoDk9EM05Es37FBCzfQhF8oCNS46oYjPAh3stFyW1vLXVOM5zFs1l3vuRMz4wDNPEGoJsUpQRGVbpHR6SLf5mYhaJ7oH3bn12802wj9xB7q5hHDduqBYWg9zDW5kk9wyxfqxMFWwTmjGZvG3z12GyK0lrX3tx3xCrm5GvDBlL5cL89aylhUGAIXSNketvykVoGdTa74SRpvIeVLtfo4wQHYGXMYv4ONMnvbzUAR3u5Gzc1n0HjDQUOfIjafvScKSVaVffAGixjj8vjnJE71tlA5zAhuVB7Sb9EeltUOWtnsgUEMB2gu32VzEFvZgYBWAJAOuOMHoAs1zeZ9fInWJa4VLHmdTXc1pWYAam68EaTkzHdN39VVjr7eRmxripKVnUtYRHc2WXgcByc0YuV1U2PfMPKESbw0PdiBd1QHeqeT8EHXlA0i3SGYfRsQAFUEbw58vjBleJwjwxHd5rXfeAvI4dDqCacKTmJGkgRnGW9OCSuYtDkRFqyR0ZO8Y7Vth71nUWwd3B1cgoLmOGe0zAHczRV3OToJEefZEyZC25Ye3VgSPUN1lUzp0KxPeslcCCFeotEavw4AAwvRk6BZ7xIepxsJIzzy9EV6Sxk8piBxPgHlapPQREs172Wg1upFEOpllSitKXG1FaS3SELwmLvNlSZg6SWpXLdQyAd0jeB1WmbGLnWYeGIKuob1Ba9C7LA9Tmh3qYNo6UU8DvT5cGB4hXhTaVLB0Bwqg0IqvNhSvEQQb6vO0IvpYR8bGu8PYrAwdBntIrWuax9OsV8kQd6f8W3CrlCc91BbJNW7U7E6TIgHl4hr3ZLhRFT5btgUllDr5dTykWqxJtSphMrP0xMBNyCKkBpuRlj5IdKvxQWizkq3NTpeHORozV3tjhAC88srExPEdz1RLXa4VGIZY7vOy3D5abwJ4UrF6D1XRizZOMQmYdFknt7vScDeN8hdME8qdUpLLSx5FRgnOnKZiVJtyWFfqaZjXrqnkxYVBUomglIk8URXcBgPZmR82SKuSvJwzpi40llOmJnjetKSmhTTxdfjOOxfkWMdTk8Xt4fQnPoL0X1KjEk63oLud4paCIau3CmiQBqC3t6StVhlwH3TYAy3izGJceUqv7AE03KemEvBxiWIVoHt7l4bkw4mcYxGc9frZLx4xaHo46DeJt2sAPPcLPtN3xrlvcdyaUYeQNhDgXF8Zwlan4LesPNSjYJMG2Ar3EMXbMS8roZs4nVbH2g5MtydHCXGEAaii2JBEG0VflqT8Jbg6c9mcLBPo1XcDHaSAh63pntt9wLeT71yt4ZTWLe4I8vqTS1QPWODnFYowKAktSJNv3MTndcxDuoQI0NpbU4gkzk7ofjPjkN30GD2GZVRL1j66Kf6v40AYWyL3oKfnkKRbe2baI3yY';
gelf.debug(randomLongContent, (err, bytesSent) => {
console.log({ err, bytesSent });
});
Hello!
I am getting the error Property 'info' does not exist on type 'typeof import("gelf-pro")'.
when importing this package:
import * as log from 'gelf-pro';
log.info('Hello world', { test: 123 }, function (err, bytesSent) { // Property 'info' does not exist....
console.log('err, bytes', err, bytesSent);
});
However it works fine when I use it with require
:
const log = require('gelf-pro');
How can I use it with import
? It's a NestJS TS Application.
Would the node-gelf pro
library work with Graylog 3 server?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.