Git Product home page Git Product logo

statsd-librato-backend's Introduction

StatsD Librato backend

npm Travis npm npm

BREAKING CHANGES: Starting with version 2.0.0 statsd-librato-backend requires a Librato account that supports tagged metrics.

If your Librato account doesn't yet support tagged metrics or you are using a heroku addon, please use the 0.1.x version.


Overview

This is a pluggable backend for StatsD, which publishes stats to Librato.

Requirements

Installation

$ cd /path/to/statsd
$ npm install statsd-librato-backend

Configuration

You will need to add the following to your StatsD config file.

librato: {
  email:  "[email protected]",
  token:  "ca98e2bc23b1bfd0cbe9041e824f610491129bb952d52ca4ac22cf3eab5a1c32"
}

Example Full Configuration File:

{
  librato: {
    email:  "[email protected]",
    token:  "ca98e2bc23b1bfd0cbe9041e824f610491129bb952d52ca4ac22cf3eab5a1c32"
  }
  , backends: ["statsd-librato-backend"]
  , port: 8125
  , keyNameSanitize: false
}

The email and token settings can be found on your Librato account settings page.

Enabling

Add the statsd-librato-backend backend to the list of StatsD backends in the StatsD configuration file:

{
  backends: ["statsd-librato-backend"]
}

Start/restart the statsd daemon and your StatsD metrics should now be pushed to your Librato account.

Additional configuration options

The Librato backend also supports the following optional configuration options under the top-level librato hash:

Parameter Description
snapTime Measurement timestamps are snapped to this interval (specified in seconds). This makes it easier to align measurements sent from multiple statsd instances on a single graph. Default is to use the flush interval time.
countersAsGauges A boolean that controls whether StatsD counters are sent to Librato as gauge values (default) or as counters. When set to true (default), the backend will send the aggregate value of all increment/decrement operations during a flush period as a gauge measurement to Librato.

When set to false, the backend will track the running value of all counters and submit the current absolute value to Librato as acounter. This will require some additional memory overhead and processing time to track the running value of all counters.
skipInternalMetrics Boolean of whether to skip publishing of internal statsd metrics. This includes all metrics beginning with 'statsd.' and the metric numStats. Defaults to true, implying they are not sent.
retryDelaySecs How long to wait before retrying a failed request, in seconds.
postTimeoutSecs Max time for POST requests to Librato, in seconds.
includeMetrics An array of JavaScript regular expressions. Only metrics that match any of the regular expressions will be sent to Librato. Defaults to an empty array.

{includeMetrics: [/^my.included.metrics/, /^my.specifically.included.metric$/]}
excludeMetrics An array of JavaScript regular expressions. Metrics which match any of the regular expressions will NOT be sent to Librato. If includedMetrics is specified, then patterns will be matched against the resulting list of included metrics. Defaults to an empty array.

{excludeMetrics: [/^my.excluded.metrics/, /^my.specifically.excluded.metric$/]}
globalPrefix A string to prepend to all measurement names sent to Librato. If set, a dot will automatically be added as separator between prefix and measurement name.
writeToLegacy Boolean of whether to also send metrics with the legacy source dimension. Defaults to false. Intended for users with hybrid accounts that support both tags and sources to help with the migration to tags. Set the source in the StatsD config file:

librato: {
email: "[email protected]",
token: "ca98e2bc23b1bfd0cbe9041e82d52ca4ac22cf3eab5a1c32",
source: "unique-per-statsd-instance"
}

Reducing published data for inactive stats

By default StatsD will push a zero value for any counter that does not receive an update during a flush interval. Similarly, it will continue to push the last seen value of any gauge that hasn't received an update during the flush interval. This is required for some backend systems that can not handle sporadic metric reports and therefore require a fixed frequency of incoming metrics. However, it requires StatsD to track all known gauges and counters and means that published payloads are inflated with zero-fill data.

Librato can handle sporadic metric publishing at non-fixed frequencies. Any "zero filling" of graphs is handled at display time on the frontend. Therefore, when using the Librato backend it is beneficial for bandwidth and measurement-pricing costs to reduce the amount of data sent to Librato. In the StatsD configuration file it is recommended that you enable the following top-level configuration directive to reduce the amount of zero-fill data StatsD sends:

{
   deleteIdleStats: true
}

You can configure your metric in Librato to display the gaps between sporadic reports in a variety of ways. Visit the knowledge base article to see how to change the display attributes.

Publishing to Graphite and Librato simultaneously

You can push metrics to Graphite and Librato simultaneously as you evaluate Librato. Just include both backends in the backends variable:

{
  backends: [ "./backends/graphite", "statsd-librato-backend" ],
  ...
}

See the statsd manpage for more information.

Using Proxy

If you want to use statsd-librato-backend througth a proxy you should install https-proxy-agent module:

    $npm install https-proxy-agent

After that you should add the proxy config to the StatsD config file in the librato configuration section:

{
  "librato" : {
    "proxy" : {
      "uri" : "http://127.0.0.01:8080"
    }
  }
}

That configuration will proxy requests via a proxy listening on localhost on port 8080. You can also use an https proxy by setting the protocol to https in the URI.

Tags

Starting in version 2.x.x and higher, this functionality is enabled by default. If you are interested in using this feature but your Librato account is not enabled for tags, please send us an email at [email protected] and request access. Otherwise, see this branch for the legacy version 0.1.7.

Our backend plugin offers basic tagging support for your metrics you submit to Librato. You can specify what tags you want to submit to Librato using the tags config in the librato configuration section of the StatsD config file:

{
  "librato" : {
    "tags": { "os" : "ubuntu", "host" : "production-web-server-1", ... }
  }
}

Once your config has been updated, all metrics submitted to Librato will include your defined tags.

We also support tags at the per-stat level should you need more detailed tagging. We provide a naming syntax for your stats so you can submit tags for each stat. That syntax is as follows:

metric.name#tag1=value,tag2=value:value

Starting with a #, you would pass in a comma-separated list of tags and we will parse out the tags and values. Given the above example, a stat matching the above syntax will be submitted as metric to Librato with a name of metric.name, a value of value and with the tags tag1=value and tag2=value. You are welcome to use any statsd client of your choosing.

Please note that in order to use tags, the statsd config option keyNameSanitize must be set to false to properly parse tags out of your stat name.

Docker

You may use bin/statsd-librato to easily bootstrap the daemon inside a container.

Invoking this via CMD or ENTRYPOINT will create a simple configuration and run the statsd daemon with this backend enabled, listening on 8125.

The following environment variables are available to customize:

  • LIBRATO_EMAIL
  • LIBRATO_TOKEN
  • LIBRATO_SOURCE

Development

If you want to contribute:

  1. Clone your fork
  2. yarn install
  3. Hack away
  4. If you are adding new functionality, document it in the README
  5. for tests, run yarn test
  6. Push the branch up to GitHub
  7. Send a pull request

statsd-librato-backend's People

Contributors

bluemaex avatar bmarini avatar brandonvfx avatar bryanmikaelian avatar caevyn avatar ecstasy2 avatar epmartini avatar geisbruch avatar gmckeever avatar jderrett avatar johanneswuerbach avatar mathieucarbou avatar mauriciogentile avatar mbeale avatar mheffner avatar miketang415 avatar nextmat avatar pat avatar petegoo avatar plmwong avatar rmblstrp avatar till avatar tlrobinson avatar tnation14 avatar tomazk avatar vcimala avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

statsd-librato-backend's Issues

extract source from metric name

would there be any interest in somehow (there can be multiple approaches to this) extracting source from metric name? That way people could use similar logical grouping between different statsd backends and still control aggregation on librato end.

Support statsd histograms

Not sure what the best way to represent these would be. Maybe send the individual buckets as separate metrics?

Use keep-alive connection to Librato API

Maintain one (or more) connections to the API with HTTP keep-alive across report intervals. This should reduce the report time POST cost by removing SSL setup costs. May reduce timeouts seen at high-frequency report intervals.

v0.1.8 is not in NPM

Hi, we at Codeship are trying to use the newly released version 0.1.8 and it does not seem to be published to NPM. Here is the error we are getting:

-----> Node.js app detected
-----> Creating runtime environment
       
       NPM_CONFIG_LOGLEVEL=error
       NPM_CONFIG_PRODUCTION=true
       NODE_VERBOSE=false
       NODE_ENV=production
       NODE_MODULES_CACHE=true
-----> Installing binaries
       engines.node (package.json):  6.10.0
       engines.npm (package.json):   3.10.10
       engines.yarn (package.json):  unspecified (use default)
       
       Resolving node version 6.10.0...
       Downloading and installing node 6.10.0...
       npm 3.10.10 already installed with node
       Resolving yarn version 0.28.4...
       Downloading and installing yarn (0.28.4)...
       Installed yarn 0.28.4
-----> Restoring cache
       Skipping cache restore (not-found)
-----> Building dependencies
       Installing node modules (yarn.lock)
       yarn install v0.28.4
       [1/4] Resolving packages...
       error Couldn't find any versions for "statsd-librato-backend" that matches "0.1.8"
       info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
-----> Build failed

Better migration path for users switching to tags

@jderrett and I have been talking and there have been some bumps in the road for users migrating from 0.1.7 to 2.x.x. We are considering making a change to the writeToLegacy behavior that would stop writes to the new API endpoint unless tags exist on the measurement if writeToLegacy is true. This config option will continue to be defaulted to false

sourceRegex does not seem to work when loading from config.js file.

I'm having some trouble using the sourceRegex option added in #18

When I use the following configuration file, JsonLint complains, as it should... it's not valid JSON.

{
    "flushInterval": 10000,
    "graphitePort": 2003,
    "librato": {
        "token": "LIBRATO_TOKEN",
        "email": "[email protected]",
        "sourceRegex": /^([^\.]+)\./
    },
   "graphiteHost": "localhost",
   "deleteIdleStats": true,
   "port": 8125,
   "backends": [
       "./backends/graphite",
       "statsd-librato-backend"
   ]
}

I thought that maybe the regex should be a string: ie.

...
"sourceRegex": "/^([^\.]+)\./"
...

But when I looked at the librato.js code and it seems the line directly uses the sourceRegex variable in the match call, rather than creating a new RegExp(sourceRegex) object.

I can make a quick pull request to fix this, but I wanted to verify that I wasn't misunderstanding this.

Use log level management rather than debug boolean

After running into some issues pushing metrics to Librato, and not seeing any issues in our logs, we discovered that certain errors (specifically 40X level errors) are only written while debug=true.

It would be great if this was determined using a real log level system (WARN, ERROR, INFO, DEBUG, VERBOSE). Running production services in debug mode is usually frowned upon in my experience.

http proxy is not working using node 0.12.7

Hi,
Unfortunately, updating the nodejs version, the librato backend is not working anymore.

More precisely, we are getting the following excpetion

_http_client.js:73
    throw new TypeError('Request path contains unescaped characters.');
          ^
TypeError: Request path contains unescaped characters.
    at new ClientRequest (_http_client.js:73:11)
    at TunnelingAgent.exports.request (http.js:49:10)
    at TunnelingAgent.createSocket (/opt/statsd/node_modules/tunnel/lib/tunnel.js:116:25)
    at TunnelingAgent.createSecureSocket [as createSocket] (/opt/emarsys/statsd/node_modules/tunnel/lib/tunnel.js:188:41)
    at TunnelingAgent.addRequest (/opt/statsd/node_modules/tunnel/lib/tunnel.js:80:8)
    at new ClientRequest (_http_client.js:154:16)
    at Object.exports.request (http.js:49:10)
    at Object.exports.request (https.js:136:15)
    at post_payload (/opt/statsd/node_modules/statsd-librato-backend/lib/librato.js:64:18)
    at post_metrics (/opt/statsd/node_modules/statsd-librato-backend/lib/librato.js:149:3)

As you can see is a problem in the tunnel library, I was wondering If it would make sense to use another library.

README.md change require

The text "contersAsGauges" has to be rename to "countersAsGauges" - in README.md. The spelling mistake is confusing to the user who is completely new for statsd-librato-backend.

Metrics become unsynchronized over time

I have noticed the following behavior when using a flushInterval of 10 seconds. Metrics sent by statsd are slowly showing up later and later on the librato interface. From the debug output it looks like statsd is sending the right data at the right time, but by logging more info I see that callbacks are actually running much later than expected.

Example after running for 10 minutes:

20 Mar 06:27:28 - Posted metrics to Librato: payload = {"gauges":[{"name":"instances","value":0},{"name":"pending_instances","value":0},{"name":"running_instances","value":0},{"name":"terminating_instances","value":0}],"counters":[],"measure_time":1363778840,"source":"vm-148-120.uc.futuregrid.org"}
20 Mar 06:27:33 - DEBUG: instances:0|g
20 Mar 06:27:33 - DEBUG: pending_instances:0|g
20 Mar 06:27:33 - DEBUG: running_instances:0|g
20 Mar 06:27:33 - DEBUG: terminating_instances:0|g
20 Mar 06:27:35 - PAYLOAD: {"gauges":[{"name":"instances","value":0},{"name":"pending_instances","value":0},{"name":"running_instances","value":0},{"name":"terminating_instances","value":0}],"counters":[],"measure_time":1363778730,"source":"vm-148-120.uc.futuregrid.org"}
20 Mar 06:27:35 - STATUS: 200
20 Mar 06:27:35 - HEADERS: {"content-type":"application/json;charset=utf-8","date":"Wed, 20 Mar 2013 11:27:35 GMT","server":"nginx/1.0.15","status":"200 OK","content-length":"0","connection":"keep-alive"}

The first line of log comes from between write and end here:

There lines with PAYLOAD, STATUS, and HEADERS were added in the request callback handling the response:

// Retry 5xx codes

Note that there is a difference of 110 seconds between the two measure_time values.

I found a simple workaround: disable connection pooling and keep-alive with the following patch:

diff --git a/lib/librato.js b/lib/librato.js
index 40b809e..52ccf86 100644
--- a/lib/librato.js
+++ b/lib/librato.js
@@ -126,6 +126,7 @@ var post_metrics = function(ts, gauges, counters)
     port: parsed_host["port"] || 443,
     path: '/v1/metrics',
     method: 'POST',
+    agent: false,
     headers: {
       "Authorization": basicAuthHeader,
       "Content-Length": payload.length,

I am using nodejs 0.10.0, statsd 0.6.0, and statsd-librato-backend 0.0.4.

Make counterAsGauges:true the default

We've found that in most cases, reporting statsd counters as Librato gauges provides a better default for most use cases. Swap the out-of-box default for a better experience. Must consider how to appropriately handle the upgrade path.

Object has no method "watch"

I'm really stumped by this one. I deployed my app after working only on the front-end, having not changed anything regarding statsd, but all of sudden this error appears and 503's my app:

4 Jan 02:55:13 - reading config file: config/statsd.js 
node.js:134 
        ^ 
started with pid 6 

        throw e; // process.nextTick error, or 'error' event on first tick 
TypeError: Object #<Object> has no method 'watch' 
    at new <anonymous> (/app/node_modules/statsd/lib/config.js:26:6) 
    at Object.configFile (/app/node_modules/statsd/lib/config.js:36:16) 
    at Object.<anonymous> (/app/node_modules/statsd/stats.js:146:8) 
    at Module._compile (module.js:404:26) 
    at Object..js (module.js:410:10) 
    at Module.load (module.js:336:31) 
    at Function._load (module.js:297:12) 
    at Array.<anonymous> (module.js:423:10) 
    at EventEmitter._tickCallback (node.js:126:26) 
exited with code 1 
sending SIGTERM to all processes 

Both statsd and statsd-librato-backend dependencies have been using the same versions since day one, and I haven't moved the config file. I'm running this using the multi-buildpack on Heroku.

I realize this error is coming from the statsd library, but wondered if you folks at Librato experienced anything similar recently. I'm really scratching my head here. Any ideas?

Gauge y-axis properties missing.

I think the gauges should set the proper y-axis properties. They're described here: http://dev.librato.com/v1/metric-attributes. I think it makes sense to set the display_units_long to "Milliseconds" and the display_units_short to "ms". This is the most common case. You cannot correctly stack metrics inside librato unless they have the same y-axis values. This change would make out of the box experience much nicer.

Unable to process JSON

Seeing this in logs with version 2.0.2:

7 Mar 15:45:18 - DEBUG: Sending Payload: {"time":1488930300,"tags":{"host":"auth9.us-west-1"},"measurements":[{"name":"auth-server.request.auth.started","value":79171,"tags":{"source":"auth9.us-west-1"},"source":"auth9.us-west-1"},{"name":"auth-server.request.auth.abort","value":56851,"tags":{"source":"auth9.us-west-1"},"source":"auth9.us-west-1"},{"name":"auth-server.request.auth.completed","value":79157,"tags":{"source":"auth9.us-west-1"},"source":"auth9.us-west-1"},{"name":"auth-server.request.more_than_1","value":1983,"tags":{"source":"auth9.us-west-1"},"source":"auth9.us-west-1"},{"name":"auth-server.debug_timing.soter.get-time","count":192677,"sum":0,"min":0.020981,"max":1352.562189,"tags":{"source":"auth9.us-west-1"},"source":"auth9.us-west-1"},{"name":"auth-server.debug_timing.soter.batchset-time","count":399,"sum":0,"min":2.07901,"max":256.52194,"tags":{"source":"auth9.us-west-1"},"source":"auth9.us-west-1"},{"name":"auth-server.debug_timing.request_1.authenticator.validate","count":1983,"sum":0,"min":1356.827974,"max":4656.633854,"tags":{"source":"auth9.us-west-1"},"source":"auth9.us-west-1"},{"name":"auth-server.debug_timing.soter.set-time","count":1,"sum":0,"min":5.872965,"max":5.872965,"tags":{"source":"auth9.us-west-1"},"source":"auth9.us-west-1"},{"name":"auth-server.debug_timing.request_1.pool.get","count":1,"sum":0,"min":1352.202892,"max":1352.202892,"tags":{"source":"auth9.us-west-1"},"source":"auth9.us-west-1"},{"name":"auth-server.debug_timing.request_1.authcache.get","count":1,"sum":0,"min":1352.558136,"max":1352.558136,"tags":{"source":"auth9.us-west-1"},"source":"auth9.us-west-1"},{"name":"auth-server.debug_timing.request_1.authenticator.fetch_with_fallback","count":1,"sum":0,"min":1352.654934,"max":1352.654934,"tags":{"source":"auth9.us-west-1"},"source":"auth9.us-west-1"}]}
7 Mar 15:45:18 - LOG_ERR: Failed to post to Librato: HTTP 400: {"code":400,"message":"Unable to process JSON"}

Rolled back to version 0.1.7 and it's working.

Log historical metrics through statsd

Librato metrics support supplying a measure_time property (not longer than 120mins old) on the payload.

Is there a mechanism within statsd that we can measure to also provide this value on a per-metric basis?

Invalid payload generated from statsd timings

Saw a payload like:

{"name":"foo.duration","count":9,"sum":null,"sum_squares":null,"min":null,"max":1234},
{"name":"foo.duration.90","count":8,"sum":null,"sum_squares":null,"min":null,"max":456},

This is invalid because sum is not set, but count is. Not sure how that is possible unless there is corruption from statsd itself.

Tag based logging is on by default

Tag based logging is seem to be on by default in this library, but it didn't seem to be on by default in Librato. I'm currently seeing this error with the latest client and have to install the older version of this library to bypass this:

27 Feb 14:38:56 - LOG_ERR: Failed to post to Librato: HTTP 403: {"errors":["Tag based measurements are not enabled for this account. Contact [email protected] for more information"]}```

Backend logging all by default

Hi,

We're deploying statsd in Kubernetes, and logging to stdout. We currently get everything in our logs, including "DEBUG" messages, e.g. https://github.com/librato/statsd-librato-backend/blob/master/lib/librato.js#L64-L66.

Our config does not set the debug value in statsd, it seems something is forcing this logAll to be true - could you please help us out?

Thanks!

{
  librato: {
    email: process.env.LIBRATO_EMAIL,
    token: process.env.LIBRATO_TOKEN,
    host: false,
    tags: {
      cluster: process.env.CLUSTER_NAME,
      environment: process.env.CLUSTER_ENVIRONMENT
    }
  },
  backends: ['statsd-librato-backend'],
  percentThreshold: [95, 99],
  port: 8125,
  deleteIdleStats: true,
  keyNameSanitize: false
}

Get Measurements error!

When I realize this GET with NodeJS it of the error:

{
 "errors": [
    "Tag based measurements are not enabled for this account. Contact [email protected] for more 
      information"
  ]
}

I didn't find any example this get using JavaScript.

Someone can help-me?

Create a tag

I'm not comfortable running master. :) Can you guys tag this? 0.1.0 or whatever you think?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.