netflix-skunkworks / atlas-native-client Goto Github PK

Currently we have a 10MB limit for the total size of the payload we get from an HTTP GET request if that payload is compressed.

Currently there is a lot of boiler plate when we need to add a new config setting. Mostly motivated by the fact that I wanted to keep the Config class immutable and that it kept growing as new settings were added, but it might make sense to group related settings in a few structs, plus use default values in the Config itself. Maybe have the ConfigManager simply create the Config by passing a JSON document.

It looks like all counters expire after 15m and never come back. I think the problem is the counter doesn't call the Updated method on the base meter type:

https://github.com/Netflix-Skunkworks/atlas-native-client/blob/master/meter/default_counter.h

Gauges do report fine, but they call updated:

https://github.com/Netflix-Skunkworks/atlas-native-client/blob/master/meter/default_gauge.cc#L24

We're currently sending batches of metrics serially, and that is the current bottleneck that limits us to around 1-1.5M time series per reporting machine. Using libcurl we should be able to use its multi interface: https://curl.haxx.se/libcurl/c/libcurl-multi.html to send multiple batches in parallel with no major increase in our footprint.

In atlas-native-client, if a metric has a value of 'nan', the metric silently fails validation (i.e., a warning message is not printed to the logs):

Other validation failures are logged as warnings. Not logging these makes tracking down dropped metrics somewhat difficult.

I noticed NaNs being reported for nodejs.gc.pause, atlas.client.sendToMain, and apache.http.request in a quick sampling. Can send instance information via Slack or another internal channel.

To get an idea of unit test coverage.

Before returning an error to the caller, functions like atlas::util::http::post should retry a number of times for certain types of errors.