netflix-skunkworks / atlas-native-client Goto Github PK

Currently we have a 10MB limit for the total size of the payload we get from an HTTP GET request if that payload is compressed.

It looks like all counters expire after 15m and never come back. I think the problem is the counter doesn't call the Updated method on the base meter type:

https://github.com/Netflix-Skunkworks/atlas-native-client/blob/master/meter/default_counter.h

Gauges do report fine, but they call updated:

https://github.com/Netflix-Skunkworks/atlas-native-client/blob/master/meter/default_gauge.cc#L24

In atlas-native-client, if a metric has a value of 'nan', the metric silently fails validation (i.e., a warning message is not printed to the logs):

Other validation failures are logged as warnings. Not logging these makes tracking down dropped metrics somewhat difficult.

I noticed NaNs being reported for nodejs.gc.pause, atlas.client.sendToMain, and apache.http.request in a quick sampling. Can send instance information via Slack or another internal channel.

We're currently sending batches of metrics serially, and that is the current bottleneck that limits us to around 1-1.5M time series per reporting machine. Using libcurl we should be able to use its multi interface: https://curl.haxx.se/libcurl/c/libcurl-multi.html to send multiple batches in parallel with no major increase in our footprint.

Currently there is a lot of boiler plate when we need to add a new config setting. Mostly motivated by the fact that I wanted to keep the Config class immutable and that it kept growing as new settings were added, but it might make sense to group related settings in a few structs, plus use default values in the Config itself. Maybe have the ConfigManager simply create the Config by passing a JSON document.

To get an idea of unit test coverage.

Before returning an error to the caller, functions like atlas::util::http::post should retry a number of times for certain types of errors.