wayfair / statsdcc Goto Github PK

Is statsdcc a drop-in replacement to statsd? Do I need to to run both the proxy and the aggregator, or can I create a statsd compatible setup with the aggregator alone?

It seems that metrics take up to 3 minutes to show up in Grafana (which is rendering from Graphite/Carbon).

I tried configuring different flush frequencies (60 seconds, 30 seconds, 10 seconds and 1 second). I have also tried several worker and UDP thread count without luck.

Our current production metrics obtained from StatsD are rendered every minute without any further lag.

Workers' queues are filling up and proxy starts dropping metrics. I tried increasing the amount of workers but needed 2048 workers to stop dropping metrics but this doesn't sound like the correct approach (besides CPU usage goes to 100%)

I have also tried increasing Boost's lock free queue to 65000 instead of 10000 (I noticed there's a TODO in the code as well) without luck. Is it possible to leave the queue without a fixed size?

I am also noticing a high system CPU usage (which I don't see in aggregator mode).

How do you guys tune this in production?

Are you planning to support multi-metric packets?

my config:

{
    "servers": {
      "udp": {
        "port": 8125
      },
      "tcp": {
        "port": 8125
      }
    },
    "frequency": 10,
    "log_level": "debug",
    "backends": {
      "stdout": true
    }
}

environment: in a docker container with os amzn2

How to repeat:

1. start statsdcc with the config above
2. echo "foo:1|c" | nc -u 127.0.0.1 8125
3. wait time over 10s
4. echo "bar:1|c" | nc -u 127.0.0.1 8125
5. info about foo is printed out
6. bar won't be printed out if no more metric sent to statsdcc

I expected that statsdcc will flush metrics every 10s by automatically. But it seems that the flushing action must be triggered by a metric sending after 10s.
Is it a behavior by design? Or am I missing anything?

I have just setup a statsdcc-proxy with two statsdcc-aggregators and one carbon backend.

After doing this, the metrics dropped to nearly a quarter as they used to be when I had a single statsdcc-aggregator.

I guess that the proxy is hashing the metric so that the same metric is always received by the same statsdcc-aggregator. So, that shouldn't be a problem.

It also seems that I'm losing metrics from certain application nodes that I used to have before adding the proxy.

Any ideas why I'm facing this behavior?

Could you provide list of dependencies for your project? I try to build it according to instruction and get

/usr/local/include/json/value.h:325:10: note:   initializing argument 1 of ‘Json::Value& Json::Value::operator=(Json::Value)’
   Value& operator=(Value other);
          ^
lib/CMakeFiles/proxy_http_server.dir/build.make:62: recipe for target 'lib/CMakeFiles/proxy_http_server.dir/net/servers/http/proxy/http_server.cc.o' failed
make[2]: *** [lib/CMakeFiles/proxy_http_server.dir/net/servers/http/proxy/http_server.cc.o] Error 1
CMakeFiles/Makefile2:218: recipe for target 'lib/CMakeFiles/proxy_http_server.dir/all' failed
make[1]: *** [lib/CMakeFiles/proxy_http_server.dir/all] Error 2
Makefile:94: recipe for target 'all' failed
make: *** [all] Error 

Is there a docker image available for setting up statsdcc?

It seems that once the proxy receives a multi-metrics packet it parses all the metrics, hashes and send each of them individually to the corresponding aggregator.

I'm wondering about the impact that sending a multi-metrics packet to the aggregator can have. This would imply accumulating metrics until certain size is exceed (or certain amount of time has past) and then sending the packet.

Have you thought of this? Are you planning to implement it?