Add option to remove tags from graphite backend about gostatsd HOT 10 CLOSED

I see what's going on, the timestamp that's sent with the metrics to graphite never changes. When I reverted the change from using ts = counter.Timestamp to using now := time.Now().Unix() everything works as it should. Adding a few more debug statements in graphite.go show that the timer.Timestamp doesn't change over time.

I suspect it's because in https://github.com/atlassian/gostatsd/blob/better-graphite-support/statsd/aggregator.go#L271 if the timer is found in a.Timers, the timestamp is never updated, it's only set when the timer is first created. Same pattern for the other receive*() methods.

from gostatsd.

Looks like out behaviour is significantly different from what original statsd does. Should make it more configurable and it should mimic the original behaviour of course.

from gostatsd.

@rtkmhart could you try better-graphite-support branch to verify it works for you?

from gostatsd.

Thanks for the quick response! Unfortunately the format the graphite backend sends out is like this

stats_counts.mike.test.counter %!f(int64=10) 1465832174
stats.mike.test.counter %!d(float64=0.9985744043244373) 1465832174

Setup to test this is as follows (all on the same host) with two config files:
config1.toml:

[statsdaemon]
    address = "localhost:9125"

config2.toml:

[graphite]
    address = "localhost:2003"

• Run netcat pretending to be graphite: nc -lk 2003
• Run instance with graphite backend: ./gostatsd --backends graphite --config-path ./config2.toml --console-addr ":9126" --flush-interval 10s --verbose --metrics-addr ":9125"
• Run instance with statsd backend: ./gostatsd --backends "statsdaemon" --config-path ./config1.toml --console-addr ":8126" --flush-interval 10s --verbose
• Send metrics: while [ 1 ]; do echo "mike.test.counter:1|c" | nc -uq1 localhost 8125; done

All output from netcat has the formatting string instead of a proper value.

from gostatsd.

I think the format strings are backwards for the Counters section, this seems to work for me. diff as follows:

diff --git a/backend/backends/graphite/graphite.go b/backend/backends/graphite/graphite.go
index 533bd8e..710cae1 100644
--- a/backend/backends/graphite/graphite.go
+++ b/backend/backends/graphite/graphite.go
@@ -119,8 +119,8 @@ func (client *client) preparePayload(metrics *types.MetricMap) *bytes.Buffer {
        if client.legacyNamespace {
                metrics.Counters.Each(func(key, tagsKey string, counter types.Counter) {
                        k := sk(key)
-                       fmt.Fprintf(buf, "stats_counts.%s%s %f %d\n", k, client.globalSuffix, counter.Value, now)
-                       fmt.Fprintf(buf, "%s%s%s %d %d\n", client.counterNamespace, k, client.globalSuffix, counter.PerSecond, now)
+                       fmt.Fprintf(buf, "stats_counts.%s%s %d %d\n", k, client.globalSuffix, counter.Value, now)
+                       fmt.Fprintf(buf, "%s%s%s %f %d\n", client.counterNamespace, k, client.globalSuffix, counter.PerSecond, now)
                })
        } else {
                metrics.Counters.Each(func(key, tagsKey string, counter types.Counter) {

from gostatsd.

@rtkmhart Yeah, it was incorrect. I wrote tests and caught few more issues. Could you try the same branch once again?

from gostatsd.

@ash2k Thanks! There's something odd about the graphite backend that I can't figure out yet. tcpdump and netcat show that the output to graphite from etsy/statsd and gostatsd is identical (i'm probably missing something?), but if I run some timer metrics through etsy/statsd and gostatsd, graphite happily stores everything from etsy/statsd and barely anything from gostatsd. I will keep digging, here's what I was doing to replicate:

• Generate timers with random values: while [ 1 ]; do METRIC="mike.test.counter:$RANDOM|ms"; echo $METRIC; echo $METRIC | nc -uq1 localhost 8125; done
• curl command against graphite: curl http://graphitehost/render/?target=stats.timers.mike.test.counter.mean&format=json&from=-5minute" |json_pp

When using etsy/statsd, every value is filled, but when gostatsd, almost none. Thoughts?

from gostatsd.

@rtkmhart I've pushed a fix, let me know if it works for you. Thanks!

from gostatsd.

@ash2k works perfectly, thank you!

from gostatsd.

@rtkmhart FYI I've just released 0.14.1 with this issue fixed.

from gostatsd.