wavefronthq / go-metrics-wavefront Goto Github PK

View Code? Open in Web Editor NEW

11.0 29.0 15.0 92 KB

Wavefront plugin for go-metrics

Home Page: https://wavefront.com

License: Apache License 2.0

Go 100.00%

wavefront monitoring go metrics time-series go-metrics

go-metrics-wavefront's Introduction

Warning

VMware has ended active development of this project. this repository will no longer be updated.

go-metrics-wavefront

This is a plugin for go-metrics which adds a Wavefront reporter and a simple abstraction that supports tagging at the host and metric level.

Imports

import (
	metrics "github.com/rcrowley/go-metrics"
	"github.com/wavefronthq/go-metrics-wavefront/reporting"
	"github.com/wavefronthq/wavefront-sdk-go/application"
	"github.com/wavefronthq/wavefront-sdk-go/senders"
)

Set Up a Wavefront Reporter

This SDK provides a WavefrontMetricsReporter that allows you to:

Report metrics to Wavefront at regular intervals or
Manually report metrics to Wavefront

The steps for creating a WavefrontMetricsReporter are:

Create a Wavefront Sender for managing communication with Wavefront.
Create a WavefrontMetricsReporter

1. Set Up a Wavefront Sender

A "Wavefront sender" is an object that implements the low-level interface for sending data to Wavefront. You can choose to send data using either the Wavefront proxy or direct ingestion.

If you have already set up a Wavefront sender for another SDK that will run in the same process, use that one. (For details, see Share a Wavefront Sender.)
Otherwise, follow the steps in Set Up a Wavefront Sender to configure a proxy Sender or a direct Sender.

The following example configures a direct Sender with default direct ingestion properties:

directCfg := &senders.DirectConfiguration{
  Server:               "https://INSTANCE.wavefront.com",
  Token:                "YOUR_API_TOKEN",
}

sender, err := senders.NewDirectSender(directCfg)
if err != nil {
  panic(err)
}

2. Create the WavefrontMetricsReporter

The WavefrontMetricsReporter supports tagging at the host level. Any tags passed to the reporter here will be applied to every metric before being sent to Wavefront.

To create the WavefrontMetricsReporter you initialize it with the sender instance you created in the previous step along with a few other properties:

reporter := reporting.NewMetricsReporter(
  sender,
  reporting.ApplicationTag(application.New("app", "srv")),
  reporting.Source("go-metrics-test"),
  reporting.Prefix("some.prefix"),
  reporting.LogErrors(true),
)

Tagging Metrics

In addition to tagging at the reporter level, you can add tags to individual metrics:

tags := map[string]string{
  "key1": "val1",
  "key2": "val2",
}
counter := metrics.NewCounter() // Create a counter
reporter.RegisterMetric("foo", counter, tags) // will create a 'some.prefix.foo.count' metric with tags
counter.Inc(47)

Extended Code Example

package main

import (
	"fmt"
	"math/rand"
	"os"
	"time"

	metrics "github.com/rcrowley/go-metrics"
	"github.com/wavefronthq/go-metrics-wavefront/reporting"
	"github.com/wavefronthq/wavefront-sdk-go/application"
	"github.com/wavefronthq/wavefront-sdk-go/senders"
)

func main() {

	//Tags we'll add to the metric
	tags := map[string]string{
		"key2": "val2",
		"key1": "val1",
		"key0": "val0",
		"key4": "val4",
		"key3": "val3",
	}

  // Create a direct sender
	directCfg := &senders.DirectConfiguration{
		Server:               "https://" + os.Getenv("WF_INSTANCE") + ".reporting.com",
		Token:                os.Getenv("WF_TOKEN"),
		BatchSize:            10000,
		MaxBufferSize:        50000,
		FlushIntervalSeconds: 1,
	}

	sender, err := senders.NewDirectSender(directCfg)
	if err != nil {
		panic(err)
	}

	reporter := reporting.NewMetricsReporter(
		sender,
		reporting.ApplicationTag(application.New("app", "srv")),
		reporting.Source("go-metrics-test"),
		reporting.Prefix("some.prefix"),
		reporting.LogErrors(true),
	)

	counter := metrics.NewCounter()                //Create a counter
	reporter.RegisterMetric("foo", counter, tags)  // will create a 'some.prefix.foo.count' metric with tags
	counter.Inc(47)

	histogram := reporting.NewHistogram()
	reporter.RegisterMetric("duration", histogram, tags) // will create a 'some.prefix.duration' histogram metric with tags

	histogram2 := reporting.NewHistogram()
	reporter.Register("duration2", histogram2) // will create a 'some.prefix.duration2' histogram metric with no tags

	deltaCounter := metrics.NewCounter()
	reporter.RegisterMetric(reporting.DeltaCounterName("delta.metric"), deltaCounter, tags)
	deltaCounter.Inc(10)

	fmt.Println("Search wavefront: ts(\"some.prefix.foo.count\")")
	fmt.Println("Entering loop to simulate metrics flushing. Hit ctrl+c to cancel")

	for {
		counter.Inc(rand.Int63())
		histogram.Update(rand.Int63())
		histogram2.Update(rand.Int63())
		deltaCounter.Inc(10)
		time.Sleep(time.Second * 10)
	}
}

Golang Runtime Metrics

To enable golang runtime metrics reporting, set the RuntimeMetric flag in reporter to true:

	reporting.NewMetricsReporter(
		sender,
		reporting.ApplicationTag(application.New("app", "srv")),
		reporting.Source("go-metrics-test"),
		reporting.Prefix("some.prefix"),
		reporting.RuntimeMetric(true),
	)
}

go-metrics-wavefront's People

Contributors

Stargazers

Watchers

Forkers

etsangsplk medallia harishspqr akodali18 sliu2013 duboc baburajvelayudhan ahobbs-cb xshipeng shavidissa joannatk keep94 bejelith

go-metrics-wavefront's Issues

Reporter does not always report data

I'm developing an application with Go 1.12

I set the ProxyConfiguraiton() to flush every 60s and Reporter to report every 60s. With this configuration, we observed periods where the data is reported sporadically, maybe 3-5m of data sent, and then a period of time where it is 2-3 minutes of no data sent at all.

2020/11/05 14:38:58 resetting wavefront proxy connection
2020/11/05 14:38:58 write tcp 10.101.3.2:59327->10.100.191.182:2878: i/o timeout
2020/11/05 14:38:58 connected to Wavefront proxy at address: wf-proxy.mgmt.internal.lacework.net:2878

The above messages occur about every minute or so.

My first thought was we had some issue with the data being flushed at the same frequency we are sending (strange but I wanted to see) so I tightened up the Sender interval to every second, so that it would flush data every second. This however made the problem worse. My log is now filled with broken pipe exceptions after about 1-2 minutes of sending data and I never am able to send data again.

I am currently trying to play with values to see what works and does not work so that data will properly report; however, something is not working properly in the library and may be related to changes in Go 1.12 golang/go#35373.

The Close function on a reporting.WavefrontMetricsReporter should either panic if called multiple times or be idempotent

The Close() function sends a message on a channel telling the event loop to exit. If the caller happens to call Close() after the event loop has already exited, the "close" channel will never get read. This will cause Close() to deadlock if the "close" channel happens to be full. The first call to Close() will eventually stop the event loop. The second call to Close() will fill the "close" channel because no one is reading the other end. The third call to Close() will deadlock.

It would be best to panic if Close() is called more than once or make Close() be idempotent meaning successive calls to Close() become no-ops once the Reporter has been closed.

Split repository

The idea of support tags in go-metrics is very good.
So why don't split this repository in two?
Because who want to use only go-metrics with tags can use.

Maybe the Close() function on a reporting.WavefrontMetricsReporter should block until all pending data are sent out and the sender is closed

The way Close() works now is that it returns immediately after advising the event loop to finish up and exit. Therefore, after Close() exists, it could be some time before everything is flushed and the sender is actually closed. This fact makes writing unit tests more difficult. It may be better to have Close() block until the closing is completed. That way when the call to Close() completes, the caller can know that everything has been sent.

This could be accomplished by using a second channel. When the event loop finishes its work, it closes this channel. If Close() reads this channel before returning, Close() will block until the event loop finishes its work.

Source should be a separate method argument of wavefront.WavefrontProxy/wavefront.WavefrontDirect

from README:

  hostTags := map[string]string{
    "source": "go-metrics-test",
  }

  go wavefront.WavefrontProxy(metrics.DefaultRegistry, 1*time.Second, hostTags, "some.prefix", addr)

Maybe source should be a separate method argument, not just one entry in hostTags map.....

README.md needs to add deltaCounter example

https://github.com/wavefrontHQ/go-metrics-wavefront/blob/master/test/tests.go#L69-L71

Implement variance and stddev

Make application tags optional for creating a reporter

ApplicationTags are a wavefront DT related concept, so I think the generic libraries should not make it mandatory.

Start/stop wavefrontReporter function

According to README.md, currently we use goroutine or a long-lasting func() to have the wavefrontReporter (either proxy or direct-ingestion)

  go wavefront.WavefrontDirect(metrics.DefaultRegistry, 5*time.Second, hostTags, "direct.prefix", server, token)

I need to use channel (maybe considered as a hacky way) to stop the wavefrontReporter. It would be nice if we can stop the reporter by calling a stop() function. Also, maybe we can consider to have some abstraction, so the users only need to call wavefront.WavefrontReporter() instead of WavefrontDirect and WavefrontProxy respectively.

LICENSE

Please add a license. Thanks!