Prometheus metrics? about envoy-microservices-patterns HOT 11 CLOSED

@mrice32 thanks for adding your comments. I wrote my comments above before realizing that work was actually underway to implement tagging/labels/dimensions for stats in Envoy. I'm curious why tagging wasn't treated as the canonical representation and converted to strings as needed (based on my presumption that labels provide a complete superset of string-based naming), but I'm nowhere near deep enough in the codebase to make a claim about that being a better approach overall.

As far as taking the tagging support in its current state all the way through to Prometheus support, the first thing I would bring up is that Prometheus uses a pull-based collection model, and as such is not a 'sink' in the same way that statsd, etc are. I suspect that it would therefore make more sense to provide Prometheus formatted metrics as another format on the /stats path of the admin endpoint (in addition to statsd-like text and JSON, as currently available). A Prometheus server would then discover the Envoy instances as 'scrape targets' using one of the available service-discovery mechanisms that Prometheus supports, and would periodically issue requests to the appropriate endpoint on each Envoy instance to retrieve stats from it.

In fact, 'Prometheus format' is actually one or both of two encodings: a Protobuf representation and a text representation. Given that there is already Protobuf machinery in play in Envoy's build system, I'm hopeful that leveraging the existing tooling to support Prometheus's Protobuf format is not unreasonable. In any case, the text format is relatively compact (moreso than JSON, certainly), and I would hope that it be supported regardless of Protobuf support, for the purpose of adhoc debugging and inspection.

Does that make sense? Do you see any major roadblocks to adding another format to the /stats endpoint?

from envoy-microservices-patterns.

There is a lot of general interest in native Prometheus support. I opened an issue to track. Can we please discuss there: envoyproxy/envoy#1947

from envoy-microservices-patterns.

With regard to option 2 above, my comment about expected effort is based on the comment here: https://www.envoyproxy.io/envoy/intro/arch_overview/statistics.html

Envoy uses statsd as the statistics output format, though plugging in a different statistics sink would not be difficult.

That said, the comment is with regard to a push-based stats approach, rather than a pull-based one like Prometheus.

from envoy-microservices-patterns.

from envoy-microservices-patterns.

from envoy-microservices-patterns.

I’m going to try #1 for now.

There’s been at least some discussion in the Envoy project about #2: naively natively supporting a tag-based (dimensional/metrics 2.0) approach, here: envoyproxy/envoy#1536. That said, it doesn’t look like anyone is working on it at the moment.

I would like to contribute native support (#2) but I will start by parsing the current output (#1) and then circle back around if it warrants the effort.

from envoy-microservices-patterns.

It would be great to see new stats sinks/exporters added to Envoy. There seems to be a lot of interest around this.

For 2, we just closed 1536 with 1852 just now. This means that tags/dimensions have first class support in Envoy. Please see here and here for documentation on how to configure these tags as a part of the bootstrap config. Please see here for Stats::Metric, which is the base class that encapsulates all of this identifying information for all types of metrics that are flushed to sinks.

However, to take advantage of these tags, there will need to be stats sinks that use them (this currently does not exist in Envoy as the only native stats sink is statsd), which goes back to 1.

Feel free to open an issue in Envoy to discuss creating a sink for Prometheus or if there are any issues you see with the existing tags/dimensions built into Envoy now.

from envoy-microservices-patterns.

I'd also like to note that stats sinks have recently become statically pluggable (like filters). This means that you can link in a factory implementation, provide the corresponding string in the config, and Envoy will use your provided sink without any modification to the Envoy codebase (although it is definitely encouraged to add these sinks to the upstream Envoy repo so other users can look it up by default). See the statsd factory and the tcp statsd sink it creates for an example of how to do this.

from envoy-microservices-patterns.

To respond to your first question, there are two reasons:

1. For the tags to be the canonical representation, Envoy's codebase would have to be opinionated about what is a tag vs a static portion of the stat name. It was seen as beneficial to allow this to be configurable (with defaults) using regexes rather than static and requiring modification of Envoy internals for a user to change.
2. Historical reasons. Envoy used to only support statsd, which (as far as I'm aware) accepts static strings from which tags can be extracted downstream. This meant that the existing canonical representation was originally static strings. There was a decent amount of infrastructure in Envoy built around this assumption (for instance, this is embedded in hot restart). Much of this would need to be redesigned to handle a different canonical representation.

As for adding new formats for scraping stats using a pull model, I think that makes sense. I agree that protobuf is probably the right way to go. As usual, it will probably need to be configurable in the Envoy config, but I don't see any reason this would not work or people would be opposed to having new formats available via /stats. Here is where those stats are dispatched in response to /stats. However, as it is noted in the code, timers (we actually changed that to histograms recently, so that should be updated) and histograms are sent to statsd when recorded and not stored. That means that a pull-based approach will not be able to capture those stats unless we implement some storage system for them (non-trivial since this storage would be unbounded without some way to aggregate or delete old samples). Another thing to note is that currently the tags are not included in the /stats response. The counters and gauges are readily available in that method, though, so that shouldn't be difficult to add.

from envoy-microservices-patterns.

cc @mattklein123 @htuch

from envoy-microservices-patterns.

Yeah, this conversation definitely belongs there. Thanks @mattklein123.

from envoy-microservices-patterns.