g-research / geras Goto Github PK

As Geras has knowledge of __name__ and resolves it itself we can fairly easily implement a filter on names. Add allowed and disallowed metric names regexes (probably simple flags with disallowed checked first to allow easy blocking).

Need to consider error handling, an error to the user for blocked metrics makes sense (to be clear that they are actually blocked, I'm thinking this would mostly be used as a rapid response to problematic queries). We don't consider a query that isn't allowed an error (e.g. an example allow would be \w+\..* as OpenTSDB metrics usually have a dot in them -- things like up can go to both thanos-store and Geras, we don't want to return errors for those).

See https://github.com/G-Research/geras/pull/10/files#r317046683

If a query doesn't expand to any named metrics, then don't warn about !~ usage, in order to avoid warning people about any usage of !~ on their Thanos instance.

• Add dockerfile
• Add CircleCI build for Docker (#42)
• Add integration test
  • FakeTSDB (#43)
  • Spin up Geras, Thanos query and FakeTSDB in CircleCI config
  • Run queries against prometheus v1 query API (via Thanos)

Potentially use circleci to check our docker build, maybe use for some kind of integration test? (Probably needs docker swarm, seems like circleci supports that somehow...)

Turn regexp matches on lists (e.g. as generated by Grafana list dropdowns), into literal_or.

Potentially also useful to implement NRE.

• Build on CircleCI
• Use GitHub package repo

It would be useful to have https://godoc.org/golang.org/x/net/trace support.

gRPC inbound and then HTTP outbound. Could maybe be done in a generic way (e.g. via gRPC interceptors like https://github.com/grpc-ecosystem/go-grpc-middleware/blob/master/tracing/opentracing/server_interceptors.go, but maybe easier to add explicit trace calls to each method for now).

https://promcon.io/2017-munich/slides/staleness-in-prometheus-2-0.pdf
https://www.robustperception.io/staleness-and-promql

We could add staleness markers, e.g. if data is regularly arriving, add a marker after expected interval * 2... But this won't work for all metrics. Consider a scheme like geras_allowstale: as a prefix when querying?

Alternatively maybe this is inferring too much about the data we should force users to work this out via range queries and/or timestamp().

Comment on #16, need to update for new labelset stuff thanos-io/thanos@05f81a5#diff-4ac32a78649ca5bdd8e0ba38b7006a1e

We use literal_or, unfortunately that results in pipes becoming meaningful.

There doesn't appear to be a way to actually escape this properly, see code at https://github.com/OpenTSDB/opentsdb/blob/master/src/query/filter/TagVLiteralOrFilter.java#L72.

See http://opentsdb.net/docs/build/html/user_guide/query/filters.html

Use https://golang.org/pkg/regexp/#Regexp.LiteralPrefix and https://golang.org/pkg/sort/#SearchStrings for a simple binary search?

See also #11.

Currently on private branch, get merged.

I get error like below

this is my docker-compose.yml, the geras is master version

If a query is instantaneous then use /api/query/last to potentially reduce query cost http://opentsdb.net/docs/build/html/api_http/query/last.html

Will need a heuristic like mintime == maxtime and in last ~10 seconds to figure out if it's instantaneous, but that should be ok.

Look into using http://opentsdb.net/docs/build/html/user_guide/query/stats.html to get some query cost stats, maybe log some details, maybe some would make good metrics to export?

Currently geras will send a single query to OpenTSDB for each Series request, therefore potentially needing an unbounded amount of memory.

Split up queries (needs thought about how to combine with downsampling if done, #31) and stream back the data as found.

Later:

• Makes it possible to paralleise queries, limit overall data volume, etc.

Make Info return an error if unable to talk to backend (probably via calling /api/version, until OpenTSDB supports a better status API, see OpenTSDB/opentsdb #1742).

Also consider a HTTP endpoint that does similar for use with Kubernetes as a readiness probe.

README has not been updated with recent additions e.g. metrics, tracing

Geras doesn't actually look at the ErrorMsg response from opentsdb-goclient. However the fix for this is a bit tricky -- we now rely on not sending an error back when the metric name doesn't exist.

The errors we want to ignore look like {"error":{"code":400,"message":"No such name for 'metrics': 'metric.name.here'","details":"[same as message]","trace":"net.opentsdb.BadRequestException: ... Caused by net.opentsdb.uid.NoSuchUniqueName..."}}

The "net.opentsdb.uid.NosuchUniqueName" looks like the most useful way to filter, but probably just code == 400 is best rather than relying on implementation details.

x:y:z{foo=~"y"} returns foo="xyz" series. Same for __name__.

Against real Prometheus this does full matches (e.g. up{job=~"rometheus"} returns nothing).

Outgoing request body has to be dumped before request is sent, otherwise it's already read and shows empty

See https://github.com/G-Research/geras/tree/reflection.

Looks like https://jbrandhorst.com/post/gogoproto/ is the issue here :(.

The -grpc-listen argument professes to have a default:

  -grpc-listen string
    	service will expose the store api on this address (default "localhost:19000")

but if you don't pass a string, it complains:

$ go run cmd/geras/main.go -grpc-listen
flag needs an argument: -grpc-listen

Either the help message should change or there should be a default. Either way is fine -- probably having the default is a reasonable choice here.

It may be possible to do some aggregations in OpenTSDB to avoid sending the raw OpenTSDB data back. Not all the aggregations make sense, in particular the Thanos aggrchunks are about downsampling mostly, the labels don't seem to change.

Maybe useful to do #15 first so we can actually measure the impact.

i get the error like below:

there are my docker-compose.yml