Comments (14)
Yea, I think adding DNS-SRV would actually be still reasonable if there's a strong use case for it. It would just be a few lines rather than pulling in the massive Prometheus SD framework and all its deps.
This is only helpful for initial discovery of peers though, much like in Prometheus Alertmanager. Arguably we are already providing a better experience for that than AM did in the past. Generally, one can always startup with a small script that pulls initial peer info before starting the Thanos component
from thanos.
I felt like it would be good to specify the federation approach in a bit more detail to make a decision here: https://docs.google.com/document/d/1-hXTQ3dSFA1yNiUrWCMFqkW84k6PicZK-9tMhTssKN0/edit
from thanos.
As discussed there is also:
B2) Thanos clusters are still regional or even more fine-grained. On top of local querier there is additional global query layer across smaller all Thanos clusters.
Local query nodes are NOT aware of other clusters or global node. It only exposes additional the Store API. This model is closer to hierarchical Prometheus federation
Pros:
- Same as B1 + even more cross-cluster communication reduction
- No need for client LB ("which query should Global Grafana to call?" in option B2)
- Logically Simpler. Local queriers are regionally sharded and isolated from each other.
Cons:
- Same as B1, but also:
- Additional component (global query) adds complexity.
- We need HA for global query to avoid SPOF again -> even more complexity and maintenance.
- We need special ruler with global rules
from thanos.
I was asked to give some thoughts here, sorry for the delay! To me it seems both easiest and cleanest to have all nodes join the same gossip network (option A), without any sense of hierarchy or federation. Scoping queries to e.g. sites or regions is, to my mind, a query-time operation, and so can be like any other decision made by the query handlers, based on per-node metadata it's already received and cached via gossip.
To me, a federation/hierarchy feels like a sort of performance optimization. Without concrete justification that it's necessary, I'd try to avoid the complexity it would introduce.
This is based on several assumptions:
- That Thanos, like OK Log, uses gossip purely for membership and basic node metadata
- That it's possible and reliable to configure e.g. memberlist to work at this scale
- That users will at least sometimes want to query Thanos at global (complete) scale
- Operational details like firewalled port ranges etc. aren't problematic enough to optimize for
from thanos.
Thank you for you input @peterbourgon! Totally agree, that with these assumptions, option A is perfect. However, at this point we are looking on federation because we are not really sure if below two assumptions you mentioned are always true (for all potential users of the Thanos):
- That it's possible and reliable to configure e.g. memberlist to work at this scale
- Operational details like firewalled port ranges etc. aren't problematic enough to optimize for
Especially when we think of the clusters from totally different geographical regions (like EU vs Asia) or some proxy-based cross-cluster communication like istio or kedge where cross-cluster communication requires bit more configuration (and RTT time).
from thanos.
There's maybe an option (C); use the Prometheus service discovery library to discover peers instead of using the memberlist library.
Pros:
- shifts all the burden of membership to the service discovery mechanism used (tuning the gossip protocol for latency would no longer be a concern for this project)
- users can use whatever mechanism suits them best
Cons:
- makes the configuration more complex (e.g. relabelling)
- introduces a lot of dependencies (but they would be mostly managed upstream)
- you'd lose the dead node detection provided by memberlist (this problem would be pushed to the service discovery mechanism chosen)
from thanos.
Thanks @mattbostock. We have considered using Prometheus's SD package. But you are pretty much spot-on with the con's you listed.
Prometheus's SD is only really useful if there's meaningful metadata you need to extract from your service discovery information (e.g. building target labels in Prometheus).
In Thanos we do have labels for Prometheus instances – but those are directly extracted from their external_labels
config section. Arguably moving this critical information to a loosely connected SD would be asking for trouble for lots of users in the end.
For basic discovery DNS is be mostly fine and in practice provided on top of any more sophisticated discovery mechanism anyway. DNS is basically allowed now through static flag-based configuration of additional data sources in the querier.
Right now Thanos is dead simple to configure, which is largely thanks to a lack of config files. Adding those with relabeling rules would change that immediately.
from thanos.
@fabxc: I agree, I think using the Prometheus service discovery library would add unnecessary complexity.
Echoing @peterbourgon's comment above, I think we should avoid an additional 'federation' layer in the interest of keeping things simple.
Store instances can be configured statically, which I think resolves most of this issue? The deployment model I'm thinking of is:
- sidecar instances implementing the store API are deployed to datacentres where the data is being generated (where Prometheus is)
- query instances (and S3/GCS store instances) are deployed closest to where users consume the data (e.g. where Grafana runs)
In this scenario, the store instances can be configured 'statically' (e.g. using confd) as part of the command-line options for the query nodes. Additionally, we should support cross-WAN cluster communication. I suggest opening a separate issue to track that.
However, at this point we are looking on federation because we are not really sure if below two assumptions you mentioned are always true (for all potential users of the Thanos)
We can always add this later when/if the use case arises.
from thanos.
@fabxc The only reason I'd be pro re-using the prometheus SD library is so that we get things like DNS-SD discovery (i.e. SRV records) for free -- it's my understanding that the existing binary can only do A record queries ATM for DNS-based SD?
from thanos.
Yea, broadly speaking, setup we ended up is similar to what you said @mattbostock
Each environment:
- have scrapers in each cluster (Prometheus + thanos sidecar) in 2-replicas HA
- have queries, compactor, store node, one scraper, ruler in "monitoring" cluster. The same where alertmanager & dashboards are.
Monitoring cluster have thanos components connected via gossip. Queriers on this cluster have statically configured scrapers from remote clusters that are connected through the proxy (kEdge), since there is no other connection (vpn) between them. That's why I said there are some cases where gossip is not possible to use. This configuration is fine for now, because we don't really need "automated cluster discovery", so no SD needed.
Additionally, we should support cross-WAN cluster communication. I suggest opening a separate issue to track that.
What do you mean by that? What exactly would you like to have for that? I have cross-WAN by using some external proxy service, so no Thanos change was required.
However, we do want some federated global layer of queries to be on top of all environments and allow global view across envs. This can be done using static --stores
query flag + my proxy in my case, and seems to be good enough for now.
from thanos.
@swsnider for peers, gossip flow needs an initial list of members. Either raw IP:port or domain:port. In case of latter the DNS lookup for all IP is done: https://github.com/improbable-eng/thanos/blob/master/pkg/cluster/cluster.go#L355
We had SRV lookup there as well, but we found it too complicated for an actual value. This proven to be sufficient for all use cases we had.
from thanos.
Additionally, we should support cross-WAN cluster communication. I suggest opening a separate issue to track that.
What do you mean by that? What exactly would you like to have for that?
@Bplotka: By cross-WAN cluster communication, I mean the ability for cluster peers to communicate and discover each other across a WAN, using appropriate timeouts such as:
https://godoc.org/github.com/hashicorp/memberlist#DefaultWANConfig
This would be an alternative to using the static --store
flag when your stores are located on the otherwise of a WAN. I don't yet have thoughts on how that would be implemented.
However, we do want some federated global layer of queries to be on top of all environments and allow global view across envs. This can be done using static --stores query flag + my proxy in my case, and seems to be good enough for now.
By 'we' do you mean the Thanos project, or Improbable?
from thanos.
@Bplotka: By cross-WAN cluster communication, I mean the ability for cluster peers to communicate and discover each other across a WAN, using appropriate timeouts such as (...)
Ah. Yea, I don't see any problem with changing Thanos to allow setting WAN defaults for gossip if you wish to setup WAN gossip, good point.
By 'we' do you mean the Thanos project, or Improbable?
Sorry for confusion (: I meant only Improbable use case here.
from thanos.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
from thanos.
Related Issues (20)
- tools bucket: Add ability to discover external labels from prometheus address for `upload-blocks` HOT 1
- Thanos Sidecar - Flush Endpoint HOT 9
- Grafana only shows raw data from Thanos HOT 3
- Instance Principal Provider - Region issue
- Consider X-Forwarded-For on HTTP/GRPC Logging
- When I restart any Receive, the entire Seek cluster is unavailable, and the reboot can only be restored after the local data is fully understood HOT 2
- MaxTime is set to a too large number when doing larger latency requests? HOT 1
- Unauthorized errors for some endpoints with query-frontend HOT 1
- External labels not applied to alerts HOT 3
- 0.35: Panic with query mode distributed HOT 1
- query: Passing `THANOS-TENANT: <tenant>` header has no effect unless `--query.enforce-tenancy` is set HOT 3
- query: different results for rate function when not dedup or using implicit step interval HOT 6
- Thanos compactor causing huge memory spikes when compacting raw blocks HOT 2
- Ruler evaluation warning false alarm caused by engine warnings
- Expose when Rule alert labels are being overwritten by its --label(s)
- Read value of remote_user in Slow Query Logs of Query Frontend from a HTTP header HOT 3
- Thanos Receive doesn't announce external_labels which are set in hashrings.json when it works in routing and ingesting mode. HOT 1
- Issue with deduplication alogrithm in Thanos
- Query Stats Returned with query including query bytes fetched HOT 4
- Max and min pointed at Sidecars not working on 0.35 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from thanos.