multi-data center configuration about consul-alerts HOT 4 CLOSED

This is pretty useful. This also opens up for node specific config if it's not overkill. Thinking about the implementation though, there's plenty of ways to go about this.

My initial though is:

consul-alerts/config/... would be the global configuration and fall-through if no dc or node specific config is available.
`consul-alerts/config/{{dc}}/... would be a datacenter specific config.

then something like consul-alerts/config/{{dc}}/{{node}}/... for node specific config, though not sure if it's something useful.

Another thought we have is something like this:

consul-alerts/{{dc}}/
  |-- config/...
  |-- checks/...

_ can be used as a wildcard to denote that it's doesn't care which dc or node it is. (eg. consul-alerts/_/... is the global config, and consul-alerts/dc1/... would be the dc1 specific config).

Having separate config and checks would avoid conflict when storing the health checks in the KV too.

Or maybe there's a better way of structuring the config.

What do you guys think? @dpetzel @hunter @aisrael

from consul-alerts.

While looking to see what structure Consul itself uses for some stuff, I found this:

It is important to note that each datacenter has its own K/V store, and that there is no replication between datacenters.By default the datacenter of the agent is queried, however the dc can be provided using the "?dc=" query parameter

This actually greatly reduces my use-case as I can rely on the default behavior. For some reason I was previously under the assumption the K/V store was Global

Personally I like the consul-alerts/config/{{dc}}/{{node}}/... structure, but don't care for needing to put a _ in the URL. What if the logic was to check each of the following (in order), with first match winning:

consul-alerts/config/{{dc}}/{{node}}/...
consul-alerts/config/{{dc}}/...
consul-alerts/config/...

Using the same pattern for checks

from consul-alerts.

I'd forgotten about each DC having it's own KV store but I wonder if this will help at all: https://github.com/hashicorp/consul-replicate

from consul-alerts.

I kinda like the default behaviour. It also means that consul-alerts will behave on the config based on which agent and datacenter it binds to.

Though there might be a case where someone wants to have the same config across datacenters. consul-replicate would come in pretty useful.

Maybe let's keep it simple and rely on the default behaviour of the KV Store. Just need to update the docs for people to be aware of it and consul-replicate. :)

from consul-alerts.