No notification is triggered if unable to retrieve health check status about consul-alerts HOT 2 OPEN

Hmm.. yeah. Maybe we need to add a notification when consul-alerts can't access consul? Consul-alerts tries to retry connecting to consul and has a backoff so it doesn't try it all the time. This needs some revision since it might be waiting for too long even though consul is already up.

from consul-alerts.

I think the backoff in principle is fine, but there ought to be a (relatively low) ceiling on it so it doesn't keep increasing - the benefit of a backoff is primarily to prevent it from over-loading an already degraded system, but if e.g. one check a minute is overloading it, it's already too far gone to make much difference in my opinion.

Also I for one consider failure to get health checks as one of the most critical things to get alerts for in a monitoring system - if Consul fails in case I'm "flying blind". While in theory another consul-alerts instance would take over (and so you can sort-of work around it by having suitable health checks for the other Consul agents defined on each Consul agent, this is one thing I'm paranoid about...

E.g. consider a case where the problem affects all Consul nodes equally in short succession (e.g. a broken update causes the same situation to trigger failures in all of them). Might sound far fetched, but I've had it happen.

from consul-alerts.