cross dc services and failover about nerve HOT 4 CLOSED

@yagnik Yea we do this at Yelp with cross registration (nerve registers the service in multiple places). Airbnb went the filtering route. See #81 for context.

My reply on #81 starting with "I apologize ahead of time for the book of a reply here" has a breakdown of how you can do cross-dc failover using HAProxy ACLs and cross registration. If you're only running one ZK cluster you can use the filters that were added as part of airbnb/synapse#164

from nerve.

I'm unsure how I missed labels!!
We are doing exactly what you suggested just needed the glue to patch it all together. This is excellent.
Thanks!

from nerve.

btw @jolynch do you guys use multiple zk clusters one per az ?
Network flapping has caused me much grieve when running cross dc cluster.

from nerve.

@yagnik we run ZKs across AZs and have not had too many issues, but we've also tuned the bejeezers out of our ZK instances and are careful with Nerve restarts (which cause a lot of zk churn). I recall us having reliability improvements by moving to a big enough instance to have SSDs and the one step up networking (mostly m3.larges, with the occasional m3.xlarge for the "High" networking).

from nerve.