The chickens and the eggs about bedrock HOT 8 CLOSED

Ah, this is not supported for a reason: each node needs to know how many other nodes are in the cluster in order to know when half of the cluster is connected for quorum purposes. If you allow dynamic node additions, it opens up the possibility of the cluster "forking" into two halves that each think they have more than half (and thus, each thinking it has quorum). This is the "split brain" scenario that is about the worst possible nightmare to avoid.

Hmmm I don't think we've ever tried that. I imagine I could be done with enough care and restarts, but it's not something we've designed for. We run with 6 nodes and just upgrade in place as needed, so we haven't really needed zero-downtime cluster reconfiguration, but it's an interesting idea! Are you solving some specific problem or just sorta kicking the tires?

Hm, well Bedrock is very explicitly designed for a WAN-replicated environment across different datacenters, so that sounds like it could be good. However, it's not really designed to be moved around easily. In our experience, it works best to just configure it to work in as many datacenters as you have, and then balance all traffic equally across them all (or use anycast to route users to the nearest datacenter). If a datacenter fails for any reason, rather than frantically spinning up a new datacenter, just allow the traffic to automatically route to the others that are already in place. However, it sounds like you might have slightly different requirements: is there a reason that you prefer to move instances frequently and in a hurry between different datacenters? How big are the databases you are dealing with? In our case it takes many hours to download a backup and such, so there isn't really any way to spin up anything fast, hence the pre-positioning of adequate resources to weather the storm. We think three datacenters is the magic number, but that also means that we're 50% over-provisioned for peak. I can imagine running that closer to the edge to optimize cost, but that's not a scenario we've really optimized for.