Dynamic ZooKeeper Ensemble Support about rdf-delta HOT 10 CLOSED

Consul uses RAFT - but with direct use of RAFT, the patch coordination could be done as a RAFT state machine.

from rdf-delta.

There are two ways of using ZooKeeper with RDF Delta. One way is to run embedded ZooKeeper, the other is a separate ZooKeeper layer.

What looks interesting for version 2 is to switch to use RAFT, specifically Apache Ratis. This is a library and so will be like using embedded Zookeeper at the moment, removing the additional admin of a separate ZooKeeper cluster. It should also be possible to clean up the abstract making alternative plugins easier.

Consul can not be co-resident with the patch server. Surely you still have to identify the servers in the SWIM pool for the Consul/Serf gossip protocol to operate?

from rdf-delta.

Consul actually uses RAFT for consensus, so using it directly would eliminate the need for anything else. I approve of this direction.

It's not as important to me that ZooKeeper/Consul runs in the same process as RDF Delta as it is that ZooKeeper/Consul does not require an additional piece to get it to work in the first place. Consul does need to be pointed at the cluster, but it doesn't require complete knowledge of the entire cluster in each member. Basically, I can stand the first one up and point the second and third at the first without needing to provide connection info about every node. With ZooKeeper, I need them all to exist before I can generate a correct configuration file, but it's impossible to update the configuration after the instances have already been stood up without introducing an additional service of some kind or doing it by hand, thereby defeating the automation effort.

from rdf-delta.

I'm seeing now that ZooKeeper has a dynamic registration mode since version 3.5. I'm struggling to find where the embedded ZooKeeper is run in here, but I think we can just tweak it to register itself with a cluster to ease my problem in the near term. If you can help me find that bit of code, I'll see what I can do.

from rdf-delta.

ZooServer zs = ZkS.runZookeeperServer(...)

from rdf-delta.

I was able to get the embedded ZooKeeper server to dynamically join an ensemble with only the zoo.cfg and some ZooKeeper CLI commands. It will require a small change in the way the connection is established with Curator which I have documented here. I should have a pull request submitted for this change later today for your review. Thank you for your help, @afs.

from rdf-delta.

Ensemble tracking defaults to enabled for current ZooKeeper.

from rdf-delta.

https://issues.apache.org/jira/browse/ZOOKEEPER-3814 which is fixed with apache/zookeeper#1356 in ZooKeeper 3.7.0 is unrelated to this issue. The problem is not getting the members of the ZooKeeper Ensemble to dynamically update their config. That always worked. The problem is getting RDF Delta, a CLIENT of the ZooKeeper Ensemble, to dynamically update its config from /zookeeper/config. If you'd only actually test RDF Delta against a dynamically changing ZooKeeper Ensemble, it would be immediately obvious why I opened this ticket originally.

from rdf-delta.

CuratorFrameworkFactory.builder.ensembleTracker.

from rdf-delta.

You closed this citing an unrelated update to ZooKeeper, but now you're saying Curator does it. You haven't updated either of those dependencies, and I've reported to you that this, in fact, does not work. Reading the javadoc and making comments in the source code is no substitute for actually testing it, but you seem to be too stubborn to just do that. Oh, well. You can lead a horse to water, but you can't make him drink.

from rdf-delta.