mgodave / barge Goto Github PK
View Code? Open in Web Editor NEWA JVM Implementation of the Raft Consensus Protocol
License: Apache License 2.0
A JVM Implementation of the Raft Consensus Protocol
License: Apache License 2.0
Hey, I'm thinking of creating a library based on barge, and just want to make sure its still being kept up, before I proceed.
Also, where is the snapshot repo? ClusterConfig isn't availble with 0.1.0-alpha1
I am trying to build barge using plain mvn clean install
on Windows 7 and some unit tests are failing.
$ mvh -v
Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 14:51:28+0100)
Maven home: d:\soft\apache-maven-3.0.5
Java version: 1.7.0_45, vendor: Oracle Corporation
Java home: d:\Program Files\Java\jdk1.7.0_45\jre
Default locale: fr_FR, platform encoding: Cp1252
OS name: "windows 7", version: "6.1", arch: "amd64", family: "windows"
Here is the output of tests execution:
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.robotninjas.barge.state.BaseStateTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.749 sec
Running org.robotninjas.barge.state.CandidateTest
21:55:35.079 [main] DEBUG o.robotninjas.barge.state.Candidate - Election starting for term 2
21:55:35.143 [main] DEBUG o.robotninjas.barge.state.Candidate - RequestVote received for term 2
21:55:35.172 [main] DEBUG o.robotninjas.barge.state.Candidate - Election starting for term 2
21:55:35.179 [main] DEBUG o.robotninjas.barge.state.Candidate - AppendEntries received for term 2
21:55:35.195 [main] DEBUG o.robotninjas.barge.state.Candidate - Election starting for term 2
21:55:35.198 [main] DEBUG o.robotninjas.barge.state.Candidate - RequestVote received for term 1
21:55:35.207 [main] DEBUG o.robotninjas.barge.state.Candidate - Election starting for term 2
21:55:35.209 [main] DEBUG o.robotninjas.barge.state.Candidate - RequestVote received for term 4
21:55:35.219 [main] DEBUG o.robotninjas.barge.state.Candidate - Election starting for term 2
21:55:35.221 [main] DEBUG o.robotninjas.barge.state.Candidate - AppendEntries received for term 4
21:55:35.232 [main] DEBUG o.robotninjas.barge.state.Candidate - Election starting for term 2
21:55:35.234 [main] DEBUG o.robotninjas.barge.state.Candidate - AppendEntries received for term 1
Tests run: 6, Failures: 6, Errors: 0, Skipped: 0, Time elapsed: 0.255 sec <<< FAILURE!
Running org.robotninjas.barge.state.DefaultContextTest
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec <<< FAILURE!
Running org.robotninjas.barge.state.MajorityCollectorTest
Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec
Running org.robotninjas.barge.state.ReplicaManagerTest
21:55:35.340 [main] DEBUG o.r.barge.state.ReplicaManager - Sending update to localhost:1000 prevLogIndex 0 prevLogTerm 0
21:55:35.459 [main] DEBUG o.r.barge.state.ReplicaManager - Sending update to localhost:1000 prevLogIndex 0 prevLogTerm 0
21:55:35.460 [main] DEBUG o.r.barge.state.ReplicaManager - Response from localhost:1000 Status false nextIndex 1, matchIndex 0
21:55:35.461 [main] DEBUG o.r.barge.state.ReplicaManager - Sending update to localhost:1000 prevLogIndex 0 prevLogTerm 0
21:55:35.462 [main] DEBUG o.r.barge.state.ReplicaManager - Response from localhost:1000 Status false nextIndex 1, matchIndex 0
21:55:35.468 [main] DEBUG o.r.barge.state.ReplicaManager - Sending update to localhost:1000 prevLogIndex 0 prevLogTerm 0
21:55:35.469 [main] DEBUG o.r.barge.state.ReplicaManager - Response from localhost:1000 Status true nextIndex 1, matchIndex 0
21:55:35.469 [main] DEBUG o.r.barge.state.ReplicaManager - Response from localhost:1000 Status true nextIndex 1, matchIndex 1
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.14 sec <<< FAILURE!
Running org.robotninjas.barge.state.DefaultContextTest
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec <<< FAILURE!
Running org.robotninjas.barge.state.MajorityCollectorTest
Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec
Running org.robotninjas.barge.state.ReplicaManagerTest
21:55:35.340 [main] DEBUG o.r.barge.state.ReplicaManager - Sending update to localhost:1000 prevLogIndex 0 prevLogTerm 0
21:55:35.459 [main] DEBUG o.r.barge.state.ReplicaManager - Sending update to localhost:1000 prevLogIndex 0 prevLogTerm 0
21:55:35.460 [main] DEBUG o.r.barge.state.ReplicaManager - Response from localhost:1000 Status false nextIndex 1, matchIndex 0
21:55:35.461 [main] DEBUG o.r.barge.state.ReplicaManager - Sending update to localhost:1000 prevLogIndex 0 prevLogTerm 0
21:55:35.462 [main] DEBUG o.r.barge.state.ReplicaManager - Response from localhost:1000 Status false nextIndex 1, matchIndex 0
21:55:35.468 [main] DEBUG o.r.barge.state.ReplicaManager - Sending update to localhost:1000 prevLogIndex 0 prevLogTerm 0
21:55:35.469 [main] DEBUG o.r.barge.state.ReplicaManager - Response from localhost:1000 Status true nextIndex 1, matchIndex 0
21:55:35.469 [main] DEBUG o.r.barge.state.ReplicaManager - Response from localhost:1000 Status true nextIndex 1, matchIndex 1
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.14 sec <<< FAILURE!
Results :
Failed tests: testRequestVoteWithSameTerm(org.robotninjas.barge.state.CandidateTest):
testAppendEntriesWithSameTerm(org.robotninjas.barge.state.CandidateTest):
testRequestVoteWithOlderTerm(org.robotninjas.barge.state.CandidateTest):
testRequestVoteWithNewerTerm(org.robotninjas.barge.state.CandidateTest):
testAppendEntriesWithNewerTerm(org.robotninjas.barge.state.CandidateTest):
testAppendEntriesWithOlderTerm(org.robotninjas.barge.state.CandidateTest):
testDefaultContext(org.robotninjas.barge.state.DefaultContextTest):
testSuccessfulAppend(org.robotninjas.barge.state.ReplicaManagerTest): expected:<2> but was:<1>
Tests run: 29, Failures: 8, Errors: 0, Skipped: 2
Am I missing something obvious?
While building an HTTP based demo of barge I noticed what I think might be a bug. Here are the (rough) steps to reproduce.
git pull foldlabs http_demo
mvn clean install
cd barge-jax-rs
java -jar target/barge-jax-rs.jar pack
# this creates a barge.sh script in current directory, not strictly necessary
# run several instances in different terminals
./barge.sh 0
./barge.sh 1
./barge.sh 2
# in a fourth terminal, init everything
./init.sh
./send http://localhost:56789 foo
In the terminal running the followers we can see logs for storing Entry in the journal:
11:59:15.395 [pool-1-thread-1] DEBUG org.robotninjas.barge.log.RaftLog - {http://localhost:56790/ 11 FOLLOWER} - http://localhost:56790/ storing Entry{command=[B@1c198fc5, term=11}
11:59:16.887 [pool-1-thread-1] DEBUG org.robotninjas.barge.log.RaftLog - {http://localhost:56790/ 11 FOLLOWER} - http://localhost:56790/ storing Entry{command=[B@2167179c, term=11}
11:59:18.393 [pool-1-thread-2] DEBUG org.robotninjas.barge.log.RaftLog - {http://localhost:56790/ 11 FOLLOWER} - http://localhost:56790/ storing Entry{command=[B@22a9014d, term=11}
11:59:19.897 [pool-1-thread-2] DEBUG org.robotninjas.barge.log.RaftLog - {http://localhost:56790/ 11 FOLLOWER} - http://localhost:56790/ storing Entry{command=[B@2ff4f32e, term=11}
11:59:21.391 [pool-1-thread-2] DEBUG org.robotninjas.barge.log.RaftLog - {http://localhost:56790/ 11 FOLLOWER} - http://localhost:56790/ storing Entry{command=[B@54074e95, term=11}
11:59:22.888 [pool-1-thread-2] DEBUG org.robotninjas.barge.log.RaftLog - {http://localhost:56790/ 11 FOLLOWER} - http://localhost:56790/ storing Entry{command=[B@302befad, term=11}
11:59:24.386 [pool-1-thread-2] DEBUG org.robotninjas.barge.log.RaftLog - {http://localhost:56790/ 11 FOLLOWER} - http://localhost:56790/ storing Entry{command=[B@2106c1fb, term=11}
11:59:25.888 [pool-1-thread-2] DEBUG org.robotninjas.barge.log.RaftLog - {http://localhost:56790/ 11 FOLLOWER} - http://localhost:56790/ storing Entry{command=[B@78bc5972, term=11}
In the leader:
12:00:22.854 [pool-1-thread-2] DEBUG org.robotninjas.barge.state.Leader - {http://localhost:56791/ 11 LEADER} - Sending heartbeat
12:00:22.855 [pool-1-thread-2] DEBUG o.r.barge.state.ReplicaManager - {http://localhost:56791/ 11 LEADER} - Sending update to http://localhost:56790/ prevLogIndex 2 prevLogTerm 9
12:00:22.863 [pool-1-thread-2] DEBUG o.r.barge.state.ReplicaManager - {http://localhost:56791/ 11 LEADER} - Sending update to http://localhost:56789/ prevLogIndex 2 prevLogTerm 9
12:00:22.893 [pool-1-thread-2] DEBUG o.r.barge.state.ReplicaManager - {http://localhost:56791/ 11 LEADER} - Response from http://localhost:56789/ Status true nextIndex 3, matchIndex 3
12:00:22.893 [pool-1-thread-2] DEBUG o.r.barge.state.ReplicaManager - {http://localhost:56791/ 11 LEADER} - Response from http://localhost:56789/ Status true nextIndex 3, matchIndex 3
12:00:22.893 [pool-1-thread-2] DEBUG org.robotninjas.barge.state.Leader - {http://localhost:56791/ 11 LEADER} - updating commitIndex to 3; sorted is [3, 3, 3]
12:00:22.898 [pool-1-thread-2] DEBUG o.r.barge.state.ReplicaManager - {http://localhost:56791/ 11 LEADER} - Response from http://localhost:56790/ Status true nextIndex 3, matchIndex 3
12:00:22.898 [pool-1-thread-2] DEBUG o.r.barge.state.ReplicaManager - {http://localhost:56791/ 11 LEADER} - Response from http://localhost:56790/ Status true nextIndex 3, matchIndex 3
12:00:22.898 [pool-1-thread-2] DEBUG org.robotninjas.barge.state.Leader - {http://localhost:56791/ 11 LEADER} - updating commitIndex to 3; sorted is [3, 3, 3]
I suspect the error lies in the way we update the leader's indices but I am still investigating and this is not clear to me (yet).
In order to remove dependency from barge-core to protocol buffer, we could use protostuff in barge-proto-rpc to be able to expose only POJOs in the API and still use that POJOs as protocol buffer messages.
It would be great either to link to BargeD as a fully working example, or to include a quick code example back in the README file.
The client driver is currently just something to smoke test barge, this needs to be refactored into a real client.
Implement term-skipping optimization mentioned in section 5.3 of the original paper. This will greatly speed up the ability of a new/long deceased replica to catch-up.
Each instance of Barge uses three threads by default, one for IO, one to run the protocol and one for completion of TCP connections. None but the IO thread are necessary. The Barge thread was added in order to allow the TCP connect to block and the socket completion thread was added in order to move the blocking connect from the protocol thread. As a simple first step the protocol thread can be removed and socket completion thread can be shared between instances of the protocol thus preserving the blocking connect semantics. Further, since the socket connect is a non-blocking operation artificially made blocking, we can further eliminate the socket completion thread by exposing the async nature of the TCP connect thus leaving the IO thread the run the protocol. We can then use a single pool of Netty IO threads to run several instances of the raft protocol, this is potentially hundreds (or thousands, although we run the very real risk of having to drastically increase the timeout and thus the responsiveness to failures)
The barge library has several non-daemon threads which are not shutdown correctly and which prevent programs from exiting.
When a replica is not accepting new connections (usually because it's crashed), each operation sent to the leader incurs a noticeable penalty before it is able to be committed. This is due to the fact that the pool which tries to create a new connection to the failed leader will block the RaftThread until it times out.
Even though writes to the log are safe, the replica, upon startup, will not read and replay them.
After exactly 8 calls to AppendEntries RPC from the leader, the connection is closed. This is true regardless of the timeout or heartbeat values.
Design Proposal:
Currently barge is tightly coupled with its communication layer based on protocol buffers and netty, with its own server port and protocol. In order to make barge integration easier in various settings, it would be great to decouple the core Raft protocol from the communication medium it uses.
Personally, I am aiming at integrating barge as part of a another piece of software which communicates throught HTTP REST-style interface so I would like to ensure the barge part also uses this transport mechanism.
step down otherwise.
There is a lot of repeated code in the Leader, Candidate, and Follower states that can be consolidated into the BaseState.
Right now the client takes the first replica and tries to connect to that one. It will redirect to the leader if one is found. If the client's bootstrap node is down it will never detect the leader. Also, if the leader is found and subsequently dies, it will never try another node. We should be able to make the client smarter and allow it to try from a list of nodes until it gets a successful connection.
Great library - thanks! Is there an example available of how to use this (as a client)? The example here still works, but I'm not not sure why it's been removed from the Readme file: a7e08b0
Also, is it possible to batch multiple appends into a commit? It looks like this might happen anyway if I call commit from multiple threads simultaneously; is that right? But then is it possible to have inter-dependencies between the append entries? (e.g. compare-and-swap X from 1 -> 2, then compare-and-swap X from 2 -> 3; there's no point trying to execute those two out of order.) Or is the right thing there to let these go through the append log, and then to check whether they can be applied to the state once they've been serialized into the log?
Sorry for so many questions!
The replicas do not keep track of a replica's largest seen sequence per clientId
The log is loaded before the state machine interface is set. We need to make sure the state machine is set before the lognis loaded.
The RaftLog entries cache is only really useful for leaders. Once a follower plays an entry to it's state machine it no longer needs to keep it cached in memory.
After running a cluster, stopping all nodes, and restarting them, an election will not complete.
I found that indentation and layout of code is not consistent. Would be good to define conventions and add files for IDEA and Eclipse.
It tries to run the gradle build instead of the maven build.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.