mgodave / barge Goto Github PK

View Code? Open in Web Editor NEW

91.0 91.0 32.0 1.4 MB

A JVM Implementation of the Raft Consensus Protocol

License: Apache License 2.0

Java 62.11% Shell 0.37% CSS 35.58% Clojure 1.65% HTML 0.29%

barge's Introduction

Hi there 👋

barge's People

Contributors

Stargazers

Watchers

barge's Issues

Is this still being developed/bug fixed?

Hey, I'm thinking of creating a library based on barge, and just want to make sure its still being kept up, before I proceed.

Also, where is the snapshot repo? ClusterConfig isn't availble with 0.1.0-alpha1

Follower proxies client request instead of redirecting client

Unit tests not passing on Windows at 3cdae1a

I am trying to build barge using plain mvn clean install on Windows 7 and some unit tests are failing.

 $ mvh -v
 Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 14:51:28+0100)
 Maven home: d:\soft\apache-maven-3.0.5
 Java version: 1.7.0_45, vendor: Oracle Corporation
 Java home: d:\Program Files\Java\jdk1.7.0_45\jre
 Default locale: fr_FR, platform encoding: Cp1252
 OS name: "windows 7", version: "6.1", arch: "amd64", family: "windows"

Here is the output of tests execution:

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.robotninjas.barge.state.BaseStateTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.749 sec
Running org.robotninjas.barge.state.CandidateTest
21:55:35.079 [main] DEBUG o.robotninjas.barge.state.Candidate - Election starting for term 2
21:55:35.143 [main] DEBUG o.robotninjas.barge.state.Candidate - RequestVote received for term 2
21:55:35.172 [main] DEBUG o.robotninjas.barge.state.Candidate - Election starting for term 2
21:55:35.179 [main] DEBUG o.robotninjas.barge.state.Candidate - AppendEntries received for term 2
21:55:35.195 [main] DEBUG o.robotninjas.barge.state.Candidate - Election starting for term 2
21:55:35.198 [main] DEBUG o.robotninjas.barge.state.Candidate - RequestVote received for term 1
21:55:35.207 [main] DEBUG o.robotninjas.barge.state.Candidate - Election starting for term 2
21:55:35.209 [main] DEBUG o.robotninjas.barge.state.Candidate - RequestVote received for term 4
21:55:35.219 [main] DEBUG o.robotninjas.barge.state.Candidate - Election starting for term 2
21:55:35.221 [main] DEBUG o.robotninjas.barge.state.Candidate - AppendEntries received for term 4
21:55:35.232 [main] DEBUG o.robotninjas.barge.state.Candidate - Election starting for term 2
21:55:35.234 [main] DEBUG o.robotninjas.barge.state.Candidate - AppendEntries received for term 1
Tests run: 6, Failures: 6, Errors: 0, Skipped: 0, Time elapsed: 0.255 sec <<< FAILURE!
Running org.robotninjas.barge.state.DefaultContextTest
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec <<< FAILURE!
Running org.robotninjas.barge.state.MajorityCollectorTest
Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec
Running org.robotninjas.barge.state.ReplicaManagerTest
21:55:35.340 [main] DEBUG o.r.barge.state.ReplicaManager - Sending update to localhost:1000 prevLogIndex 0 prevLogTerm 0
21:55:35.459 [main] DEBUG o.r.barge.state.ReplicaManager - Sending update to localhost:1000 prevLogIndex 0 prevLogTerm 0
21:55:35.460 [main] DEBUG o.r.barge.state.ReplicaManager - Response from localhost:1000 Status false nextIndex 1, matchIndex 0
21:55:35.461 [main] DEBUG o.r.barge.state.ReplicaManager - Sending update to localhost:1000 prevLogIndex 0 prevLogTerm 0
21:55:35.462 [main] DEBUG o.r.barge.state.ReplicaManager - Response from localhost:1000 Status false nextIndex 1, matchIndex 0
21:55:35.468 [main] DEBUG o.r.barge.state.ReplicaManager - Sending update to localhost:1000 prevLogIndex 0 prevLogTerm 0
21:55:35.469 [main] DEBUG o.r.barge.state.ReplicaManager - Response from localhost:1000 Status true nextIndex 1, matchIndex 0
21:55:35.469 [main] DEBUG o.r.barge.state.ReplicaManager - Response from localhost:1000 Status true nextIndex 1, matchIndex 1
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.14 sec <<< FAILURE!
Running org.robotninjas.barge.state.DefaultContextTest
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec <<< FAILURE!
Running org.robotninjas.barge.state.MajorityCollectorTest
Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec
Running org.robotninjas.barge.state.ReplicaManagerTest
21:55:35.340 [main] DEBUG o.r.barge.state.ReplicaManager - Sending update to localhost:1000 prevLogIndex 0 prevLogTerm 0
21:55:35.459 [main] DEBUG o.r.barge.state.ReplicaManager - Sending update to localhost:1000 prevLogIndex 0 prevLogTerm 0
21:55:35.460 [main] DEBUG o.r.barge.state.ReplicaManager - Response from localhost:1000 Status false nextIndex 1, matchIndex 0
21:55:35.461 [main] DEBUG o.r.barge.state.ReplicaManager - Sending update to localhost:1000 prevLogIndex 0 prevLogTerm 0
21:55:35.462 [main] DEBUG o.r.barge.state.ReplicaManager - Response from localhost:1000 Status false nextIndex 1, matchIndex 0
21:55:35.468 [main] DEBUG o.r.barge.state.ReplicaManager - Sending update to localhost:1000 prevLogIndex 0 prevLogTerm 0
21:55:35.469 [main] DEBUG o.r.barge.state.ReplicaManager - Response from localhost:1000 Status true nextIndex 1, matchIndex 0
21:55:35.469 [main] DEBUG o.r.barge.state.ReplicaManager - Response from localhost:1000 Status true nextIndex 1, matchIndex 1
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.14 sec <<< FAILURE!

Results :

Failed tests:   testRequestVoteWithSameTerm(org.robotninjas.barge.state.CandidateTest):
  testAppendEntriesWithSameTerm(org.robotninjas.barge.state.CandidateTest):
  testRequestVoteWithOlderTerm(org.robotninjas.barge.state.CandidateTest):
  testRequestVoteWithNewerTerm(org.robotninjas.barge.state.CandidateTest):
  testAppendEntriesWithNewerTerm(org.robotninjas.barge.state.CandidateTest):
  testAppendEntriesWithOlderTerm(org.robotninjas.barge.state.CandidateTest):
  testDefaultContext(org.robotninjas.barge.state.DefaultContextTest):
  testSuccessfulAppend(org.robotninjas.barge.state.ReplicaManagerTest): expected:<2> but was:<1>

Tests run: 29, Failures: 8, Errors: 0, Skipped: 2

Am I missing something obvious?

Heartbeat messages causes followers to loop on same AppendEntry

While building an HTTP based demo of barge I noticed what I think might be a bug. Here are the (rough) steps to reproduce.

git pull foldlabs http_demo
mvn clean install
cd barge-jax-rs
java -jar target/barge-jax-rs.jar pack
# this creates a barge.sh script in current directory, not strictly necessary
# run several instances in different terminals
./barge.sh 0
./barge.sh 1
./barge.sh 2
# in a fourth terminal, init everything
./init.sh
./send http://localhost:56789 foo

In the terminal running the followers we can see logs for storing Entry in the journal:

11:59:15.395 [pool-1-thread-1] DEBUG org.robotninjas.barge.log.RaftLog - {http://localhost:56790/ 11 FOLLOWER} - http://localhost:56790/ storing Entry{command=[B@1c198fc5, term=11}
11:59:16.887 [pool-1-thread-1] DEBUG org.robotninjas.barge.log.RaftLog - {http://localhost:56790/ 11 FOLLOWER} - http://localhost:56790/ storing Entry{command=[B@2167179c, term=11}
11:59:18.393 [pool-1-thread-2] DEBUG org.robotninjas.barge.log.RaftLog - {http://localhost:56790/ 11 FOLLOWER} - http://localhost:56790/ storing Entry{command=[B@22a9014d, term=11}
11:59:19.897 [pool-1-thread-2] DEBUG org.robotninjas.barge.log.RaftLog - {http://localhost:56790/ 11 FOLLOWER} - http://localhost:56790/ storing Entry{command=[B@2ff4f32e, term=11}
11:59:21.391 [pool-1-thread-2] DEBUG org.robotninjas.barge.log.RaftLog - {http://localhost:56790/ 11 FOLLOWER} - http://localhost:56790/ storing Entry{command=[B@54074e95, term=11}
11:59:22.888 [pool-1-thread-2] DEBUG org.robotninjas.barge.log.RaftLog - {http://localhost:56790/ 11 FOLLOWER} - http://localhost:56790/ storing Entry{command=[B@302befad, term=11}
11:59:24.386 [pool-1-thread-2] DEBUG org.robotninjas.barge.log.RaftLog - {http://localhost:56790/ 11 FOLLOWER} - http://localhost:56790/ storing Entry{command=[B@2106c1fb, term=11}
11:59:25.888 [pool-1-thread-2] DEBUG org.robotninjas.barge.log.RaftLog - {http://localhost:56790/ 11 FOLLOWER} - http://localhost:56790/ storing Entry{command=[B@78bc5972, term=11}

In the leader:

12:00:22.854 [pool-1-thread-2] DEBUG org.robotninjas.barge.state.Leader - {http://localhost:56791/ 11 LEADER} - Sending heartbeat
12:00:22.855 [pool-1-thread-2] DEBUG o.r.barge.state.ReplicaManager - {http://localhost:56791/ 11 LEADER} - Sending update to http://localhost:56790/ prevLogIndex 2 prevLogTerm 9
12:00:22.863 [pool-1-thread-2] DEBUG o.r.barge.state.ReplicaManager - {http://localhost:56791/ 11 LEADER} - Sending update to http://localhost:56789/ prevLogIndex 2 prevLogTerm 9
12:00:22.893 [pool-1-thread-2] DEBUG o.r.barge.state.ReplicaManager - {http://localhost:56791/ 11 LEADER} - Response from http://localhost:56789/ Status true nextIndex 3, matchIndex 3
12:00:22.893 [pool-1-thread-2] DEBUG o.r.barge.state.ReplicaManager - {http://localhost:56791/ 11 LEADER} - Response from http://localhost:56789/ Status true nextIndex 3, matchIndex 3
12:00:22.893 [pool-1-thread-2] DEBUG org.robotninjas.barge.state.Leader - {http://localhost:56791/ 11 LEADER} - updating commitIndex to 3; sorted is [3, 3, 3]
12:00:22.898 [pool-1-thread-2] DEBUG o.r.barge.state.ReplicaManager - {http://localhost:56791/ 11 LEADER} - Response from http://localhost:56790/ Status true nextIndex 3, matchIndex 3
12:00:22.898 [pool-1-thread-2] DEBUG o.r.barge.state.ReplicaManager - {http://localhost:56791/ 11 LEADER} - Response from http://localhost:56790/ Status true nextIndex 3, matchIndex 3
12:00:22.898 [pool-1-thread-2] DEBUG org.robotninjas.barge.state.Leader - {http://localhost:56791/ 11 LEADER} - updating commitIndex to 3; sorted is [3, 3, 3]

I suspect the error lies in the way we update the leader's indices but I am still investigating and this is not clear to me (yet).

Snapshots not implemented

Replace proto compilation with protostuff-runtime

This is a follow-up to #44 (and #46).

In order to remove dependency from barge-core to protocol buffer, we could use protostuff in barge-proto-rpc to be able to expose only POJOs in the API and still use that POJOs as protocol buffer messages.

Should link to BargeD / add a quick usage doc

It would be great either to link to BargeD as a fully working example, or to include a quick code example back in the README file.

Allow a barge configuration to be loaded from file

Dynamic membership

Dynamic Membership

Client

The client driver is currently just something to smoke test barge, this needs to be refactored into a real client.

Candidate and Leader do not discover newer terms on RPC responses

Implement term skipping optimization

Implement term-skipping optimization mentioned in section 5.3 of the original paper. This will greatly speed up the ability of a new/long deceased replica to catch-up.

Reduce Threads

Each instance of Barge uses three threads by default, one for IO, one to run the protocol and one for completion of TCP connections. None but the IO thread are necessary. The Barge thread was added in order to allow the TCP connect to block and the socket completion thread was added in order to move the blocking connect from the protocol thread. As a simple first step the protocol thread can be removed and socket completion thread can be shared between instances of the protocol thus preserving the blocking connect semantics. Further, since the socket connect is a non-blocking operation artificially made blocking, we can further eliminate the socket completion thread by exposing the async nature of the TCP connect thus leaving the IO thread the run the protocol. We can then use a single pool of Netty IO threads to run several instances of the raft protocol, this is potentially hundreds (or thousands, although we run the very real risk of having to drastically increase the timeout and thus the responsiveness to failures)

Program does not shutdown cleanly

The barge library has several non-daemon threads which are not shutdown correctly and which prevent programs from exiting.

Leadership will not stabilize after reboot of cluster

Enable timeouts on a per call basis

Pipeline RPCs

RPC blocking connect slows down committal

When a replica is not accepting new connections (usually because it's crashed), each operation sent to the leader incurs a noticeable penalty before it is able to be committed. This is due to the fact that the pool which tries to create a new connection to the failed leader will block the RaftThread until it times out.

The replica does not read it's log on startup

Even though writes to the log are safe, the replica, upon startup, will not read and replay them.

Documentation

Client connections are prematurely closed

After exactly 8 calls to AppendEntries RPC from the leader, the connection is closed. This is true regardless of the timeout or heartbeat values.

Log Compaction

Design Proposal:

Leader detects that the nextEntry is not in it's log and sets 'contains_snapshot' field on next AppendEntries RPC request.
Follower reads the 'contains_snapshot' field and transitions to a Follower' state.
Follower initiates a pull of the snapshot form the Leader (over HTTP for a first cut, pluggable transport later)
- Follower' services AppendEntries RPC calls in a speculative manner, accepting all entries assuming the snapshot has been applied.
- No entries are committed until the snapshot is downloaded and applied
- After the snapshot is downloaded and applied Follower' applied the commits and transitions back to Follower

Provide a way to plug custom communication layer

Currently barge is tightly coupled with its communication layer based on protocol buffers and netty, with its own server port and protocol. In order to make barge integration easier in various settings, it would be great to decouple the core Raft protocol from the communication medium it uses.

Personally, I am aiming at integrating barge as part of a another piece of software which communicates throught HTTP REST-style interface so I would like to ensure the barge part also uses this transport mechanism.

Make sure leader can communicate with majority of cluster

step down otherwise.

Consolidate append and vote logic in BaseState

There is a lot of repeated code in the Leader, Candidate, and Follower states that can be consolidated into the BaseState.

Give client a list of replicas to try

Right now the client takes the first replica and tries to connect to that one. It will redirect to the leader if one is found. If the client's bootstrap node is down it will never detect the leader. Also, if the leader is found and subsequently dies, it will never try another node. We should be able to make the client smarter and allow it to try from a list of nodes until it gets a successful connection.

JMX hooks for monitoring and control

Incorrect usage of GenericKeyedObjectPool is causing a large memory leak

Client example?

Great library - thanks! Is there an example available of how to use this (as a client)? The example here still works, but I'm not not sure why it's been removed from the Readme file: a7e08b0

Also, is it possible to batch multiple appends into a commit? It looks like this might happen anyway if I call commit from multiple threads simultaneously; is that right? But then is it possible to have inter-dependencies between the append entries? (e.g. compare-and-swap X from 1 -> 2, then compare-and-swap X from 2 -> 3; there's no point trying to execute those two out of order.) Or is the right thing there to let these go through the append log, and then to check whether they can be applied to the state once they've been serialized into the log?

Sorry for so many questions!

mgodave / barge Goto Github PK

barge's Introduction

Hi there 👋

barge's People

Contributors

Stargazers

Watchers

Forkers

barge's Issues

Recommend Projects

Recommend Topics

Recommend Org