Congrats on the idea. It's much more easier to understand a concept when you are able to see it.
As a suggestion, i'd love to see implemented Gossip protocol and PBFT consensus protocol.

Greetings

Overview

Status: Proposal

Apache Kafka is a scalable, distributed publish-subscribe messaging system. It allows organizations to use it as an event bus and decouple systems from one another.

Sections

• Basics of Apache Kafka
• Replication
• Internals / Zookeeper interaction

https://tendermint.com/

Described here:
https://tendermint.readthedocs.io/en/master/specification/byzantine-consensus-algorithm.html

Respected author,
this project's raft introduce give me great help, is there any way we can restart this project again? or there is anybody who cares this repo we can do this together

Overview

The Raft distributed consensus protocol allows a collection of processes to maintain consistency even in the face of multiple node failure. The two main tenants of the protocol are leader election and log replication.

This visualization will lay out the problem of distributed consensus followed by a general overview of leader election and log replication. It will then follow up with details of Leader Election using best case (Single Candidate) and worst case (Split Vote) scenarios. Then it will show details of Log Replication using the best case (Network OK) and worst case (Network Partitions) scenarios. Finally, it will conclude with additional resources on where to learn more.

URL

http://thesecretlivesofdata.com/raft/

Frames

- [x] What is Distributed Consensus?
- [x] Overview
  - [x] States (Follower, Candidate, Leader)
- [x] Leader Election
  - [x] Election Timeout
  - [x] Candidacy
  - [x] Leadership & heartbeat timeout.
  - [x] Re-election
  - [x] Split Vote
- [ ] Log Replication
  - [X] Complex state machine example.
  - [X] Commitment rules
  - [X] Network Partitions
  - [X] Client reads.
- [X] Conclusion & Additional Resources

Follower's term is only incremented after it receives a heartbeat.

Here is the relevant section from the spec.

\* Server i receives a RequestVote request from server j with
\* m.mterm <= currentTerm[i].
HandleRequestVoteRequest(i, j, m) ==
    LET logOk == \/ m.mlastLogTerm > LastTerm(log[i])
                 \/ /\ m.mlastLogTerm = LastTerm(log[i])
                    /\ m.mlastLogIndex >= Len(log[i])
        grant == /\ m.mterm = currentTerm[i]
                 /\ logOk
                 /\ votedFor[i] \in {Nil, j}
    IN /\ m.mterm <= currentTerm[i]
       /\ \/ grant  /\ votedFor' = [votedFor EXCEPT ![i] = j]
          \/ ~grant /\ UNCHANGED votedFor
       /\ Reply([mtype        |-> RequestVoteResponse,
                 mterm        |-> currentTerm[i],
                 mvoteGranted |-> grant,
                 \* mlog is used just for the `elections' history variable for
                 \* the proof. It would not exist in a real implementation.
                 mlog         |-> log[i],
                 msource      |-> i,
                 mdest        |-> j],
                 m)
       /\ UNCHANGED <<state, currentTerm, candidateVars, leaderVars, logVars>>

The reply contains original term of the follower.

Round Robin scheduling except it's dynamic

This was fascinating. Such great work. Please do more.

Hi, wanted to say that I love the animations here for RAFT and it helped me understand it better when implementing it for my Distirbuted Systems course. Wanted to ask if there are any plans to make animations for perhaps Paxos or PBFT and other implementations of distributed systesm

Please add Paxos!

Hi Ben
Let me start off by thanking you for the amazing Raft visualiser - I watched it after your Go-Time podcast, and it has really helped me so much now that I started reading the paper
Thanks to your work it was really easy for me to mentally visualise while reading the paper

One callout that I have - I noticed that the nodes seem to experience election timeout at fixed (individual) intervals. Was that a conscious choice?

I'm not sure if the paper specifies that behaviour, and it might help with lowering the probability of candidacy collisions even further by choosing a random duration after every timeout

Just wanted to run this by you in case you agree

Thanks!

What is "majority" exactly? More detailed questions:

1. If the network is split, will be "majority" be recalculed for each partition separately?

2. What if node A connects to nodes B and C, but node B doesn't connect to C? What is "majority" in such a case?

Request for BlockChain Visualizations.

Thanks.

First, thank you for the presentation, extremely easy to understand. I have a few questions regarding this protocol:

1. How long is the election term ? You talk about the election timout 150~300ms to become a candidate, but I missed the term timeframe info.
2. How do you handle attackers ? If nodes can decide themselves to become candidate, couldn't one bypass the election timeout and send out request vote messages instantly ?
3. How do you insure that the leader is fair ? Even if bypassing the election timeout isn't possible, what if I wait to become a leader then start sending whatever message I want to send (ie in a blockchain, let everyone know that some follower owns me all his coins) ?

This kind of central authority, even if for a brief term, is a bit worrying ^^' Wouldn't a number of cluster centers sharing the authority (and elected more carefully with some trust weight from past operations) be more robust ? [/me go study about the federated consensus]

Add support for the right arrow key instead of requiring the user to constantly hit the "Continue" button.

/cc: @philips

This'd probably be a nice visualization, something showing how hot paths get optimized in JIT compilation and all that.

Of course, I'm not really a JIT expert, so I don't know what would be considered notable enough to visualize.

Paxos visualization
will to help code :)

nc

how about consistency?

would service orchestration make sense? Things like etcd and Serf seem like they are going to be important in managing multi node clusters.

use python to draw the print may be a good choice,by the way, is there any other project like this?

the view is very wonderful. maybe it can be a complete organization to help people understand the complex algorithm

Isn't the leader elected by majority of all nodes? If so, why during the presentation the network partition allows to create two leaders in disjointed partitions.

WebRTC consists of P2P + an unspecified initial communication channel that is often a central server. An explanation of WebRTC and ICE/STUN would be helpful since so many sites are using datachannels and/or WebRTC.

Would be nice to have a presentation for kubernetes

PBFT is popular, if possible, please make an introduction of PBFT.

THANKS A LOT!!!!
I HOPE TO LEARN MORE!!!!
IT'S VERY USEFUL!!!!

Hi Ben. I am preparing teaching materials as part of Samsung's effort to provide free to use education material. I came across your most excellent visualization tool and hope to use it as part of my curriculum. Unfortunatley, Samsung's
policy prevents me from simply including links. This is because many of the countries where the educatiion material will
be provided often have very poor or no internet connections. So, in order to use your visualization, I created a small video
of the visualization. (Screen recording with me pushing the buttons).

Please let me know if I may use your materials in this manner, and if so, what type of copyright disclosure you would like me to present in the materials.

Thank you.

Henry Park

If I look at https://github.com/benbjohnson/thesecretlivesofdata/blob/gh-pages/raft/Makefile I see that it uses paths such as " ../node_modules/.bin/mocha-phantomjs" which are not in the source distro. Not being familiar with the testing frameworks used can something be documented about the such tests (e.g. how to run them)?

Really awesome visualizations with Raft. Great work. I'd like to request if you can add Nakamoto Consensus algorithm used in Bitcoin invented by Satoshi Nakamoto in 2008 to solve the Byzantine General's Problem. It allows consensus between a network of nodes in a completely decentralized and trustless way.

It's truly awesome and I would really appreciate if you can make a visualization for it.

Thanks. Keep going.

Not an issue, just a quick note to say that this is freakin' awesome. The raft protocol explanation was fantastic. It's like the Khan Academy meets data. Clearly explains the protocol in a few short minutes in super simple terms. Thank you kindly for your inspiring work in this area. You rock.

I find Gossip Protocols have very little information about them. The information is typically within implementations such as Riak and Cassandra, for example. Even though Gossip Protocols are extremely powerful once understood, even though certain trade-offs are made (higher-availability while sacrificing consistency).

Although, Gossip Protocols vary quite a bit, so it might be hard to display the gossip protocol -- like Raft; you'd have to show a variation of sort.

Add a dropdown menu to jump to sections of the current visualization.

a decent animation is here: https://medium.com/@rhome/automatic-differentiation-26d5a993692b, forward and backwards propagation could be improved and made more cohesive "story"

I'm not sure that Paxos lends itself to simple conceptual examples, but if it can be done, an overview to Paxos allowing people to compare/contrast with Raft would be very interesting.

Hi!

On the last example, when a partition occurs, the current leader was elected on term 2. The new leader should be elected on term 3. However, it seems to be elected on term 2.

I developed a good understanding about raft after going through the visualization. Can we have one for Gossip protocol?
Please let me know if I can contribute here.

In few cases there is support for going back to revisit last page.

I think it would add educational value if there wan an option to go backwards step-by-step.
To the left of the continue button, perhaps.

If I go to "http://thesecretlivesofdata.com/raft/test/" I get test errors when using Chrome. If I checkout the latest code at the github page and run a phython local server I get the same set of errors:

Node

initialize()

should initialize as a follower ‣
should initialize with model ‣

bbox()

should return a bbox around the circle ‣

execute()

should append a log entry ‣
TypeError: Cannot read property 'invalidate' of null
at Node.onStateChange (http://thesecretlivesofdata.com/raft/scripts/model/node.js:703:30)
at Node.EventDispatcher.dispatchEvent (http://thesecretlivesofdata.com/scripts/playback/playback.js:776:26)
at Node.dispatchEvent (http://thesecretlivesofdata.com/raft/scripts/model/node.js:739:53)
at Node.EventDispatcher.dispatchChangeEvent (http://thesecretlivesofdata.com/scripts/playback/playback.js:791:17)
at Node.state (http://thesecretlivesofdata.com/raft/scripts/model/node.js:280:14)
at Node.currentTerm (http://thesecretlivesofdata.com/raft/scripts/model/node.js:243:18)
at Context. (http://thesecretlivesofdata.com/raft/test/model/node.js:39:22)
at Test.Runnable.run (http://thesecretlivesofdata.com/scripts/mocha/mocha-1.14.0.js:4263:32)
at Runner.runTest (http://thesecretlivesofdata.com/scripts/mocha/mocha-1.14.0.js:4635:10)
at http://thesecretlivesofdata.com/scripts/mocha/mocha-1.14.0.js:4681:12

nextIndex()

should default to 1 ‣
should set and return value ‣

matchIndex()

"before each" hook ‣
TypeError: Cannot read property 'invalidate' of null
at Node.onLeaderIdChange (http://thesecretlivesofdata.com/raft/scripts/model/node.js:707:30)
at Node.EventDispatcher.dispatchEvent (http://thesecretlivesofdata.com/scripts/playback/playback.js:776:26)
at Node.dispatchEvent (http://thesecretlivesofdata.com/raft/scripts/model/node.js:739:53)
at Node.EventDispatcher.dispatchChangeEvent (http://thesecretlivesofdata.com/scripts/playback/playback.js:791:17)
at Node.candidateEventLoop (http://thesecretlivesofdata.com/raft/scripts/model/node.js:359:14)
at Node.state (http://thesecretlivesofdata.com/raft/scripts/model/node.js:267:18)
at Context. (http://thesecretlivesofdata.com/raft/test/model/node.js:75:19)
at Hook.Runnable.run (http://thesecretlivesofdata.com/scripts/mocha/mocha-1.14.0.js:4263:32)
at next (http://thesecretlivesofdata.com/scripts/mocha/mocha-1.14.0.js:4526:10)
at http://thesecretlivesofdata.com/scripts/mocha/mocha-1.14.0.js:4538:5