Break a Raft cluster
Five servers agree on one log with Raft. Crash them, cut the network and lose messages, and watch the five safety properties hold.
Logs
- Node 1
- 1
- 1
- 1
- Node 2
- 1
- 1
- 1
- Node 3
- 1
- 1
- 1
- Node 4
- 1
- 1
- 1
- Node 5
- 1
- 1
- 1
Each lamp is one entry, with its term under it, as its node knows it. A filled lamp is committed: a majority holds it, and it can never be lost. A hollow lamp is one its node has not heard is committed yet. A crash keeps a node's log but not what it knew was committed, which Raft holds only in memory, so a node that is down shows its whole log hollow until the leader tells it again.
All five safety properties held over 249 events.
Virtual time 1.400 s, seed 7
What happened
- 0.159 s Node 2 times out and asks for votes in term 1
- 0.213 s Node 1 wins term 1 with 3 votes
- 0.227 s Node 2 sees term 1 and becomes a follower
- 0.700 s Node 1 takes write 1 at index 1 in term 1
- 0.767 s Node 1 commits up to index 1, an entry of term 1
- 0.769 s Write 1 is committed at index 1: it can never be lost
- 0.900 s Node 1 takes write 2 at index 2 in term 1
- 0.960 s Node 1 commits up to index 2, an entry of term 1
- 0.963 s Write 2 is committed at index 2: it can never be lost
- 1.100 s Node 1 takes write 3 at index 3 in term 1
- 1.153 s Node 1 commits up to index 3, an entry of term 1
- 1.157 s Write 3 is committed at index 3: it can never be lost
What Raft promises
Raft keeps one log on several servers, so that they act as one machine that stays up when some of them fail. One server leads. It takes every write, copies it to the others, and calls it committed once a majority holds it. If the leader goes quiet, a follower whose timer runs out asks for votes, and a server that wins a majority leads the next term.
What it guarantees is safety, whatever the network does: two servers never commit different entries at the same place in the log, and a committed entry is never lost. While a majority can talk to each other, it also makes progress. A minority never can, by design.
Three things to try
Cut off the leader. Press Run, then Cut off the leader. The others stop hearing its heartbeats; one of them times out, wins a new term and leads. The old leader still thinks it leads. Send a write: it takes it, but can never commit it, since it cannot reach a majority.
Split 2 and 3. The leader and one neighbour end up on the small side. Writes sent to it pile up uncommitted, while the three on the other side elect a leader of their own and commit theirs.
Heal the network. The old leader hears the newer term and steps down. Its uncommitted entries conflict with the new leader’s log, so they are dropped and replaced. The event log names each one. That is the rule doing its job: those entries were never promised to anyone.
Then try crashing two servers at once, or three, and losing a third of all messages.
What the colors and shapes mean
Each server’s ring is its election timer: when it closes, the server asks for votes. The leader’s ring stays full. A hollow dot on the wire asks for a vote; a filled dot grants one. A filled square carries log entries; an outlined square is a heartbeat with none. When the network is cut, a dashed outline fences off the smaller side, and no message crosses it. In each log, a cell is one entry with its term under it, filled once committed.
Credits
The algorithm is Diego Ongaro and John Ousterhout’s, from In Search of an Understandable Consensus Algorithm (USENIX ATC 2014). The picture of five servers on a circle comes from Ongaro’s RaftScope, the visualization on the Raft website. This demo shares none of its code.
What it costs
Nothing to run. Every file is static, and your browser runs the simulation.
How it is built
The servers are one TypeScript module that follows Figure 2 of the Raft paper. Time, the network and every random choice belong to a simulator around them: virtual time, one seeded random stream, and your presses recorded at the moment they happened. The same seed and the same presses give the same run, every time. That is how the page opens with a picture the build drew, and how the browser then takes over at the exact same moment.
After every event, a checker tests the five safety properties of Figure 3. The same simulator runs 3,000 seeded runs of crashes, splits and lost messages in the site's tests, and two deliberately broken servers prove those runs can find a real bug: one that votes without checking the candidate's log, and one that commits an old term's entry by counting copies.