One node in your cluster owns every write. It crashes at 3am. Which of your other nodes gets to take over — and what did you just lose?
The primary (top) has written 7 entries. Each backup (bottom) has replicated some prefix of that log — click one to see what promoting it right now would cost you. A backup that's behind doesn't have the missing writes anywhere else to recover them from.
In primary-backup replication, one designated node — the primary — accepts every write, appends it to its own log, and ships that log to a set of backups. The backups apply entries in the same order, so each one's state is just an earlier prefix of the primary's.
The protocol rule for handling a primary failure:
- Find the most caught-up backup
Compare every backup's
replicatedUpTo— how many of the primary's log entries it has durably received. - Promote it
The backup with the largest
replicatedUpTobecomes the new primary. - Count the damage
Any entry in the old primary's log beyond that backup's
replicatedUpTois permanently lost — there's no third copy to recover it from.
- Primary log — the full, authoritative sequence of writes, in order.
replicatedUpTo— how far a given backup has replicated into that log.- Writes lost —
primary log length − replicatedUpToof whichever backup gets promoted.
Click a lagging backup a few times to replicate it forward. Watch the "best to promote" readout update — and notice the writes-lost count shrink toward zero as a backup catches up. Replication lag isn't a bug; it's the gap that determines how much a crash costs you.
The primary has written 7 entries: w1 … w7. At the moment it crashes, the three backups are at:
- Read off the replication state
- B1 has replicated 5 entries
- B2 has replicated all 7
- B3 has replicated only 3
- Pick the most up-to-date backup
B2, at
replicatedUpTo = 7, is furthest along — it becomes the new primary. - Count writes lost for the winner
. Promoting B2 loses nothing.
- Compare to the alternatives
- Promoting B1 instead would have lost writes (
w6,w7) - Promoting B3 would have lost writes (
w4–w7)
- Promoting B1 instead would have lost writes (
The primary just crashed. Click the backup that should be promoted — the one that loses the fewest writes.
Primary-backup replication is simple: one node takes writes, ships a log, backups replay it. But the primary is a single point of failure, and a crash always costs you whatever the best-replicated backup hadn't yet received. The next chapter's quorum systems are one way to bound that cost precisely instead of leaving it to chance.