Amazon's shopping cart must accept a write even when part of the network is unreachable — a customer adding an item should never be told "no." But if two datacenters each accept a concurrent write to the same cart while cut off from each other, how does the store even know they conflict, let alone reconcile them later?
Click "A writes" once, then "B writes" once, without ever syncing — the verdict reads concurrent: neither clock has seen the other's update, so neither can claim to be newer. Now hit "Sync" and write on A again: the verdict flips to a clean after, because A's new clock has now seen everything B had.
A vector clock gives every replica its own counter, incremented only on that replica's own writes. Comparing two clocks and coordinate-by-coordinate tells you their causal relationship:
If neither nor holds, the two writes are concurrent — a genuine conflict, not just a stale read. Separately, Dynamo's sloppy quorum guarantees any read quorum of size and write quorum of size , out of replicas, must share at least one node:
- — replica 's counter as recorded in vector clock .
- — " happened-before ": every counter in is the matching one in , with at least one strictly less.
- , , — read quorum size, write quorum size, and total replicas in the preference list.
- Pigeonhole guarantees the overlap
If a size- read set and a size- write set were completely disjoint, together they'd only need distinct nodes. The moment , there aren't enough distinct nodes left for that — some node has to be in both.
- Hinted handoff keeps writes flowing around a down node
When a node in the write quorum is unreachable, the write goes to the next healthy node in the preference list instead, which holds a hint: "forward this to the down node once it's back." The write still succeeds; nothing is silently dropped.
Slide R from 1 up to 5 and watch the overlap column: below R=3 it can hit zero, at R=3 and above it always includes at least N3. Then flip N2 to "down" and watch the W=3 write targets skip straight past it to N4, with N3 now shown holding a hint for N2 — the write never blocks on the down node.
A 5-node preference list N1..N5, write quorum N1, N2, N3:
- Two concurrent writes, then a causal one
- Node A writes with an empty clock:
- Node B writes independently: — concurrent with A's write, since neither nor holds
- After a sync, A writes again: , which is now strictly after B's , since every one of B's counters () is matched or exceeded
- R=2 is not enough
Read set N4, N5 against write set N1, N2, N3: no overlap at all. , which is not strictly greater than — exactly the boundary case where a gap is still possible.
- R=3 closes the gap
Read set N3, N4, N5 against the same write set: they share N3. , so this overlap isn't a coincidence — every size-3 read set must hit the size-3 write set.
- Hinted handoff around N2
Walking the preference list N1..N5 to fill 3 slots:
- N1: healthy → included (slot 1)
- N2: down → skipped; its hint attaches to the next healthy node reached
- N3: healthy → included (slot 2), and holds N2's hint
- N4: healthy → included (slot 3)
Write quorum becomes N1, N3, N4 — and N3, the node that stepped in right after the gap, is recorded as holding N2's hint.
With the write quorum fixed at W=3 out of N=5, slide the read quorum R to the exact minimum value that guarantees every possible read overlaps every possible write — not just any R past that point.
Dynamo's whole design is "never refuse a write": sloppy quorums keep writes and reads overlapping often enough to stay useful without demanding every replica be reachable, hinted handoff keeps a write succeeding around a temporarily down node, and vector clocks are the only honest way to tell a genuine conflict apart from a stale copy once those concurrent writes need reconciling. The capstone puts these same vector clocks to work again, this time reconciling a write made mid-flight through a gossip-based membership change.