Part IV — Peer-to-Peer & Decentralized Data · Chapter 4

Kademlia & structured P2P overlays

Hook

BitTorrent's DHT and IPFS both route lookups across millions of peers with no ring, no clockwise successor, no central index — just a way to measure "distance" between two ids that has nothing to do with where either one sits on a number line. What could "distance" even mean if not that?

Intuition
self is node 6 — distance is XOR, so bar length has nothing to do with numeric size

Click through the three targets. The two "closest" nodes are a completely different pair each time — not because the nodes moved, but because XOR distance depends only on which bits differ, not on how far apart the numbers look.

Formalize

Kademlia measures distance between two ids by XOR-ing them together and reading the result as a number:

d(a,b)=abd(a, b) = a \oplus b

Every node keeps its known peers in k-buckets: bucket ii holds peers at distance [2i,2i+1)[2^i,\, 2^{i+1}) from itself. A lookup for target tt simply asks whichever known peers are currently closest to tt, who in turn know peers even closer — narrowing in each round.

  • aba \oplus b — bitwise XOR: a 1 in a bit position exactly where aa and bb differ there.
  • bucket ii — holds peers at distance 2i2^i to 2i+112^{i+1}-1 from self; low ii means "very close."
  • closest-kk — the kk known ids with the smallest d(,t)d(\cdot, t) to a target tt; the contact list a lookup uses.
  1. XOR ignores magnitude entirely

    d(a,b)d(a,b) depends only on which bits differ, never on which number is "bigger" — ids 00 and 1515 can be just as close as ids 66 and 77, if the right bits happen to match.

  2. Distance is symmetric and zero only for identical ids

    ab=baa \oplus b = b \oplus a, and ab=0a \oplus b = 0 exactly when a=ba = b — so XOR behaves like a genuine distance, just not a familiar numeric one.

Play

Drag the target id across the whole range and switch kk between 1, 2, and 3. The bars resort completely with every move — there's no "nearby" region of the number line the way there was on Chord's ring; closeness is purely bitwise.

Worked example

Self is node 66 (binary 01100110); other known nodes are {7,4,3,0,9,13}\{7, 4, 3, 0, 9, 13\}:

  1. Every node's k-bucket, by hand
    • d(6,7)=67=1d(6,7)=6\oplus7=1 (bucket 0)
    • d(6,4)=64=2d(6,4)=6\oplus4=2 (bucket 1)
    • d(6,3)=63=5d(6,3)=6\oplus3=5 (bucket 2, since 45<84 \le 5 < 8)
    • d(6,0)=60=6d(6,0)=6\oplus0=6 (bucket 2, since 46<84 \le 6 < 8)
    • d(6,9)=69=15d(6,9)=6\oplus9=15 (bucket 3, since 815<168 \le 15 < 16)
    • d(6,13)=613=11d(6,13)=6\oplus13=11 (bucket 3, since 811<168 \le 11 < 16)
  2. Closest 3 to target 10
    • d(9,10)=910=3d(9,10)=9\oplus10=3
    • d(13,10)=1310=7d(13,10)=13\oplus10=7
    • d(3,10)=310=9d(3,10)=3\oplus10=9
    • d(0,10)=010=10d(0,10)=0\oplus10=10
    • d(7,10)=710=13d(7,10)=7\oplus10=13
    • d(4,10)=410=14d(4,10)=4\oplus10=14

    Sorted ascending, the 3 nearest are {9,13,3}\{9, 13, 3\} — none of them is even in the same bucket as another.

  3. Closest 2 to target 1
    • d(0,1)=01=1d(0,1)=0\oplus1=1
    • d(3,1)=31=2d(3,1)=3\oplus1=2
    • d(4,1)=41=5d(4,1)=4\oplus1=5
    • d(7,1)=71=6d(7,1)=7\oplus1=6
    • d(9,1)=91=8d(9,1)=9\oplus1=8
    • d(13,1)=131=12d(13,1)=13\oplus1=12

    The 2 nearest are {0,3}\{0, 3\} — the same pair that shares bucket 2 relative to self, because both happen to share their two lowest bits with target 11.

Checkpoint

Slide the target id so the 2 nearest contacts (by XOR distance) are exactly nodes 0 and 3.

Drag the slider to pick a target id
Summary
d(a,b)=ab,closest-k(t)=the k ids minimizing d(,t)d(a,b) = a \oplus b, \qquad \text{closest-}k(t) = \text{the } k \text{ ids minimizing } d(\cdot, t)

Kademlia replaces Chord's ring-and-successor routing with a metric that needs no ordering at all: XOR distance, plus k-buckets that keep detailed knowledge of nearby ids and coarse knowledge of far ones. It's the same "narrow in toward the target, one hop at a time" idea as Chord's finger tables, just built on a different notion of "close" — and it's the routing scheme underneath BitTorrent's mainline DHT and IPFS's peer discovery today.