Self-attention & multi-head attention

Row "sat" is exactly uniform — it sits equally close to both "the" and "cat", so it has no reason to favor either.

This is self-attention: the query, the keys, and the values are all the same sequence. Every row is one token acting as a query; every column is a token it can attend to, itself included. "The" mostly attends to itself; "sat," sitting exactly between "the" and "cat," has no reason to prefer either.

A single attention computation is one view of the sequence. Multi-head attention runs several of these views in parallel, each looking at the tokens through a different projection, then combines what they found:

• $Q, K, V$ — the query, key, and value matrices, here all built from the same input sequence since this is self-attention.
• $W_i^Q, W_i^K, W_i^V$ — head $i$'s own learned projection matrices, giving that head its own view of the sequence.
• $\text{head}_i$ — the output of attention computed inside head $i$, using its own projections.
• $h$ — the number of attention heads running in parallel.
• output — the combined result of all heads, concatenated together.

1. Each head learns its own view

   Each head's $W^Q, W^K, W^V$ are learned separately, so each head is free to pick up on a completely different signal in the same tokens — one might track syntax, another meaning, another position.

2. No single head does all the work

   Because the heads specialize independently, the model isn't forced to squeeze every kind of relationship through one shared projection.

query = "sat" — Head A row: [0.67, 0.09, 0.24], Head B row: [0.09, 0.67, 0.24] — disagreement = 0.813

Two toy heads, one projecting each token onto just its horizontal axis, the other onto just its vertical axis. Click through the query tokens. For "the" and "cat," one head goes flat (uniform) while the other has a clear opinion — that head simply has nothing to say, not a real disagreement. Watch "sat," though: both heads have a strong, opposite opinion about where to look.

1. Project the query

   As a query, "sat" $=(1,1)$ projects to $1$ on both axes.

2. Head A (horizontal)

   Multiplying the query's projection ($1$) by each token's own horizontal projection:

   • "the" $=2$: score $1\times2=2$
   • "cat" $=0$: score $1\times0=0$
   • "sat" $=1$: score $1\times1=1$

   Softmax of $(2,0,1)$ gives roughly $(0.665, 0.090, 0.245)$ — favoring "the."

3. Head B (vertical)

   The exact same query projection ($1$), now against each token's vertical projection:

   • "the" $=0$: score $1\times0=0$
   • "cat" $=2$: score $1\times2=2$
   • "sat" $=1$: score $1\times1=1$

   Softmax of $(0,2,1)$ gives roughly $(0.090, 0.665, 0.245)$ — favoring "cat," the mirror image.

4. Concatenate

   Combining both heads' outputs keeps both signals side by side, instead of forcing one projection to average them away.