The attention mechanism

Three tokens sit at fixed points; a query (the ×) is what's currently "looking." Drag it around. The token whose vector points in a similar direction to the query gets more weight — dragging toward "cat" pulls almost all the attention onto "cat," dragging to the middle spreads it evenly across all three.

For a query $q$ and a set of token (key) vectors $k_i$, attention computes a similarity score for each one, turns those scores into weights that sum to $1$, and returns their weighted blend:

• $q$ — the query vector: what's currently "looking," trying to find relevant tokens.
• $k_i$ — the key vector for token $i$, one of the fixed set of tokens being attended to.
• $d$ — the dimensionality of the query and key vectors, used to rescale the dot product.
• $\text{score}_i$ — the raw, scaled similarity between the query and token $i$'s key.
• $w_i$ — the attention weight on token $i$, the softmax-normalized score.
• context — the weighted blend of all the key vectors: the actual output of attention.

1. Why scale by the square root of d

   The $\sqrt{d}$ divides out the fact that dot products grow with dimension $d$ — without it, scores in a high-dimensional model would blow up and softmax would saturate into an almost one-hot distribution regardless of which token actually matches best.

Watch the context vector — the actual output of attention — as you drag the query. It's always some blend of "the," "cat," and "sat," and it moves smoothly as the weights shift. This is the payoff: instead of one RNN hidden state trying to remember an entire sequence, attention produces a fresh, query-specific blend of all of it, recomputed on demand.

At query $q=(2,0)$ — aligned with "the" $=(2,0)$:

1. Raw dot products

   With "the"$=(2,0)$, "cat"$=(0,2)$, "sat"$=(1,1)$:

   • "the": $(2)(2)+(0)(0)=4$
   • "cat": $(2)(0)+(0)(2)=0$
   • "sat": $(2)(1)+(0)(1)=2$
2. Scale by the square root of d

   Dividing each by $\sqrt{2}\approx1.414$:

   • "the": $4/1.414\approx2.828$
   • "cat": $0/1.414=0$
   • "sat": $2/1.414\approx1.414$
3. Softmax into weights

   Exponentiating each score and dividing by their sum $16.92+1+4.11\approx22.03$:

   • "the": $e^{2.828}\approx16.92$, so $16.92/22.03\approx0.768$
   • "cat": $e^0=1$, so $1/22.03\approx0.045$
   • "sat": $e^{1.414}\approx4.11$, so $4.11/22.03\approx0.187$

   "The" dominates, but "sat" (whose vector partially overlaps) still gets a meaningful share, and "cat" (pointing in an unrelated direction) gets almost none.