Cross-attention mechanisms

This is Part IV's attention mechanism again, but $Q$ now comes from a text token while $K$ and $V$ come from image patches — sky, ground, and dog. Drag the query: it's still one attention computation, just spanning two modalities instead of one sequence attending to itself.

The formula hasn't changed at all from Part IV:

• $q$ — the query vector, coming from a text token.
• $k_i$ — the key vector for image patch $i$.
• $\text{score}_i$ — the raw compatibility between the query and image patch $i$, before normalizing.
• $w_i$ — the attention weight on image patch $i$, after softmax turns the scores into a distribution.
• $\text{context}$ — the weighted blend of all image patches that the query actually attends to.
• $d$ — the dimensionality of the key vectors, used to rescale the raw scores.

1. Self-attention: everything from one sequence

   In Part IV, $q$, $k_i$, and $v_i$ all came from the same sequence attending to itself.

2. Cross-attention: q and k/v from different modalities

   Here $q$ comes from one modality's encoder (text) while $k_i, v_i$ come from a completely different one (image patches) — the same mechanism, just pointed across a modality boundary instead of within one sequence.

Watch the context vector as you drag. It's always a weighted blend of all three image patches — never any single one exactly, unless one weight reaches all the way to $1$. This is what a text token "looking at" an image actually produces: not a chosen patch, but a soft mixture shaped by how much each patch matters to that particular word.

At query $q=(1,2)$ over patches sky$=(2,0)$, ground$=(0,2)$, dog$=(1,1)$:

1. Raw scores, scaled by root d

   Dot product with each key, then divide by $\sqrt{2}$:

   • Sky: $q\cdot k_{\text{sky}} = 1(2)+2(0) = 2$, so $2/\sqrt2\approx1.414$
   • Ground: $q\cdot k_{\text{ground}} = 1(0)+2(2) = 4$, so $4/\sqrt2\approx2.828$
   • Dog: $q\cdot k_{\text{dog}} = 1(1)+2(1) = 3$, so $3/\sqrt2\approx2.121$
2. Softmax into weights

   Exponentiate each and divide by their sum $\approx4.114+16.92+8.339=29.373$:

   • Sky: $e^{1.414}\approx4.114$, so $4.114/29.373\approx0.140$
   • Ground: $e^{2.828}\approx16.92$, so $16.92/29.373\approx0.576$
   • Dog: $e^{2.121}\approx8.339$, so $8.339/29.373\approx0.284$

   The ground patch dominates, since this query leans more toward "ground" than "sky" or "dog" in raw dot product.

3. The context vector

   Weighting each patch by its share and summing:

   • Sky: $0.140(2,0) = (0.280, 0)$
   • Ground: $0.576(0,2) = (0, 1.152)$
   • Dog: $0.284(1,1) = (0.284, 0.284)$

   Summing componentwise: $(0.280+0+0.284,\ 0+1.152+0.284) = (0.564,\ 1.436)$ — noticeably closer to the ground patch's own coordinates than to sky's, but pulled partway toward the other two, exactly reflecting their smaller but nonzero weights.