Vision-Language Models (VLM) assembled

Drag the query — the embedding of a question like "what's in the sky?" Cross-attention (Chapter 3) blends the image patches into one context vector; that context vector is then matched, nearest-neighbor style (Chapter 1), against a small set of answer prototypes. Two ideas, chained, produce one answer.

The whole pipeline is two formulas run back to back, neither one new:

• $q$ — the query: the embedding of the question being asked about the image.
• $k_i$ — the key vector for image patch $i$, the same cross-attention mechanism from Chapter 3.
• $d$ — the dimensionality used to rescale the raw attention scores.
• $\text{context}$ — the blended vector cross-attention produces, summarizing where the query looked.
• $a$ — one candidate answer prototype from a small set of possible answers.
• $\text{answer}$ — the answer prototype nearest to the context vector, chosen by nearest-neighbor search.

1. Cross-attention decides where to look

   It blends the image patches into a single context vector, weighted by relevance to the query.

2. Nearest-neighbor decides what to call it

   It matches that context vector against a small set of answer prototypes and picks the closest one.

3. Neither step knows about the other's internals

   The context vector is just a point in space — the nearest-neighbor search doesn't care how that point was produced, only where it landed.

Drag the query anywhere and check the full ranked list, not just the winner. The runner-up answer is always the patch that received the second-most attention — the ranking isn't a coincidence, it's a direct consequence of how much each patch pulled the context vector toward itself.

Query aligned with the sky patch, $q=(2,0)$ — the exact numbers from Chapter 3:

1. Cross-attention produces a context vector

   With $q=(2,0)$, the same score→softmax steps as Chapter 3 give:

   • Sky: score $4/\sqrt2\approx2.828$, weight $\approx0.768$
   • Ground: score $0/\sqrt2=0$, weight $\approx0.045$
   • Dog: score $2/\sqrt2\approx1.414$, weight $\approx0.187$

   Weighting each patch and summing: $0.768(2,0)+0.045(0,2)+0.187(1,1) \approx (1.723,\ 0.277)$.

2. Nearest-neighbor picks the answer

   Distance from $(1.723, 0.277)$ to each prototype:

   • Sky $(2,0)$: $\sqrt{(1.723-2)^2+(0.277-0)^2}\approx0.39$
   • Dog $(1,1)$: $\sqrt{(1.723-1)^2+(0.277-1)^2}\approx1.02$
   • Ground $(0,2)$: $\sqrt{(1.723-0)^2+(0.277-2)^2}\approx2.44$

   Sky wins by more than double the runner-up's distance.

3. A query with no preference at all

   At $q=(0,0)$, attention is exactly uniform — $\frac13$ to each patch — and the context vector lands at $(1,1)$, which happens to be exactly the dog prototype's own coordinates. Perfectly even attention still produces a definite, confident answer here, just not necessarily the one you'd expect.