Vision Transformers

A 3x3 grid of image patches, with a bright diagonal running through it. Pick a query patch and watch which other patches it attends to — not just its neighbors, but any patch anywhere in the grid, scored purely by how similar its content is.

A Vision Transformer (ViT) cuts an image into fixed-size patches, embeds each one, adds a position embedding, and feeds the resulting sequence into the exact same self-attention block from Part IV — no convolution anywhere:

• $\text{patch}_i$ — the $i$-th fixed-size image patch, treated like a token in a sequence.
• $\text{embedding}_i$ — the learned vector representation of patch $i$.
• $\text{position}_i$ — a position embedding added to patch $i$, since self-attention has no inherent sense of spatial order.

1. A conv kernel's reach grows one hop at a time

   A convolution kernel's receptive field grows by one step of Chebyshev distance per layer — reaching a patch $d$ steps away takes at least $d$ stacked layers.

2. Self-attention reaches everywhere in one layer

   Self-attention has no such restriction: every patch can attend to every other patch in a single layer, regardless of how far apart they sit in the image.

Top-left's attention lands overwhelmingly on the two other bright patches — including bottom-right, which sits on the opposite corner of the grid. A single 3x3 convolution centered on top-left couldn't reach bottom-right at all; self-attention reaches it in one hop, using nothing but content similarity.

A 3x3 grid, bright diagonal (top-left, center, bottom-right), query = top-left:

1. Distance rules out a single conv layer entirely

   Top-left and bottom-right sit at Chebyshev distance $2$ — outside a single $3{\times}3$ kernel's reach, which only covers distance $1$. Connecting them with convolutions alone would take at least two stacked layers.

2. Self-attention connects them anyway

   Query = top-left, embedding $(2,0)$. Both top-left and bottom-right are "bright" patches, so their dot products with the query are identical: $q\cdot k_{\text{bottom-right}} = (2)(2)+(0)(0.8) = 4$, scaled to $4/\sqrt2\approx2.828$ — the same score top-left gets against itself. Softmax over all nine patches:

   • Three "bright" scores of $2.828$: $e^{2.828}\approx16.92$ each
   • Six "dark" scores of $0$: $e^0=1$ each

   Summing to $3(16.92)+6(1)=56.76$. Each bright patch's share is $16.92/56.76\approx0.298$ — bottom-right gets that same $\approx0.298$ of top-left's attention, in a single layer, with no distance penalty of any kind.

3. Content beats proximity

   The adjacent top-middle and mid-left patches are "dark," so their dot product with the query is $0$ (e.g. $q\cdot k_{\text{top-middle}}=(2)(0)+(0)(0.1)=0$), giving $e^0=1$ in that same softmax. From the sum of $56.76$ above, each dark patch's share is $1/56.76\approx0.018$ — squarely inside a conv kernel's reach, but dark, they receive only $\approx0.018$ each. Being close doesn't matter if the content doesn't match; being far away doesn't matter if it does.