Positional encodings: Sinusoidal vs. Relative

Swap which token comes first. The score computed from content alone never budges — it can't, since nothing about it mentions position at all. The moment position gets added in, the swap changes the number.

Sinusoidal positional encoding builds a vector out of sines and cosines at geometrically spaced frequencies, one pair of dimensions per frequency, and adds it directly to the token's content embedding before anything else happens:

• $pos$ — the token's position in the sequence: $0, 1, 2, \dots$
• $i$ — which frequency pair of dimensions this is, out of $d/2$ total pairs.
• $d$ — the embedding dimension: how many numbers make up the encoding vector.

1. Every dimension pair spins at its own rate

   • Pair $i=0$ has a rate of $10000^0=1$, so it completes a full rotation roughly every $2\pi\approx6.3$ positions — fast.
   • Pair $i=1$ divides $pos$ by $10000^{2/d}$ first, so for the same $pos$ it's barely turned at all.

   Stacking fast and slow pairs together lets nearby positions look different in the fast dimensions while far-apart positions still differ in the slow ones.

2. Added, not appended

   $PE_{pos}$ is added elementwise to the content embedding, so a query·key dot product afterward mixes content and position inseparably — exactly what the intuition demo just showed happening to the score.

3. A relative scheme skips the embedding entirely

   Instead of writing where a token is into its vector, a relative scheme (used by T5 and Transformer-XL) adds a bias term straight onto the attention score, looked up by the distance $j-i$ between the query and key positions. Slide the whole sequence over and every distance stays the same, so the bias — and the score — doesn't move at all.

The bold curve is the sinusoidal score for a query and key that are always exactly one position apart, as the pair's starting offset slides from 0 to 20 — it wanders, because it reads each token's absolute position. The dashed line is the relative-bias score for that same distance of 1: flat, because distance is all it ever looks at.

Tokens $A=(1,0,0.5,-0.5)$ and $B=(0,1,-0.5,0.5)$, content score $A\cdot B = -0.5$, with $d=4$ (one fast frequency pair, one slow one):

1. Position 0 is the trivial case

   $\sin(0)=0$ and $\cos(0)=1$ at every frequency, so $PE_0 = (0,1,0,1)$ exactly, regardless of $d$.

2. Position 10, then 11 — the sinusoidal score

   • At $pos=10$: rate $i=0$ is $1$, rate $i=1$ is $10000^{-1/2}=0.01$, so $PE_{10}=(\sin(10),\,\cos(10),\,\sin(0.1),\,\cos(0.1))\approx(-0.544,\,-0.839,\,0.0998,\,0.995)$.
   • At $pos=11$: $PE_{11}=(\sin(11),\,\cos(11),\,\sin(0.11),\,\cos(0.11))\approx(-1.0,\,0.0044,\,0.110,\,0.994)$.

   Elementwise sums:

   • $A+PE_{10} = (1-0.544,\ 0-0.839,\ 0.5+0.0998,\ -0.5+0.995) \approx (0.456,\,-0.839,\,0.600,\,0.495)$
   • $B+PE_{11} = (0-1.0,\ 1+0.0044,\ -0.5+0.110,\ 0.5+0.994) \approx (-1.0,\,1.004,\,-0.390,\,1.494)$

   Dot product: $0.456(-1.0) + (-0.839)(1.004) + 0.600(-0.390) + 0.495(1.494)$ $\approx -0.456 - 0.843 - 0.234 + 0.739 \approx -0.793$ — close to, but not equal to, the relative score below.

3. The relative score for the same pair, distance 1

   $A\cdot B + \text{bias}(1) = -0.5 + (-0.3) = -0.8$. This number doesn't depend on which two positions were 1 apart — only that they were.