Rotary Position Embedding (RoPE)

The key sits fixed at position 0. Step the query's position up and its arrow spins around the origin — the vector's length never changes, only its angle. Watch the dot product: it isn't tracking where the query "is" in any absolute sense, it's tracking how far its angle has swung away from the key's.

RoPE rotates a 2D pair of coordinates by an angle proportional to position, before the dot product is ever taken:

• $pos$ — the token's position in the sequence.
• $\theta$ — the fixed angular rate: how many radians the pair rotates per position step.
• $R(pos)$ — the rotation matrix applied to that coordinate pair before the dot product.

1. Rotation preserves length, and preserves relative angle

   $R$ is orthogonal: it never stretches a vector, only turns it. Turning two vectors by the same angle leaves the angle between them — and so their dot product — completely unchanged.

2. Turning them by different amounts is the same as turning one by the difference

   $R(a)q \cdot R(b)k = q \cdot R(b-a)k$. The rotated dot product between a query at position $a$ and a key at position $b$ is identical to rotating the key alone by $b-a$ and leaving the query untouched. Only the difference of the two positions ever enters the calculation.

3. A real model repeats this across many frequency pairs

   Full RoPE splits the embedding into $d/2$ coordinate pairs, each with its own $\theta$ — exactly the geometric spacing of frequencies from the sinusoidal scheme, just used to rotate instead of to add. This demo uses one pair to keep every angle exact and checkable by hand.

Slide the query's position continuously. The key stays anchored at position 0, so the angle swept by the slider is the relative distance — and the dot product it produces depends on nothing else. At $\theta = \pi/2$, four steps is a full turn: the vector — and the dot product — returns exactly to where it started every four positions.

$Q_0=(1,\,0.5)$, $K_0=(0.5,\,1)$, $\theta = \pi/2$ (a quarter turn per position step):

1. Distance 0: the plain dot product

   $Q_0\cdot K_0 = 1(0.5) + 0.5(1) = 1.0$. With no rotation at all, this is just the ordinary dot product.

2. Distance 2: rotate the key by a half turn

   $\cos(\pi)=-1$, $\sin(\pi)=0$, so $R(\pi)K_0 = (0.5(-1) - 1(0),\ 0.5(0) + 1(-1)) = (-0.5,\,-1)$. Then $Q_0\cdot R(\pi)K_0 = 1(-0.5)+0.5(-1) = -1.0$.

3. Same distance, different absolute positions — identical answer

   Query at position 1, key at position 3 (still 2 apart):

   • $\cos(\pi/2)=0$, $\sin(\pi/2)=1$, so $R(\pi/2)Q_0 = (1(0) - 0.5(1),\ 1(1) + 0.5(0)) = (-0.5,\,1)$
   • $\cos(3\pi/2)=0$, $\sin(3\pi/2)=-1$, so $R(3\pi/2)K_0 = (0.5(0) - 1(-1),\ 0.5(-1) + 1(0)) = (1,\,-0.5)$

   Their dot product: $(-0.5)(1)+(1)(-0.5) = -1.0$ — exactly matching the distance-2 score computed from positions 0 and 2.