Video understanding & spatiotemporal attention

Six frames of an up-down motion. Pick a query frame and see which other frames it attends to — not just its neighbors in time, but any frame anywhere in the clip, scored by how similar its motion state is.

Treat each frame as a token — exactly the way ViT treats each image patch as a token — and add a temporal position embedding instead of a spatial one:

• $\text{frame}_t$ — the video frame at time step $t$, treated as one token, the way ViT treats one image patch.
• $\text{embedding}_t$ — that frame's learned token embedding.
• $\text{position}_t$ — the temporal position embedding, marking when in the clip the frame occurs, in place of ViT's spatial position embedding.

1. Local methods are capped by a fixed window

   A local method — frame differencing, a small 1D convolution, a short-memory recurrence — can only relate frames within some fixed temporal window. Anything outside that window is invisible to it.

2. Self-attention has no such limit

   Any two frames, however far apart in time, can attend to each other directly, in a single layer — reach doesn't depend on window size at all.

Frame 0's attention concentrates almost entirely on frame 4 — four frames away, well outside anything a local window could ever reach — because it shares the same motion state. Distance in time doesn't matter; matching content does.

Six frames, alternating high/low motion state, query = frame 0:

1. Temporal distance rules out a local method entirely

   Frame 0 and frame 4 are $4$ frames apart — outside a local window of size $1$, which only covers immediately adjacent frames. A frame-differencing method comparing only neighbors would never connect them.

2. Self-attention connects them anyway

   Query = frame 0, embedding $(2,0)$. Frame 4 shares the same "high" motion state, so its dot product with the query is identical: $q\cdot k_{\text{frame 4}} = (2)(2)+(0)(0.4) = 4$, scaled to $4/\sqrt2\approx2.828$ — the same score frame 0 gets against itself. Softmax over all six frames (three "high" scores of $2.828$, three "low" scores of $0$):

   • $e^{2.828}\approx16.92$ (three of these)
   • $e^0=1$ (three of these)

   Summing to $3(16.92)+3(1)=53.76$. Each high-state frame's share is $16.92/53.76\approx0.315$ — frame 4 gets that same $\approx0.315$ of frame 0's attention, in one layer, with no penalty for the temporal gap.

3. Motion state beats recency

   Frame 1 is "low" state, so its dot product with the query is $0$ (e.g. $q\cdot k_{\text{frame 1}}=(2)(0)+(0)(0.1)=0$), giving $e^0=1$ in that same softmax. From the sum of $53.76$ above, each low-state frame's share is $1/53.76\approx0.019$ — sixteen times less than the temporally distant frame 4. Being close in time doesn't help if the motion doesn't match.