Selective state spaces (Mamba / Mamba-2)

Step through the sequence. The faint curve is last chapter's fixed rule — it decays every step, filler or not. The bold curve is selective: dead flat through every run of zeros, then snapping instantly to a new value the moment a real signal arrives.

A selective state-space model (the mechanism behind Mamba) makes $A$ and $B$ functions of the current input instead of fixed constants:

• $h_t$ — the hidden state at time step $t$.
• $h_{t-1}$ — the hidden state from the previous time step.
• $x_t$ — the current input.
• $A(x_t)$ — the decay factor, now itself a function of the current input instead of a fixed constant.
• $B(x_t)$ — the input gate, also a function of the current input instead of a fixed constant.

This chapter's selection rule is the simplest one that still makes the point — a token is either signal ($x_t \neq 0$) or filler ($x_t = 0$):

1. On filler: freeze the state

   $A{=}1,B{=}0$ freezes the state exactly — nothing decays, because there's nothing worth forgetting from.

2. On signal: overwrite it completely

   $A{=}0,B{=}1$ overwrites the state completely — the model chooses to forget everything it was holding, on purpose, because something worth remembering just arrived.

Same 7-token sequence, same starting state, two different final answers. The selective rule's answer is exactly the sequence's last real value; the fixed rule's is a decayed, blended approximation of it — contaminated by a signal from four steps earlier that the fixed rule had no way to just drop.

Sequence $x = (5,\ 0,\ 0,\ 0,\ {-}3)$ — one signal, three filler tokens, a second signal:

1. The first signal writes the state

   $x_1=5$ is signal, so $A{=}0, B{=}1$: $h_1 = 0(0) + 1(5) = 5$.

2. Three filler tokens change nothing

   $x_2{=}x_3{=}x_4{=}0$, each filler: $A{=}1,B{=}0$ gives $h_t = 1(h_{t-1}) + 0 = h_{t-1}$ every time. $h_2=h_3=h_4=5$ — the exact same number, three steps running.

3. The second signal overwrites it completely

   $x_5={-}3$ is signal again: $h_5 = 0(5) + 1({-}3) = -3$. The $5$ isn't faded — it's gone, replaced outright, in exactly the step where replacing it was the right call.