Recurrent Neural Networks (RNN)

A recurrent network keeps a single running number, the hidden state, starting at $h_0 = 0$. At every step it folds in the next token and its own previous state: $h_1$ depends on token 1, $h_2$ depends on token 2 and on $h_1$ — which already encoded token 1. Nothing is ever summed away. Slide $W_h$ and watch how much the earlier token's fingerprint survives into $h_2$.

• $h_t$ — the hidden state at timestep $t$: the running summary the network carries forward.
• $x_t$ — the input token at timestep $t$.
• $W_x$ — the weight matrix applied to the current input token.
• $W_h$ — the weight matrix applied to the previous hidden state (the recurrent weight).
• $h_{t-1}$ — the hidden state carried over from the previous timestep.
• $b$ — the bias term.
• $\tanh$ — the squashing activation that keeps the hidden state bounded between $-1$ and $1$.

1. Weights shared across time

   The same weights $W_x$, $W_h$, $b$ are reused at every timestep — exactly the "share one small set of weights across many positions" idea from convolution, just applied across time instead of space.

2. Recurrence carries the whole history

   The recurrence is what makes $h_2$ depend on the entire sequence seen so far, not just the token that just arrived.

These are the exact two sequences from Chapter 5 — "X then Y" and "Y then X" — the ones a sum-pooled network could never tell apart, because they pooled to the identical vector $(1,1)$ no matter the order. Watch $h_1$ here: it's already different for the two sequences, before the second token even arrives, simply because the RNN saw a different first token. Sum-pooling could never do that.

With $W_x = (1,-1)$, $b=0$, $W_h=1$:

1. "X then Y"

   $h_1 = \tanh(1) \approx 0.762$, then $h_2 = \tanh(-1 + 1 \times 0.762) = \tanh(-0.238) \approx -0.234$.

2. "Y then X"

   $h_1 = \tanh(-1) \approx -0.762$, then $h_2 = \tanh(1 + 1 \times (-0.762)) = \tanh(0.238) \approx 0.234$.

3. Compare

   Same weights, same two tokens — but processed in the opposite order, the final hidden states land on opposite signs. That sign is something a simple readout layer can actually use.