The Gated Recurrent Unit (GRU)

The same fixed input every step, an update gate biased almost shut. Watch the hidden state creep upward, a little more each step, never resetting and never exploding — the signature of a gate that's mostly saying "keep what you already have."

A GRU — short for Gated Recurrent Unit — cell has two gates instead of the LSTM's three, and only one state instead of two:

• $z_t$ — the update gate: how much of the new candidate state to blend in versus keeping the old hidden state.
• $r_t$ — the reset gate: how much of the previous hidden state the candidate is allowed to see.
• $x_t$ — the input at time step $t$.
• $h_{t-1}$ — the hidden state carried over from the previous time step.

• $\tilde{h}_t$ — the candidate hidden state: a fresh proposal for the new state, computed from the current input and the (possibly reset) previous state.
• $h_t$ — the new hidden state: a blend of the old state and the candidate, weighted by the update gate.

1. The update gate merges two LSTM jobs into one

   $z_t$ plays the LSTM's forget gate and input gate at once: it's a single dial between "keep $h_{t-1}$ exactly" ($z_t{\approx}0$) and "replace it with the new candidate" ($z_t{\approx}1$).

2. The reset gate controls what the candidate sees

   $r_t$ controls how much of the old state the candidate itself is allowed to see before being blended in.

The same 10-step gradient comparison from the LSTM chapter, with the GRU added: a plain RNN's gradient has collapsed to essentially zero, the LSTM's typical forget gate of $0.9$ holds onto a modest fraction, and this GRU's update gate — biased hard toward "remember" — preserves more gradient than either, using one fewer gate to do it.

Fixed input $x_t=1$ every step, weights biasing $z$ toward $0$ and $r$ toward $1$:

1. Both gates settle to the same value every step

   With $W_z{=}U_z{=}0$, $b_z{=}-4$, $b_r{=}4$:

   • $z_t = \sigma(-4) \approx 0.0180$ at every timestep, regardless of the hidden state.
   • $r_t \approx 0.9820$ — the reset gate stays wide open, barely touching the candidate.
2. The hidden state creeps forward

   With $z_t\approx0.018$ and $r_t\approx0.982$ fixed every step, the recurrence becomes $h_t \approx 0.982\,h_{t-1} + 0.018\tanh(1+0.491\,h_{t-1})$ (since $W{=}1$, $U{=}0.5$, and $x_t{=}1$). Starting from $h_0=0$:

   • $h_1 = 0.982(0)+0.018\tanh(1+0) \approx 0.0137$
   • $h_2 = 0.982(0.0137)+0.018\tanh(1+0.491\times0.0137) \approx 0.0272$
   • $h_3 \approx 0.982(0.0272)+0.018\tanh(1+0.491\times0.0272) \approx 0.0406$

   Each step keeps about $98.2\%$ of the old state and blends in only a shrinking sliver of the (slowly rising) candidate. Continuing the same recurrence for 7 more steps reaches $h_{10} \approx 0.128$ — small, steadily increasing, never overshooting.

3. The same gate that shapes memory shapes the gradient

   The dominant path for $\partial h_{10}/\partial h_1$ is a product of $(1-z_t)$ at each step: $(0.982)^9 \approx 0.849$. The LSTM's typical forget gate of $0.9$ gives $(0.9)^9 \approx 0.387$ over the same span — this GRU's bias toward remembering preserves gradient even more aggressively, with no separate cell state required.