Vanishing gradients in RNNs & LSTMs

An RNN (Recurrent Neural Network) unrolled over $T$ timesteps is a $T$-layer chain with the same weight reused at every layer — Part III's vanishing-gradient math, applied to time. Slide the timestep count and watch the plain RNN's gradient (dim line) collapse by roughly an order of magnitude every step, while an LSTM (Long Short-Term Memory)'s cell-state gradient (bold line) barely bends.

A plain RNN's gradient reaching timestep 1 from timestep $T$ is a product of $T{-}1$ factors, each one a squashing derivative times the recurrent weight:

• $\dfrac{\partial h_T}{\partial h_1}$ — the gradient of the last hidden state with respect to the first: how much a change at step 1 still affects step $T$.
• $\tanh'(z_t)$ — the derivative of the tanh activation at timestep $t$'s pre-activation, a value always less than $1$ that shrinks the gradient at every step.
• $W_h$ — the recurrent weight reused at every timestep.
• $T, t$ — the total number of timesteps, and $t$ the index running from $2$ up to $T$.

An LSTM keeps a separate cell state $c_t = f_t \, c_{t-1} + i_t \, g_t$, and its backward path through the cell state is just:

• $\dfrac{\partial c_T}{\partial c_1}$ — the gradient of the final cell state with respect to the first cell state.
• $c_t$ — the LSTM's cell state at timestep $t$: a separate running memory, distinct from the hidden state.
• $f_t$ — the forget gate at timestep $t$: how much of the previous cell state to keep, between $0$ and $1$.

1. No squashing-derivative factor

   Unlike the plain RNN's gradient, there's no $\tanh'$ factor in this product — nothing here is forced below $1$ by a squashing derivative at every step.

2. A forget gate near 1 preserves the gradient

   If the forget gate $f_t$ stays close to $1$, this product barely shrinks at all — the "constant error carousel" that gives LSTMs their long memory.

Both curves use the exact same weight value — this isn't about the LSTM having bigger numbers to work with. The plain RNN multiplies by a squashing derivative and the weight at every step; the LSTM's cell-state path multiplies by only the forget gate. Removing one shrinking factor per timestep is the entire difference, and over enough timesteps it compounds into an astronomical gap.

At weight (or forget gate) $0.9$ and $10$ timesteps:

1. Plain RNN gradient

   Each timestep multiplies by that step's own $\tanh'(z_t)\cdot w_h$. The recurrence's input is constant every step, so $z_t$ converges fast toward a fixed point ($\approx1.86$), where $\tanh'(z_t)\approx0.094$ — a steady-state per-step factor of $\approx0.094\times0.9\approx0.084$. Multiplying nine such factors together (nine transitions across ten timesteps): $0.084^9\approx2\times10^{-10}$ — the same order of magnitude as the actual $\approx2.9\times10^{-10}$, the small difference coming from the first couple of steps, before $z_t$ has fully settled, contributing slightly larger factors. Nine orders of magnitude gone either way.

2. LSTM cell-state gradient

   $0.9^9 \approx 0.387$ — still very much alive.

3. Compare

   The ratio between them is over a billion, using the same $0.9$ on both sides.