Vanishing & exploding gradients

This is a chain of sigmoid layers, same weight repeated at every layer. Drag depth up and watch the gradient reaching the input drop — not by a little each layer, but by roughly one more order of magnitude every single time.

Going backward one layer multiplies the gradient by $\sigma'(z)\cdot w$ — Chapter 5's chain rule, once per layer. Sigmoid's derivative peaks at exactly $0.25$ and is smaller everywhere else:

• $L$ — the loss.
• $h_i$ and $h_{i-1}$ — the hidden layer outputs at layer $i$ and the layer just before it.
• $z_i$ — layer $i$'s raw weighted sum, before its activation.
• $\sigma'(z_i)$ — sigmoid's derivative at that point, capped at $0.25$.
• $w$ — the weight, repeated at every layer in this example.

1. Repeated shrinking compounds to nothing

   Multiply a number smaller than $1$ by itself ten times and it's gone.

2. No weight scale rescues sigmoid

   Unlike Chapter 8's forward pass, there's no weight scale that fixes this — a bigger $w$ pushes $z$ further from $0$, which makes $\sigma'(z)$ even smaller, not larger. Sigmoid's derivative is capped at $0.25$, so this particular chain can only ever shrink going backward.

3. Exploding gradients are the mirror image

   The mirror-image failure — exploding gradients — shows up instead in linear or ReLU layers with large weights, where nothing caps that per-layer factor above $1$; it's the exact same backward multiplication as Chapter 8's forward one, just run in reverse.

Slide from a shallow to a deep chain. Five layers in, the gradient is already down around $10^{-5}$; by ten layers it's down to about $2\times10^{-9}$ — utterly unusable for updating a weight that far back. This is the vanishing gradient problem, and it's exactly why sigmoid fell out of favor as networks got deep.

With weight $w=3$:

1. Check a single layer

   $\sigma'(z_0)\cdot w$ is at most $0.25\times3=0.75$, so even the very first backward step shrinks the gradient.

2. Follow it to depth 6

   Each backward step multiplies by that layer's own $\sigma'(z_i)\cdot w$ — and $w=3$ pushes the forward pass toward saturation fast (the signal climbs from $0.5$ toward $\approx0.94$ within a couple of layers, where $\sigma'$ is tiny), so the six per-layer factors, computed layer by layer from the output backward, are all well under the depth-1 bound of $0.75$:

   • $\approx0.158,\ \approx0.158,\ \approx0.159,\ \approx0.167,\ \approx0.219,\ \approx0.447$

   Multiplying the output gradient ($\approx-0.056$) by all six: $-0.056\times0.158\times0.158\times0.159\times0.167\times0.219\times0.447\approx-3.6\times10^{-6}$ — already roughly four orders of magnitude smaller than the loss gradient at the output.

3. One layer further

   It drops under $10^{-6}$ entirely: for practical purposes, that weight has stopped learning.