ResNet & residual skip connections

Same depth, same per-layer weight, same nonlinearity — the only difference is whether each layer computes a brand new value or adds a small adjustment on top of the one it received. Step through the depths and watch one gradient collapse while the other holds steady.

A plain layer replaces its input entirely: $h_i = F(h_{i-1})$. A residual (skip) connection instead adds the layer's output back onto its input:

• $h_i$ — the representation produced at layer $i$, after the residual connection is applied.
• $h_{i-1}$ — the representation entering layer $i$, produced by the previous layer.
• $F$ — the layer's own transformation (e.g., a weight multiply and nonlinearity) — the same function a plain layer would apply on its own.

1. A plain layer's gradient compounds toward zero

   Backpropagating through a plain layer multiplies the gradient by $F'(h_{i-1})$ — a term that's often well below 1, so a long chain of layers shrinks it toward zero.

2. A residual layer keeps a direct path open

   Backpropagating through a residual layer multiplies by $1 + F'(h_{i-1})$ instead. That leading $1$ is a direct, unobstructed path for the gradient — an "identity shortcut" that depth alone can't erode.

The plain network's gradient shrinks so fast with depth that its bars are barely visible next to the residual network's. By depth 20, the plain network's gradient has vanished to the order of $10^{-17}$; the residual network's has stayed above $100$.

Same weight $w=3$ and sigmoid $\sigma$ in both networks — the plain network's per-layer function is $\sigma(wh)$, and the residual network's added term is $w\cdot\sigma(wh)$ — starting from $h_0=0.5$, at depths 5, 10, and 20:

1. The plain network vanishes on schedule

   Once the forward pass saturates (by around depth 2, same as Part III's chapter), each additional layer's backward factor $\sigma'(z)\cdot w$ settles near a steady $\approx0.158$. Compounding that factor 5 more times multiplies the gradient by roughly $0.158^5\approx9.8\times10^{-5}$ — about 4 orders of magnitude — exactly the pattern in the numbers:

   • Depth 5: $|\partial \mathcal{L}/\partial h_0| \approx 2.29\times10^{-5}$
   • Depth 10 ($\approx2.29\times10^{-5}\times9.8\times10^{-5}$): $\approx2.21\times10^{-9}$
   • Depth 20 (two more such 5-layer jumps): $\approx2.06\times10^{-17}$
2. The residual network doesn't just survive — it stabilizes

   Because each layer adds $w\cdot\sigma(z_i)$ instead of replacing $h$, and $\sigma$ saturates to $1$ almost immediately here, $h$ grows by very close to $w=3$ every layer past the first couple:

   • Depth 5: $h_5\approx14.95$
   • Depth 10: $h_{10}\approx29.95$
   • Depth 20: $h_{20}\approx59.95$

   So the raw gradient at the output, $h_{\text{depth}}-1$, grows roughly linearly with depth too. Meanwhile every backward factor $1+w^2\sigma'(z_i)$ is essentially $1$ once $z_i$ is deep in saturation (true from the second layer on) — only the first layer's factor, $1+9\sigma'(1.5)\approx2.342$, differs meaningfully from $1$. So the accumulated gradient is essentially just $(h_{\text{depth}}-1)\times2.342$:

   • Depth 5: $(14.95-1)\times2.342\approx32.7$
   • Depth 10: $(29.95-1)\times2.342\approx67.9$
   • Depth 20: $(59.95-1)\times2.342\approx138.3$
3. One structural change fixes it, not a different F

   The two networks share the same weight and nonlinearity — no new gate, no new parameter. What changes is purely structural: whether $h_i$ replaces $h_{i-1}$ or adds to it. That single addition, $+h_{i-1}$, is the entire fix — not some special property of the per-layer function itself.