Backpropagation, step by step

This is the loss of a real two-layer network — 2 inputs, 2 hidden neurons, 1 output — as a function of just one weight, $w_{11}$, holding everything else fixed. Drag it: the tangent line is backprop's answer for $dL/dw_{11}$, computed by chaining through both layers.

Backpropagation is the chain rule, applied backward from the loss, one layer at a time:

• $L$ — the loss, measuring how wrong the network's prediction is.
• $w_{11}$ — the weight connecting input 1 to hidden neuron 1, the one being differentiated.
• $y$ — the network's final output.
• $z_{\text{out}}$ — the output neuron's raw weighted sum, before its activation.
• $h_1$ — hidden neuron 1's activation, feeding into the output.
• $z_1$ — hidden neuron 1's raw weighted sum, before its activation.

1. The loss adds a new starting term

   Five factors total, but only two are genuinely new beyond Chapter 4: the loss contributes its own local derivative, $\partial L/\partial y$, at the very start of the chain.

2. One factor is just a connecting weight

   $\partial z_{\text{out}}/\partial h_1$ is nothing more than $v_1$ — the weight connecting $h_1$ to the output. Everything else in the five-factor product is the same local-slope idea from Chapter 4.

3. Every weight reuses the same shared terms

   Every other weight in the network gets its gradient the same way, reusing the exact same $\partial L/\partial z_{\text{out}}$ and $\partial L/\partial z_1$ terms computed once and shared.

Push $w_{11}$ negative and watch the loss keep falling — this particular network, on this particular input, never stops preferring a more negative $w_{11}$ within this range. The gradient still tells you the direction correctly at every point, even where it isn't near zero.

At $w_{11}=0.5$:

1. Forward pass

   $z_1 = 0.5(1) + (-0.3)(2) + 0.1 = 0$ exactly, so $h_1=\sigma(0)=0.5$. Forward through the rest:

   • $z_2 = -0.2(1)+0.4(2)+0.2 = 0.8$, so $h_2=\sigma(0.8)\approx0.690$
   • $z_{\text{out}} = 1.5(0.5)+(-1.0)(0.690)+0.3 \approx 0.360$
   • $y=\sigma(0.360)\approx0.589$
   • loss $=0.5(y-0)^2 \approx 0.173$
2. Backward into the output neuron

   $\partial L/\partial y = y = 0.589$ (target is $0$), times $\sigma'(z_{\text{out}})=y(1-y)\approx0.242$ gives $\partial L/\partial z_{\text{out}}\approx0.143$. That one number is shared by every weight feeding the output:

   • $\partial L/\partial v_1 = 0.143\cdot h_1 \approx 0.071$
   • $\partial L/\partial v_2 = 0.143\cdot h_2\approx0.098$
   • $\partial L/\partial c \approx0.143$
3. Push further back, into the hidden neurons

   For hidden neuron 1: $\partial L/\partial h_1 = 0.143\cdot v_1\approx0.214$, times $\sigma'(z_1)=h_1(1-h_1)=0.25$ gives $\partial L/\partial z_1\approx0.053$ — shared by every weight feeding $h_1$:

   • $\partial L/\partial w_{11}\approx0.053(1)=0.053$
   • $\partial L/\partial w_{12}\approx0.053(2)=0.107$
   • $\partial L/\partial b_1\approx0.053$

   The exact same mechanism, run on neuron 2: $\partial L/\partial h_2 = 0.143\cdot v_2\approx-0.143$, times $\sigma'(z_2)=h_2(1-h_2)\approx0.214$ gives $\partial L/\partial z_2\approx-0.030$ — shared by every weight feeding $h_2$:

   • $\partial L/\partial w_{21}\approx-0.030(1)=-0.030$
   • $\partial L/\partial w_{22}\approx-0.030(2)=-0.061$
   • $\partial L/\partial b_2\approx-0.030$
4. Tally the result

   All nine gradients, from two shared numbers ($\partial L/\partial z_{\text{out}}$, and per hidden neuron $\partial L/\partial z_1$ or $\partial L/\partial z_2$) and the inputs each weight multiplies.