Stochastic & mini-batch gradient descent

This is the exact same landscape and step button from the Part I capstone — except now the bowl is a real loss surface, fit to five data points, and each step uses every one of them to compute its direction. Watch how smoothly it heads for the minimum.

That's batch gradient descent — the gradient is the average over every example:

• $\nabla L_{\text{batch}}$ — the batch gradient: the direction used for one step, averaged over the whole dataset.
• $n$ — the total number of training examples.
• $\nabla L_i$ — the gradient of the loss computed from a single example $i$.
• $i$ — the index of one training example.

1. Stochastic gradient descent

   Uses just one example per step, $\nabla L_i$ for a single $i$ — cheap, but noisy.

2. Mini-batch gradient descent

   Splits the difference, averaging over a small subset $B$: $\frac{1}{|B|}\sum_{i \in B} \nabla L_i$.

3. All three trade accuracy for cost

   Batch, stochastic, and mini-batch are all estimates of the same true gradient — they just trade accuracy per step for cost per step.

Switch modes and take a few steps each time. Batch heads straight for the minimum. Stochastic zigzags — each step only "knows about" one point, so it's regularly a little wrong — while mini-batch sits somewhere in between.

At $(w,b)=(0,0)$:

1. Compute one point's stochastic gradient

   For squared-error loss, each point's gradient is $(\partial L_i/\partial w, \partial L_i/\partial b) = (-2x_i r_i,\ -2r_i)$, where $r_i=y_i-(wx_i+b)$ is that point's residual. At $(w,b)=(0,0)$, the point $(-2,-3.2)$ has residual $r=-3.2-0=-3.2$, giving:

   $$(-2(-2)(-3.2),\ -2(-3.2)) = (-12.8,\ 6.4)$$
2. Compare it to the full batch gradient

   At the same $(w,b)=(0,0)$, every residual equals $y_i$ (the prediction is $0$ everywhere). Each point's $(-2x_iy_i,\ -2y_i)$ pair:

   • $x=-2,y=-3.2$: $(-12.8,\ 6.4)$
   • $x=-1,y=-0.8$: $(-1.6,\ 1.6)$
   • $x=0,y=1.3$: $(0,\ -2.6)$
   • $x=1,y=3.1$: $(-6.2,\ -6.2)$
   • $x=2,y=4.7$: $(-18.8,\ -9.4)$

   Averaging each column: $w$-gradient $=(-12.8-1.6+0-6.2-18.8)/5=-7.88$, $b$-gradient $=(6.4+1.6-2.6-6.2-9.4)/5=-2.04$ — same rough neighborhood as the single point above, but a different $w$-magnitude and even a flipped sign on $b$.

3. Average over a full pass

   The individual gradients exactly reconstruct the batch gradient — noise that cancels out over time, not bias.