Batch normalization

Same broken scale from Chapter 8 — the faint line still vanishes exactly as before. The bold line is the same network with one thing added after every layer: batch normalization. It barely moves.

Batch normalization re-centers and rescales a layer's output across a batch of examples, at every layer, regardless of what produced it:

• $\hat h_j$ — neuron $j$'s output after normalization, mean $0$ and variance $1$ by construction.
• $h_j$ — neuron $j$'s raw output, before normalization.
• $\mu_j$ — the mean of neuron $j$'s output, measured across the current batch.
• $\sigma_j^2$ — the variance of neuron $j$'s output, measured across the current batch.
• $\epsilon$ — a tiny constant that keeps the division from blowing up when the variance is near zero.

1. Mean 0, variance 1 — by construction

   $\mu_j$ and $\sigma_j^2$ are simply the mean and variance of neuron $j$'s output, measured across the current batch. Plug them in and every neuron comes out with mean $0$ and variance $1$ — by construction, not by hoping the weight scale happened to be right.

Drag the scale anywhere — tiny, huge, exactly ideal. The faint line swings across nine orders of magnitude. The bold one barely leaves $1$. Batch norm doesn't care what scale produced the input; it forces the output to a fixed scale every time.

1. Without batch norm, at scale 0.05

   Six layers shrink the RMS (short for Root Mean Square) activation from about $1$ down to under $10^{-5}$ — dead, exactly as in Chapter 8.

2. With batch norm, at every scale tried

   Insert batch normalization after each layer instead, and every single layer's RMS activation comes out within about $0.5\%$ of exactly $1$ — from $0.05$ all the way to $1.0$.

3. Compare the spread

   The five-scale spread in final-layer RMS with batch norm is under $0.05$, compared to nine orders of magnitude without it.