Layer normalization & Group normalization

Same idea as batch norm — force a healthy mean and variance — but measured across a completely different axis: not across examples in a batch, but across this one example's own channels. No batch required, no matter how many examples you're processing at once.

Split a layer's output channels into groups, then normalize each group independently using only that group's own statistics:

• $h_i$ — channel $i$'s raw output, for this one example.
• $\mu_g$, $\sigma_g^2$ — the mean and variance of only the channels in group $g$, for this one example.
• $\hat h_i$ — channel $i$ after normalization.

1. One group (LayerNorm)

   Every channel lands in the same group — $\mu_g$ and $\sigma_g^2$ are the mean and variance across all of this example's channels at once.

2. Several groups (GroupNorm)

   Channels split into smaller groups, each normalized against only its own members — a middle ground between one giant group and every channel alone.

3. One channel per group (InstanceNorm, the degenerate limit)

   Each group has exactly one member, so that member's "mean" is itself — every value normalizes to exactly $0$, which is why this extreme is rarely useful on its own.

Slide from one group up to six. The channels get shuffled into smaller and smaller clusters, each computing its own mean and variance — completely independent of whatever else is in the batch, or whether there's a batch at all.

1. Six raw activations

   $[2, -1, 0.5, 3, -2.5, 1]$: mean $=(2-1+0.5+3-2.5+1)/6=3/6=0.5$. Squared deviations from that mean:

   • $1.5^2=2.25$
   • $(-1.5)^2=2.25$
   • $0^2=0$
   • $2.5^2=6.25$
   • $(-3)^2=9$
   • $0.5^2=0.25$

   These sum to $20$, so variance $=20/6\approx3.33$.

2. One group (LayerNorm)

   Dividing by $\sigma=\sqrt{3.33}\approx1.826$ after subtracting the mean $0.5$ from each value:

   • $(2-0.5)/1.826\approx0.82$
   • $(-1-0.5)/1.826\approx-0.82$
   • $(0.5-0.5)/1.826=0$
   • $(3-0.5)/1.826\approx1.37$
   • $(-2.5-0.5)/1.826\approx-1.64$
   • $(1-0.5)/1.826\approx0.27$

   Averaging these six gives mean $\approx0$ and variance $\approx1$ for the whole vector — the same guarantee batch norm gives, just measured across channels instead of across a batch.

3. Three groups of two

   Splitting into consecutive pairs $(2,-1)$, $(0.5,3)$, $(-2.5,1)$ — each pair's two values happen to sit equally far from their own mean:

   • $(2,-1)$: mean $0.5$, half-spread $1.5$, normalizing to $1.5/1.5=1$ and $-1.5/1.5=-1$
   • $(0.5,3)$: mean $1.75$, half-spread $1.25$, normalizing to $-1$ and $1$
   • $(-2.5,1)$: mean $-0.75$, half-spread $1.75$, normalizing to $-1$ and $1$

   Every one of the six outputs lands at exactly $+1$ or $-1$, regardless of how spread out the original six numbers were.