RMSNorm

Same six activations as the last chapter — mean $0.5$, not $0$. LayerNorm would force the output's mean back to $0$. RMSNorm doesn't: it never subtracts a mean at all, so whatever bias was in the input mean survives into the output, just rescaled.

RMSNorm divides by the root-mean-square of the vector — no mean subtraction anywhere in the formula:

• $h_i$ — channel $i$'s raw value.
• RMS($h$) — the root-mean-square of the whole vector: the square root of its average squared value.
• $g$ — a learned (here, shared-scalar) gain applied after normalizing.
• $\epsilon$ — a tiny constant keeping the division well-defined near zero.

1. LayerNorm: subtract the mean, then divide by the standard deviation

   $\hat h_i = (h_i-\mu)/\sigma$ — two statistics computed from the data, one subtraction, one division.

2. RMSNorm: skip the subtraction entirely

   $\hat h_i = h_i/\text{RMS}(h)$ — one statistic, one division. Whatever mean the input had is still there, just divided down along with everything else.

Drag the gain. Every one of the six channels scales up or down by exactly the same factor — RMSNorm never treats any channel differently based on where the group's center happens to sit, because it never computed a center.

1. RMS of the six activations

   Squaring each of $[2,-1,0.5,3,-2.5,1]$ and summing: $2^2+(-1)^2+0.5^2+3^2+(-2.5)^2+1^2 = 4+1+0.25+9+6.25+1 = 21.5$. Mean-of-squares $=21.5/6\approx3.583$, so $\text{RMS}=\sqrt{3.583}\approx1.893$.

2. RMSNorm's output mean isn't 0

   Dividing every value by $1.893$ divides the mean by $1.893$ too: $0.5/1.893\approx0.264$. LayerNorm on the identical input forces that same quantity to exactly $0$ by construction.

3. One fewer statistic, one fewer reduction

   RMSNorm needs only $\sum h_j^2$, not both $\sum h_j$ and $\sum(h_j-\mu)^2$ — cheaper to compute, and in practice competitive with LayerNorm's accuracy on large transformer models, which is why models like LLaMA and T5 use it instead.