Scaling laws

The same four training runs, fit two different ways. One method transforms the data before fitting a line; the other fits the raw numbers directly. Only one of them produces a sane prediction at a size fifty times bigger than anything actually trained.

Loss often follows a power law in model size: $L(N) = A \cdot N^{-\alpha}$. Taking logs turns this into something linear:

• $L$ (or $L(N)$) — the model's loss at a given size.
• $N$ — the model size (number of parameters).
• $A$ — a constant scale factor in the power law, fit from data.
• $\alpha$ — the scaling exponent: how fast loss falls as $N$ grows.

1. Log-log OLS recovers the law directly

   Fitting ordinary least squares to $(\log N, \log L)$ recovers $\alpha$ and $A$ directly — the exact same OLS machinery from Part IX, just applied after a log transform.

2. Fitting the raw numbers assumes the wrong shape

   Fitting a line to the raw $(N, L)$ pairs instead assumes a linear relationship that was never true, and extrapolates accordingly.

Four training runs at sizes 1, 4, 9, and 16 — and a fifth point at $N=100$, never trained, just computed from the true law for comparison. All five points sit exactly on one straight line in log-log space, because that line is the power law.

True law $L(N) = 100 \cdot N^{-0.5}$, observed at $N \in \{1, 4, 9, 16\}$ giving losses $\{100, 50, 33.\overline{3}, 25\}$:

1. Log-log OLS recovers the law exactly

   $(\log N, \log L)$ pairs: $(0, 4.605)$, $(1.386, 3.912)$, $(2.197, 3.507)$, $(2.773, 3.219)$.

   • Mean $\log N \approx 1.589$, mean $\log L \approx 3.811$
   • $\text{slope} = \dfrac{\sum(\log N-\overline{\log N})(\log L-\overline{\log L})}{\sum(\log N-\overline{\log N})^2} \approx \dfrac{-2.168}{4.337} \approx -0.500$
   • $\text{intercept} = \overline{\log L} - \text{slope}\times\overline{\log N} \approx 3.811-(-0.500)(1.589)\approx4.605\approx\ln(100)$

   Both match the true law ($\alpha=0.5$, $A=100$) to full floating-point precision, since the training data has no noise at all.

2. Extrapolating to N=100: almost exactly right

   $\log(100)\approx4.605$. Plugging into the fitted line: $4.605 + (-0.500)(4.605) \approx 2.303$. Exponentiating back: $e^{2.303}\approx10.000$ — matching the true value of exactly $10$, despite $N=100$ being far outside the training range of $1$ to $16$.

3. Fitting the raw numbers instead: a nonsensical answer

   Raw $(N, L)$ pairs: $(1,100)$, $(4,50)$, $(9,33.33)$, $(16,25)$.

   • Mean $N=7.5$, mean $L\approx52.08$
   • $\text{slope} \approx -562.5/129 \approx -4.360$
   • $\text{intercept} \approx 52.08-(-4.360)(7.5)\approx84.79$
   • At $N=100$: $84.79 + (-4.360)(100) \approx 84.79 - 436.0 \approx -351.26$

   A negative loss is meaningless; the raw-linear model was extrapolating a relationship that was never actually linear.