Part IX — Deep Learning Regularization, Normalization & Training Dynamics · Chapter 9

Mixed precision training (FP16 & BF16)

Hook

Half-precision floats train faster and use half the memory. They also can't represent a gradient like 0.000050.00005 at all — it just rounds down to zero. So how does anyone train in FP16 without every small gradient silently vanishing?

Intuition

This gradient is real — it's genuinely pushing a weight in a direction — but it's too small for the low-precision format to store. Cast it directly and it becomes exactly 00: not "small," gone. A gradient of 00 means that weight gets no update at all.

Formalize

Loss scaling multiplies the loss (and hence every gradient) by a constant before the backward pass, then divides the optimizer's update by the same constant afterward:

gused=castfp16(s⋅g)sg_{\text{used}} = \frac{\text{cast}_{\text{fp16}}(s \cdot g)}{s}
  • gg — the true, full-precision gradient.
  • ss — the loss scale, a constant chosen large enough to lift small gradients above FP16's representable range.
  • castfp16\text{cast}_{\text{fp16}} — rounding to low precision, including flushing anything still too small to zero.
  • gusedg_{\text{used}} — the gradient actually applied to the weight, after undoing the scale.
  1. Scale up before storage

    s⋅gs\cdot g is large enough to survive being rounded into FP16, even though gg alone wasn't.

  2. Scale back down before the update

    Dividing by ss afterward recovers a value extremely close to the original gg — the update itself still happens in full precision, only the storage was ever low-precision.

Play

Drag the scale slider up from 1×1\times. Below a threshold, the recovered gradient is a flat 00 — the update is silently skipped. Cross it, and the true gradient comes back essentially exactly, every time, for every larger scale you try afterward.

Worked example
  1. Without loss scaling

    A true gradient of 5×10−55\times10^{-5} is smaller than the toy format's representable floor. Cast directly, it stores as exactly 00 — that parameter stops learning for this step, and nothing downstream indicates why.

  2. With loss scaling at 2×

    Scaled up to 1×10−41\times10^{-4}, the value survives casting. Divided back down by 22, it comes back out as 5×10−55\times10^{-5} — the exact true gradient, recovered.

  3. A second problem: the weight itself

    Loss scaling protects gradients in flight, but the weight being updated has its own precision problem. A weight of 1.2345671.234567 stored directly in the toy low-precision format rounds to 1.231.23, and a tiny update of −0.0001-0.0001 added to 1.231.23 still rounds right back to 1.231.23. Applying that same update for 10 steps down two different paths:

    • fp16-only: every update rounds away, so the stored weight never moves from 1.231.23
    • full-precision master copy: the same 10 updates accumulate in full precision, reaching 1.234567−10(0.0001)=1.2335671.234567-10(0.0001)=1.233567

    A real, accumulated difference of about 0.00360.0036 between the two paths.

Checkpoint

Raise the loss scale until the recovered gradient matches the true gradient (5e-5) instead of underflowing to zero.

Move the scale slider to try it
Summary
gused=castfp16(s⋅g)sg_{\text{used}} = \frac{\text{cast}_{\text{fp16}}(s \cdot g)}{s}

Mixed precision training isn't "just use smaller floats" — it's two separate fixes stacked together. Loss scaling keeps small-but-real gradients from being flushed to zero on the way through the backward pass. A full-precision master copy of the weights keeps small-but-real updates from vanishing on contact once they're applied. Drop either fix and FP16's speed and memory advantages come with silent, hard-to-diagnose training stalls. That's the last individual fix this part introduces — the closing chapter puts it back to back with normalization, warmup, and decoupled decay, all at once, on a single network that won't train without them.