This part has three chapters left to teach before it's done: instruction tuning, preference optimization, and RL against verifiable rewards. It's tempting to treat them as three competing ideas. They're not — they're three stages of one pipeline, each one only possible because the previous one already ran.
Step forward through the pipeline, one stage at a time, on one held-out benchmark. Nothing here jumps straight from a raw pretrained model to a fully aligned reasoning model — every stage takes whatever the previous stage produced as its starting point.
Each stage's output accuracy is whatever the previous stage achieved, plus that stage's own contribution:
- — benchmark accuracy after stage has run.
- — the previous stage's output; for stage (SFT), this is the raw pretrained model's accuracy.
- — stage 's own marginal contribution, on top of whatever it started from.
- Stage 1 — SFT teaches the model to attempt an answer at all
Chapter 2's masked cross-entropy loss turns a text-completer into something that at least tries to follow an instruction.
- Stage 2 — preference optimization picks the better of the attempts it can already make
RLHF or DPO (Chapters 3-4, and the KTO/SimPO variants) refine which SFT-style answer the model prefers to produce — no new capability, just better taste among outputs it could already generate.
- Stage 3 — RLVR pushes past what any human preference alone could teach
On tasks with a checkable ground truth, RLVR (this part's next chapters) optimizes directly against verifiable correctness — a signal preference data never had access to.
Three bars, three completely different mechanisms, three different-sized contributions. SFT does the heaviest lifting here — going from "can't follow instructions at all" to "follows them passably" is a bigger jump than any later polish. That doesn't make the later stages optional: each is solving a problem the previous one structurally can't.
A pretrained base model scoring on some held-out benchmark, run through all three stages:
- After SFT
Accuracy rises to — a gain of , the largest single jump in the pipeline.
- After preference optimization
Accuracy rises to — a gain of .
- After RLVR
Accuracy rises to — a gain of .
- Total improvement
, exactly the sum of the three individual gains: . Every stage's contribution is additive and separately measurable — none of them do each other's job.
Find the stage whose own marginal accuracy gain is the largest of the three.
"Post-training" isn't one algorithm competing with the others for the title of best technique — it's a pipeline where SFT, preference optimization, and RLVR each solve a problem structurally out of reach for the others: attempting an answer at all, picking the better of two attempts, and optimizing against ground truth where one exists. The rest of this part builds each stage's machinery in turn, starting with why a learned reward model — not yet a verifiable one — was the first way anyone taught a language model what "better" even means.