A model pretrained on raw text is extremely good at one thing: predicting what word comes next in a document. Ask it "Explain gravity:" and it might just continue the pattern — writing "Explain friction:" next, the way a dictionary of definitions would. It was never trained to answer anything. What has to change to fix that?
no masking yet — every token votes, mean NLL = 1.0875 (this is pretraining's loss)
Slide the mask boundary across the same eight tokens. Everything left of the boundary is parenthesized — masked out, contributing nothing to the loss. Watch the loss change as the boundary moves, and notice that nothing about how the loss is computed ever changes — only which tokens are allowed to vote.
Instruction tuning (SFT, short for Supervised Fine-Tuning) uses the exact same next-token cross-entropy loss as pretraining, with one change: a binary mask that zeroes out every prompt position, so only response tokens contribute.
- — the SFT loss for one (prompt, response) example.
- — the prompt tokens, masked out of the loss entirely.
- — the response tokens, the only ones that vote.
- — the -th response token; y_{<t} is every token (prompt and response) before it.
- p(y_t \mid x, y_{<t}) — the model's predicted probability of the true next token, given everything before it.
- Pretraining's loss masks nothing
Every position in the sequence, prompt included, contributes its own term to the mean — the model is graded on predicting the prompt too, since during pretraining there was no such thing as a "prompt" at all, just one long document.
- SFT's loss masks the prompt
The architecture, the optimizer, the loss function itself — none of it changes. Only the mask does: prompt positions still get fed forward through the model (they're needed as context), but their prediction error no longer counts.
- What the model actually learns from this
Since only response tokens are graded, the model is pushed toward producing response-shaped text after a prompt — the behavior of answering, not just continuing whatever pattern the text happened to be in.
same prompt, but the untuned base model just continues the "Explain X:" document pattern it saw during pretraining — it never learned to stop and answer
Same three-token prompt, same underlying model architecture. The base model falls back on the dictionary-entry pattern it saw constantly during pretraining. The SFT-tuned model — trained with exactly the masked loss above, on examples that look like this one — actually answers. Nothing about the model's capacity changed; only what it was graded on.
The toy sequence Explain gravity : It pulls masses together . — a 3-token prompt followed by a
5-token response — with a hand-picked per-token NLL of
[2.1, 1.8, 0.3, 1.2, 0.9, 1.5, 0.7, 0.2]:
- Pretraining's loss: every token votes
Mean of all 8 values: .
- SFT's loss: only the response's 5 tokens vote
Mean of just the last 5 values: — lower than pretraining's loss on this example, because the two high-NLL prompt tokens ( and ) no longer drag the average up.
- The boundary has to land exactly on the prompt
Move the mask one token short (boundary at index instead of ) and the last prompt token sneaks back into the average: — a different number, for a genuinely different (and wrong) training signal.
Drag the mask boundary until the masked loss lands on 0.9 — SFT's actual loss on this example, when the mask covers the prompt and nothing more.
masked loss = 1.0875
Instruction tuning isn't a new loss function or a new architecture — it's pretraining's own cross-entropy loss, with a mask deciding which tokens are allowed to vote. That's the entire mechanism that turns a text-completer into something that follows instructions at all. Everything that comes after in this part — preference optimization, RLVR, reasoning — assumes this step already happened: a model that at least attempts to answer, rather than continue.