Speculative decoding & Medusa heads

Still-remaining target: the cat sat on the mat and slept

Draft model guesses: the, cat, dog

Big model verifies in one pass: accepts 2 of 3, plus one bonus token — emits the cat sat

Step through one speculative-decoding session, round by round. A small draft model proposes a few tokens; the big model verifies all of them in a single forward pass, accepting a matching prefix and always contributing one more correct token for free.

Given the tokens still remaining and the draft model's guesses for this round, the big model accepts the longest matching prefix, then adds one bonus token — its own correct next prediction, which came out of the same verification pass regardless of whether every guess was right:

• $\text{guess}_i$ — the draft model's $i$-th proposed token this round.
• $\text{target}_i$ — the token the big model would actually generate at that position.
• $\text{accepted}$ — how many leading guesses matched, before the first mismatch (or the end of the guesses).

1. One big-model call verifies the whole round

   Checking whether several guesses match doesn't cost more big-model calls than checking whether one does — the whole draft sequence is scored in one parallel forward pass.

2. Throughput is tokens emitted per call, not per token

   Standard decoding is exactly 1 token/call — that's the floor. Any round that accepts even one guess, plus its bonus, already beats it.

Across the whole session, tokens emitted divided by big-model calls made — comfortably above the autoregressive baseline of 1, purely because most rounds get more than zero guesses right.

1. Round 1: guesses [the, cat, dog] against [the, cat, sat, ...]

   "the" and "cat" match, "dog" doesn't match "sat" — accepted = 2. Target continues past that, so one bonus token: emits [the, cat, sat], 3 tokens.

2. Round 2: guesses [on, the, moon] against [on, the, mat, ...]

   Same shape: 2 accepted, 1 bonus, 3 tokens emitted — [on, the, mat].

3. Round 3: guesses [and, slept, forever] against [and, slept] (target ends here)

   Both guesses match, but the target is exhausted after them — no bonus token available. 2 accepted, 2 emitted: [and, slept]. Session total: 8 tokens in 3 calls, $8/3 \approx 2.67$ tokens/call.