Interleaved multimodal generation

step 1/10: the model picked “a” — a text token — out of 3 candidates.

Step through generation one token at a time. The sequence starts as ordinary words — "a," "photo," "of" — then, with no special mode switch, starts producing image-patch tokens instead. After all four patches appear, it drops right back into words: "on," "the," "mat." One stream, one mechanism, the whole way through.

Every step, text or image, runs through the identical decision rule — Part IV's softmax, over whatever candidates happen to be on the table this step:

• $\ell$ — this step's candidate logits, one number per candidate token.
• $w_i$ — the softmax weight on candidate $i$, after normalizing the logits into a distribution.
• $\text{next}$ — the token actually emitted: whichever candidate has the highest weight.

1. The candidates can be anything

   A step's candidate set might be all words, all image tokens, or — as at the handoff points — a mix of both.

2. Softmax and argmax don't know or care

   The formula never looks at whether a candidate is a word or a patch; it only looks at its logit.

3. Interleaving falls out for free

   If an image-token candidate happens to have the highest logit at some step, it gets emitted — right in the middle of a sentence, with no separate "switch to image mode" instruction anywhere.

Step forward past the third word. Watch the bar chart: the winning bar is always dramatically taller than the other two, whether the winner is a word or an image token — softmax doesn't get any less confident just because this candidate happens to be a patch. Once an image token wins, its embedding is printed too: the exact same numbers Chapter 5's projector produced for that patch, carried into this sequence unchanged.

Step 4, right at the text-to-image handoff. Candidates: "top-left" (image, patch 0) with logit $2$, "of" with logit $-1$, "the" with logit $-1$:

1. Exponentiate each logit

   • "top-left": $e^{2}\approx 7.389$
   • "of": $e^{-1}\approx 0.368$
   • "the": $e^{-1}\approx 0.368$
2. Normalize into weights

   Sum $\approx 7.389+0.368+0.368=8.125$. Dividing each by that sum:

   • "top-left": $7.389/8.125\approx0.909$
   • "of": $0.368/8.125\approx0.045$
   • "the": $0.368/8.125\approx0.045$

   "top-left" wins by a wide margin, the same kind of margin every earlier all-text step also produced.

3. Emit and carry the embedding forward

   The emitted token is the image patch "top-left," carrying embedding $(3, 7)$ — Chapter 5's own number for that exact patch, now just the fourth entry in this sequence.