Text-conditioned diffusion

The exact same starting noise, run through the exact same reverse-diffusion formula from the earlier chapter — the only thing that changes between runs is which caption's predicted-noise sequence gets used at each step.

Nothing about the reverse step changes:

• $x_t$ — the noisy value at diffusion step $t$.
• $x_{t-1}$ — the slightly-less-noisy value this reverse step produces.
• $t$ — the current diffusion timestep.
• $\alpha_t$, $\beta_t$ — the same noise-schedule coefficients from the earlier diffusion chapter, controlling how much signal versus noise remains at step $t$.
• $\hat\epsilon(x_t, t, \mathbf{c})$ — the denoiser's predicted noise, now taking the caption embedding $\mathbf{c}$ as an extra input.
• $\mathbf{c}$ — the text embedding of the caption being generated toward — the one new ingredient this chapter adds.

1. One new input: the caption embedding

   The only addition is $\mathbf{c}$ — a text embedding fed into the denoiser alongside $x_t$ and $t$. Nothing else about the reverse step's math changes.

2. Same state, different prediction, different image

   A denoiser trained on captioned images learns to predict different noise, at the same $(x_t, t)$, depending on $\mathbf{c}$ — which means the same starting noise unrolls into a different final image for every different caption.

One shared starting point, three trajectories. The unconditional path (no caption at all) takes a middle course; "a cat" and "a dog" each pull the same starting noise toward a different final value, in opposite directions.

Shared starting noise $x_T = 2.0$, same noise schedule as the earlier chapter, three different predicted-noise sequences:

1. No conditioning: a generic middle-ground result

   Predicting zero noise at every step reduces the reverse formula to just $x_{t-1}=x_t/\sqrt{\alpha_t}$. Unrolling from $x_4=2.0$ with $\alpha=[0.9,0.8,0.7,0.6]$, from $t=3$ down to $t=0$:

   1. $x_3 = 2.0/\sqrt{0.6} \approx 2.582$
   2. $x_2 = 2.582/\sqrt{0.7} \approx 3.086$
   3. $x_1 = 3.086/\sqrt{0.8} \approx 3.450$
   4. $x_0 = 3.450/\sqrt{0.9} \approx 3.637$

   A plain, unconditioned denoising.

2. 'A cat': the same starting noise, steered higher

   Swapping in the cat-conditioned noise sequence $[-1.2, 0.8, -0.5, 1]$ (read at each step from $t{=}3$ down to $t{=}0$) — otherwise identical math. Just the first step already diverges from the unconditional case: $x_3 = (2.0-\sqrt{0.4}\times(-1.2))/\sqrt{0.6} \approx (2.0+0.759)/0.775 \approx3.562$, compared to $2.582$ with no conditioning. Carrying the same three remaining steps forward reverses the same $x_4=2.0$ to $\approx4.330$ instead.

3. 'A dog': the same starting noise, steered lower

   The dog-conditioned sequence is $[0.7, -0.3, 0.9, -0.6]$. Its first step: $x_3 = (2.0-\sqrt{0.4}\times0.7)/\sqrt{0.6} \approx (2.0-0.443)/0.775 \approx2.010$ — barely moved from $2.0$, unlike the cat case's big jump. Carrying the same three remaining steps forward pulls that same $x_4=2.0$ down to $\approx2.789$ — on the opposite side of the unconditional baseline from the cat result.

Same schedule, same formula, same starting noise. Three different captions, three different destinations.