Assemble a tiny multimodal assistant

Pick an audio clip. It retrieves its nearest caption in the shared embedding space — exactly Chapter 1's mechanism — and that caption's identity selects which noise-conditioning sequence steers the reverse-diffusion process — exactly Chapter 2's mechanism.

Two stages, two chapters, one pipeline:

• $\mathbf{c}$ — the text embedding of whichever caption audio retrieval selected, threaded into the denoiser exactly as in the diffusion chapter.
• $\hat\epsilon(x_t, t, \mathbf{c})$ — the same conditioned noise predictor from the diffusion chapter, now driven by the retrieved caption's embedding.
• $x_t$, $t$ — the noisy diffusion state and current timestep, unchanged from that chapter's process.

1. No new mechanism — just reused pieces

   Nothing here is a new mechanism — it's the exact retrieval rule from audio embeddings, feeding the exact conditioning slot from text-conditioned diffusion.

2. This is the general shape of multimodal systems

   A working multimodal system is very often just two or three single-modality mechanisms, connected so one's output becomes the next one's input.

Three audio clips, the same shared starting noise, the same reverse-diffusion formula — and three completely different generated values, because retrieval routed each one to a different conditioning sequence before generation ever started.

Three audio inputs, run through the full pipeline:

1. Dog barking routes to the dog caption and dog conditioning

   Retrieval: "Audio: a dog barking" $(1.9,3.2)$ to "Caption: a dog running" $(2.2,2.8)$ — $\sqrt{(1.9-2.2)^2+(3.2-2.8)^2}=\sqrt{0.09+0.16}=\sqrt{0.25}=0.5$. Generation: the dog-conditioned noise sequence $[0.7,-0.3,0.9,-0.6]$ reverses the shared $x_4=2.0$ (same mechanism as Chapter 2's first-step check, $x_3\approx2.010$) all the way to a final value $\approx2.789$.

2. Cat meowing routes to the cat caption and cat conditioning

   Retrieval: "Audio: a cat meowing" $(0.1,1.2)$ to "Caption: a cat sleeping" $(0.3,0.8)$ — $\sqrt{(0.1-0.3)^2+(1.2-0.8)^2}=\sqrt{0.04+0.16}=\sqrt{0.2}\approx0.45$. Generation: the cat-conditioned noise sequence $[-1.2,0.8,-0.5,1]$ reverses the same $x_4=2.0$ (first step $x_3\approx3.562$, per Chapter 2) to $\approx4.330$ — a different destination from the same starting point.

3. Bird chirping completes the set

   Retrieval: "Audio: a bird chirping" $(4.2,0.2)$ to "Caption: a bird flying" $(3.7,0.4)$ — $\sqrt{(4.2-3.7)^2+(0.2-0.4)^2}=\sqrt{0.25+0.04}=\sqrt{0.29}\approx0.54$. Generation: the bird-conditioned noise sequence $[-0.2,0.4,-0.9,0.5]$ (read $t{=}3$ down to $t{=}0$), first step $x_3=(2.0-\sqrt{0.4}\times(-0.2))/\sqrt{0.6}\approx2.746$, reverses all the way to $\approx3.866$ — a third distinct value, confirming all three retrieval-then-generation chains work independently and correctly.