Audio & speech embeddings (Whisper)

Squares are images, circles are captions, triangles are audio clips. Pick an audio clip and ask for its nearest image or nearest caption — the same distance-based retrieval from Part VI, just with a third kind of point added to the map.

A speech encoder maps a spoken clip into the same embedding space as an image encoder and a text encoder, trained so that a recording of "dog barking," the word "dog," and a photo of a dog all land near each other:

• $\text{Encode}_{\text{audio}}$ — the speech encoder, mapping a spoken clip into the shared embedding space.
• $\text{Encode}_{\text{image}}$ — the image encoder, mapping a photo into that same shared space.

1. Retrieval itself is unchanged

   Nothing about nearest-neighbor retrieval changes — it's the identical distance computation from the two-modality case, just applied to a third kind of point on the same map.

2. Any pair of modalities can now retrieve from each other

   What's new is that any pair among the three modalities can retrieve from each other:

   • audio→image
   • audio→text
   • image→audio
   • text→audio

   The shared space makes every direction equally available.

Click any item — image, caption, or audio clip — and every other modality's nearest match comes back correct. Three modalities, one shared space, the same simple distance rule doing all the work in every direction.

Three audio clips, each placed near its matching image and caption cluster:

1. Audio retrieves its image by a wide margin

   "Audio: a dog barking" sits at $(1.9,3.2)$, "Image: dog" at $(2,3)$: $\sqrt{(1.9-2)^2+(3.2-3)^2}=\sqrt{0.01+0.04}=\sqrt{0.05}\approx0.22$. The next-nearest image, "Image: cat" at $(0,1)$: $\sqrt{(1.9-0)^2+(3.2-1)^2}=\sqrt{3.61+4.84}=\sqrt{8.45}\approx2.91$ — over 10x farther away. No ambiguity.

2. The same clip also retrieves its caption

   That same audio clip $(1.9,3.2)$ against "Caption: a dog running" at $(2.2,2.8)$: $\sqrt{(1.9-2.2)^2+(3.2-2.8)^2}=\sqrt{0.09+0.16}=\sqrt{0.25}=0.5$ — closer to its own caption than to either other caption by a comparable margin.

3. Every one of the three clips behaves the same way

   The same distance computation for the other two clips:

   • "Audio: a cat meowing" $(0.1,1.2)$ → "Image: cat" $(0,1)$: $\sqrt{0.01+0.04}\approx0.22$; → "Caption: a cat sleeping" $(0.3,0.8)$: $\sqrt{0.04+0.16}\approx0.45$
   • "Audio: a bird chirping" $(4.2,0.2)$ → "Image: bird" $(4,0)$: $\sqrt{0.04+0.04}\approx0.28$; → "Caption: a bird flying" $(3.7,0.4)$: $\sqrt{0.25+0.04}\approx0.54$

   Both retrieve their own image and their own caption correctly, with the same kind of clear separation — the geometry generalizes cleanly to a third modality.