Joint multimodal embedding spaces

Hook

Part IV's word embeddings put every word in one space, so distance meant "similar meaning." What if a photograph and a sentence describing it could share that exact same space — so a picture and a caption become directly comparable, the same way two words are?

Intuition

nearest cross-modal match: "Caption: a dog running" — distance 0.28

Squares are images, circles are captions, and both live in the same 2D space. Click any point — its nearest neighbor is always the other modality's matching item, never another image or another caption. The dog photo sits closest to "a dog running," not to the cat or bird photos.

Formalize

With an image embedding $u$ and a text embedding $v$ in the same space, "how well does this caption describe this image" is just ordinary distance — exactly Part IV's word-embedding formula, now spanning two different kinds of input:

d(u, v) = \sqrt{(u_1-v_1)^2 + (u_2-v_2)^2 + \dots}

$u$ — the image embedding, produced by an encoder built for pixels.
$v$ — the text embedding, produced by a separate encoder built for text.
$d(u, v)$ — the distance between the two embeddings: how well the caption matches the image.

The formula itself didn't change
Nothing about the distance formula is new — it's exactly Part IV's word-embedding distance, applied unchanged.
What changed is where u and v come from
$u$ comes from a completely different encoder (one built for pixels) than $v$ (one built for text) — two separate networks, not one.
Both encoders share one coordinate system
They're trained so their outputs land in the same coordinate system, not just two similar-looking but separate spaces — that's what makes comparing them by distance meaningful at all.

Play

ranked matches for "Caption: a bird flying": Image: bird (0.50), Image: dog (3.11), Image: cat (3.75)

Click through every image and caption. The nearest-match pairing holds in both directions — an image's nearest caption is its true one, and a caption's nearest image is its true one. That symmetry is exactly what makes this space useful for search in either direction: type text, find images; or show an image, find matching text.

Worked example

The dog photo sits at $(2, 3)$ ; here's its distance to all three captions:

Distance to its true caption
"A dog running" sits at $(2.2, 2.8)$ : $d = \sqrt{0.2^2+0.2^2} \approx 0.28$ — very close.
Distance to the other two captions
- "A cat sleeping" is $\approx 2.78$ away.
- "A bird flying" is $\approx 3.11$ away.
Both roughly ten times farther than the true caption.
The nearest-neighbor search is unambiguous
There's no contest: the true caption wins by a wide margin, exactly the same way "prince" clearly beat "man" or "queen" as king's nearest neighbor back in Part IV.

Checkpoint

Click the image whose embedding sits nearest to the caption “a cat sleeping.”

click an image

Click an image to try it

Summary

d(u, v) = \sqrt{(u_1-v_1)^2 + (u_2-v_2)^2 + \dots}

A joint embedding space turns "does this caption match this image" into the same nearest-neighbor question Part IV already solved for words — the only real difference is that the two points being compared came from entirely different encoders. This chapter hand-placed the coordinates to make the geometry obvious, exactly like Part IV's word space did. The next chapter asks the much harder question: how does a real model learn to place them there itself, with no human ever labeling which caption matches which image?