Contrastive learning & CLIP

Before training, every image and every caption embed to the same point — there's no signal to tell them apart, so the model can only guess uniformly among the three captions for any image. After training (Chapter 1's exact placements), the probability matrix sharpens hard onto the diagonal: each image's own caption dominates.

Score every image against every caption with a similarity function, turn each image's row of scores into a probability distribution over captions with softmax, and push that distribution toward the true pairing — exactly Part II's cross-entropy loss, just computed in both directions at once:

• $\mathcal{L}$ — the contrastive loss being minimized, averaged over both directions.
• $n$ — the number of image-caption pairs in the batch.
• $\text{image}_i$, $\text{caption}_i$ — the $i$-th image and its true, paired caption.
• $P(\text{caption}_i \mid \text{image}_i)$ — the probability the model assigns to the true caption given the image (and symmetrically for the other direction).

1. No hand-written labels anywhere

   The "true pairing" is just whichever image and caption arrived together in the training data — nobody annotated what's in either one.

2. Negatives come for free

   Every other image in the same batch automatically becomes a negative example for every other caption, with no extra labeling effort.

Compare raw similarity scores to what softmax turns them into. The raw numbers are just negative distances — useful, but not yet a probability. Softmax is what turns "closer" into "more likely," and it's exactly the same softmax from Part II's cross-entropy chapter and Part IV's attention chapter, doing the same job a third time.

Three images, three captions, correctly paired:

1. Before training

   Every embedding collapses to the same point, so every similarity score is $0$ and softmax gives exactly $\frac{1}{3}$ to each caption. The loss is $\ln 3 \approx 1.099$ — precisely the entropy of a uniform guess over three options, because that's genuinely all the model can do.

2. After training

   Using Chapter 1's placements, the dog image assigns its true caption a probability of about $0.876$ — more than ten times any rival caption's share.

3. The loss reflects that gap

   Averaged over both directions and all three pairs, the loss drops to about $0.112$ — roughly a tenth of where it started, without a single hand-written label anywhere in the process.

4. It falls smoothly, not in one jump

   Halfway through training (linearly interpolating every embedding between its start and end position), the loss sits at about $0.39$ — already down from $1.099$, but nowhere near $0.112$ yet. The curve is concave: most of the improvement happens in the second half of training, not the first.