Autoencoders & latent space representations

Hook

Every architecture so far has taken an input and produced a different kind of output — a class label, a next character, a translation. An autoencoder does something stranger: it tries to reproduce its own input, after being forced through a narrower pipe than the input itself.

Intuition

bottleneck direction — mean squared reconstruction error = 1.753

These are the same seven points from Part II's PCA — short for Principal Component Analysis — chapter. The encoder compresses each 2D point down to a single number — its position along one direction. The decoder expands that one number back out to 2D, by placing it back on that same line. Drag the direction: every point snaps onto whatever line you've chosen, and the gap between a point and its snapped-back position is exactly what got lost by compressing.

Formalize

With a 1D bottleneck and linear encoder/decoder, encoding is a projection and decoding places the result back on the line:

z = x \cdot w, \qquad \hat{x} = z\, w, \qquad \mathcal{L} = \frac{1}{n}\sum_i \lVert x_i - \hat{x}_i \rVert^2

$x$ — one input data point (2D here).
$w$ — the unit direction defining the bottleneck: the line the encoder projects onto and the decoder places points back on.
$z$ — the encoded value: $x$ 's position along $w$ , the single-number bottleneck.
$\hat{x}$ — the reconstruction: $z$ placed back on the line, the autoencoder's best guess at $x$ .
$\mathcal{L}$ — the reconstruction loss: mean squared distance between each point and its reconstruction.
$n$ — the number of data points being averaged over.

Reconstruction error is maximizing variance in disguise
For a unit direction $w$ , because $\hat{x}$ is the closest point on the line to $x$ , Pythagoras gives $\lVert x \rVert^2 = \lVert \hat{x} \rVert^2 + \lVert x - \hat{x} \rVert^2$ — so minimizing reconstruction error is exactly the same problem as maximizing $\lVert \hat{x} \rVert^2$ , the projected variance from Part II.
A linear autoencoder is PCA
A linear autoencoder with one bottleneck dimension is PCA.

Play

angle = 1.20 rad — reconstruction error = 0.820

Drag along this curve — it's reconstruction error as a function of the bottleneck direction's angle. There's exactly one minimum, and it's a real critical point: the tangent is flat right at the bottom, not just low. Autoencoders only get interesting when the encoder and decoder stop being linear — then this single smooth bowl can become a much more complicated surface, capable of capturing bends and curves a straight line never could.

Worked example

At the true PCA direction (0.6, 0.8)
Reconstruction error is the mean of each point's squared distance to its own projection — exactly the variance left over in the perpendicular direction $(-0.8, 0.6)$ . Projecting the seven points onto that perpendicular direction (dot product $-0.8x+0.6y$ ) gives $1,-1,0,0,0,-1,1$ (mean $0$ ), so the mean squared error is $(1^2+1^2+0^2+0^2+0^2+1^2+1^2)/7 = 4/7 \approx 0.571$ — and every other direction does strictly worse.
Rotate away
Even a small rotation away from $(0.6, 0.8)$ makes the error rise immediately, because you're no longer capturing the one direction these seven points actually vary along.

Checkpoint

Drag the bottleneck direction until the reconstruction error drops to within 0.05 of the best possible value, 0.571.

reconstruction error = 2.766

Drag the direction to try it

Summary

\mathcal{L} = \frac{1}{n}\sum_i \lVert x_i - \hat{x}_i \rVert^2

A linear, 1D-bottleneck autoencoder rediscovers PCA's principal direction purely by trying to minimize reconstruction error — nobody told it to maximize variance. Swap the linear encoder/decoder for the nonlinear networks built throughout this course, and the same idea compresses far richer data — images, audio, whatever — into a bottleneck that still tries to keep only what's needed to reconstruct. The next two chapters ask what happens when, instead of reconstructing real data, a network tries to generate new data that never existed.