Variational Autoencoders (VAE)

The same input, encoded to the same distribution, decoded through a different noise draw each time. One of those draws reproduces the input exactly — the rest land nearby, at points the network never directly trained on.

A variational autoencoder encodes $x$ to a mean $\mu$ and log-variance $\log\sigma^2$ instead of a single point, then samples via the reparameterization trick:

• $z$ — the sampled latent vector, passed to the decoder.
• $\mu$ — the mean of the encoded distribution, output by the encoder.
• $\sigma$ — the standard deviation of the encoded distribution (derived from $\log\sigma^2$).
• $\varepsilon$ — an external noise term, drawn from a standard normal and independent of the network's parameters.

1. Sampling becomes a deterministic function of noise

   Writing sampling as a function of $\mu$, $\sigma$, and an external noise term $\varepsilon$ keeps the whole pipeline differentiable, instead of asking gradients to pass through a random draw directly.

2. Gradients still reach the encoder

   Gradients flow through $\mu$ and $\sigma$ even though $z$ itself is random — the randomness is isolated entirely in $\varepsilon$, which needs no gradient at all.

A second loss term, the KL divergence — short for Kullback–Leibler divergence — to the standard normal, keeps every input's distribution from collapsing to a single point or drifting off to some far corner of latent space:

• $D_{KL}\big(\mathcal{N}(\mu,\sigma^2)\,\|\,\mathcal{N}(0,1)\big)$ — the KL divergence between the encoded distribution and the standard normal target; the regularization term added to the reconstruction loss.

Four different noise draws, one shared input. The mean alone decodes back to exactly the original value — but every other draw lands at a distinct nearby point, none of them ever explicitly trained on. That spread is the entire point: the decoder has learned a neighborhood, not a single memorized answer.

Encoding $x=2$ to $\mu=1$, $\log\sigma^2=-1.8$ (so $\sigma\approx 0.4066$), with a fixed sequence of stand-in noise values $\varepsilon \in \{0, 1, -1, 2\}$:

1. No noise reproduces the input exactly

   $\varepsilon=0$ gives $z=\mu=1$, and decoding $z$ back through $\hat{x}=2z$ gives exactly $2$ — the original input, recovered perfectly when there's no randomness at all.

2. Every other draw lands somewhere new

   • $\varepsilon=1$: $z=1+0.4066(1)\approx1.4066\Rightarrow\hat x=2z\approx2.8131$
   • $\varepsilon=-1$: $z=1+0.4066(-1)\approx0.5934\Rightarrow\hat x\approx1.1869$
   • $\varepsilon=2$: $z=1+0.4066(2)\approx1.8131\Rightarrow\hat x\approx3.6263$

   Four genuinely different outputs from one input, one encoder, one decoder.

3. KL divergence penalizes distributions that stray from standard normal

   Plugging into $D_{KL}=\tfrac12(\sigma^2+\mu^2-1-\log\sigma^2)$:

   • Encoding $x=2$: $\mu=1$, $\sigma^2=e^{-1.8}\approx0.1653$, so $D_{KL}=\tfrac12(0.1653+1-1-(-1.8))=\tfrac12(1.9653)\approx0.983$.
   • Encoding $x=0$ instead: $\mu=0$, $\log\sigma^2=-2$, $\sigma^2=e^{-2}\approx0.1353$, so $D_{KL}=\tfrac12(0.1353+0-1-(-2))=\tfrac12(1.1353)\approx0.568$.

   The second is a smaller KL divergence, closer to $\mu{=}0,\sigma{=}1$, the one distribution where this penalty vanishes entirely.