VQ-VAE & discrete latent codebooks

Switch images. The continuous "×" never sits exactly on a code — it gets replaced by whichever square is closest, every time. The decoder that comes after this never sees the "×"; it only ever sees one of the four fixed squares.

A VQ-VAE — short for Vector-Quantized Variational Autoencoder — keeps a small, learned codebook of $K$ vectors $\{e_1, \ldots, e_K\}$. The encoder still outputs a continuous $z_e(x)$, but before decoding it's replaced by its nearest codebook entry:

• $z_e(x)$ — the encoder's continuous output for input $x$, before quantization.
• $z_q(x)$ — the quantized latent actually passed to the decoder.
• $e_k$ — the codebook vector selected as the nearest match to $z_e(x)$.
• $e_j$ — one of the $K$ learned codebook vectors, indexed by $j$.
• $k$ — the index of the nearest codebook entry.

1. The lookup itself isn't learned

   That's the entire mechanism: nearest-neighbor lookup, nothing learned about the lookup itself.

2. What is learned: the codebook and its encoder/decoder

   The codebook's $K$ locations and the encoder/decoder around them are what's learned, so that whichever code a given input snaps to still decodes back into something close to the original — training has to shape the encoder's outputs to cluster near useful codes, not just quantize whatever it would have produced anyway.

Four different images, four different codes used — a discrete vocabulary, four entries deep, emerging entirely from where the encoder happens to place each image's continuous output. A fifth image mapping to code 0 again would mean the model judged it more similar to image 0 than to the other three.

Codebook at $(0,0)$, $(3,0)$, $(0,3)$, $(3,3)$; encoder output $z_e=(1,1)$:

1. Distance to every code

   • To $(0,0)$: $\sqrt{1^2+1^2}=\sqrt{2}\approx1.414$
   • To $(3,0)$: $\sqrt{2^2+1^2}=\sqrt{5}\approx2.236$
   • To $(0,3)$: $\sqrt{1^2+2^2}=\sqrt{5}\approx2.236$ — same as $(3,0)$
   • To $(3,3)$: $\sqrt{2^2+2^2}=\sqrt{8}\approx2.828$
2. Snap to the nearest

   $(0,0)$ is closest, so $z_q = (0,0)$ — the decoder receives exactly $(0,0)$, never $(1,1)$.

3. The gap that's thrown away

   The quantization error is $\sqrt{2}\approx1.414$ — everything about $z_e$ beyond "closest to code 0" is gone. A different image whose encoder output was $(0.1, 0.1)$ would round to the exact same code, indistinguishable to the decoder from this one.