Normalizing flows & invertible transformations

Drag $z$. The point traces out the density curve for $x=z^2$ — not because it's been fit to data, but because it's derived, exactly, from how much $f$ stretches space at that $z$.

A normalizing flow builds $p_X$ from a trivial base density $p_Z$ by pushing it through an invertible, differentiable function $x = f(z)$. In one dimension, the change-of-variables formula says exactly how the density transforms:

• $p_X(x)$ — the density of the generated data $x$, computed exactly rather than merely bounded or left undefined.
• $p_Z(z)$ — the trivial base density (e.g. uniform) that the flow starts from.
• $f$ — the invertible, differentiable function (the flow) that maps a base-space point $z$ to a data-space point $x$.
• $f^{-1}(x)$ — the flow's inverse, recovering the base-space point $z$ that produced a given $x$.
• $df/dz$ — the derivative of $f$ at $z$ (in higher dimensions, the Jacobian determinant), measuring how much the flow stretches or compresses space there.
• $x$, $z$ — a point in data space and its corresponding point in the simple base (latent) space.

1. Stretching lowers density

   Wherever $f$ stretches space ($|df/dz|>1$), the same probability mass spreads over a wider region, so density there goes down.

2. Compressing raises density

   Wherever $f$ compresses space ($|df/dz|<1$), density goes up — the same mass is packed into a narrower region.

3. Total probability is conserved

   Mass is only ever redistributed, never created or destroyed — which is exactly what makes the density exactly computable instead of merely bounded.

$Z\sim\text{Uniform}(0,1)$ starts perfectly flat. Squaring it compresses space near $z=0$ (small numbers get squeezed together even harder by squaring) and stretches space near $z=1$ — so density piles up near $x=0$ and thins out near $x=1$, with no information lost anywhere in between.

The flow $x = z^2$, evaluated at two points:

1. x = 0.25

   $z = f^{-1}(0.25) = \sqrt{0.25} = 0.5$. $df/dz = 2(0.5) = 1$ — no stretching or compressing at all here. $p_X(0.25) = 1/1 = 1$, identical to the base density.

2. x = 0.04

   $z = \sqrt{0.04} = 0.2$. $df/dz = 2(0.2) = 0.4$ — space is compressed to less than half its width here. $p_X(0.04) = 1/0.4 = 2.5$, two and a half times denser than the flat base.