Score-based generative models

Click "Take a Langevin step" a few times. The point starts at $x=-5$ — nowhere near the data — and drifts steadily toward $x=3$, the single peak of this chapter's target distribution, one gradient step at a time.

The score of a distribution is $s(x) = \nabla_x \log p(x)$ — a vector field pointing toward higher density. For a Gaussian $\mathcal{N}(\mu,\sigma^2)$, it has a clean closed form:

• $s(x)$ — the score function: the gradient of the log-density, a vector field pointing toward higher-probability regions.
• $p(x)$ — the probability density of the target distribution.
• $\mu$ — the mean of the Gaussian target distribution.
• $\sigma^2$ — the variance of the Gaussian target distribution.

1. Climbing the score reaches the mode

   Follow it uphill, repeatedly — $x \leftarrow x + \eta\, s(x)$ — and $x$ climbs toward $\mu$.

2. Langevin dynamics samples the whole distribution

   Real score-based models add noise to this update (Langevin dynamics) so the process samples the whole distribution instead of always collapsing onto the single peak.

3. Learned directly, no invertibility required

   A neural network is trained to estimate $s(x)$ directly from data, without ever needing an invertible architecture or a tractable normalizing constant.

Every bar here comes from a closed-form formula, not a simulation — the distance to the mode shrinks by exactly the same factor, $0.7$, every single step, no matter how far the trajectory started. That factor is $1 - \eta/\sigma^2$: it's baked into the step size and the target's own spread, not something that has to be measured empirically.

Starting at $x_0=-5$, target $\mathcal{N}(3, 1)$, step size $\eta=0.3$:

1. Step 1

   $s(-5) = -(-5-3)/1 = 8$. $x_1 = -5 + 0.3(8) = -2.6$.

2. Step 2

   $s(-2.6) = -(-2.6-3)/1 = 5.6$. $x_2 = -2.6 + 0.3(5.6) = -0.92$.

3. The distance to the mode obeys a clean pattern

   • $|x_0-3|=|-5-3|=8$
   • $|x_1-3|=|-2.6-3|=5.6$
   • $|x_2-3|=|-0.92-3|=3.92$

   Each one is exactly $0.7\times$ the last ($5.6/8=0.7$, $3.92/5.6=0.7$). That's not a coincidence of these particular numbers; it falls straight out of the fact that $s(x)$ is linear in $x$ for a Gaussian.