Instant-NGP made NeRF's MLP faster to query, but it's still an MLP — one dense function you have to evaluate at every sample along every ray. What if a scene were made of pieces simple enough to skip the network querying almost entirely: a pile of soft, colored blobs you can just draw?
Two overlapping "Gaussians" — soft ellipses that fade out from their center, red one nearer the camera, blue one behind it. Drag the blue one's center around. Right on top of the query pixel, its color is a purple-ish blend of both; slide it away and the pixel settles back to solid red, because the far Gaussian's fading tail no longer reaches that point at all.
Every Gaussian is a small bundle of parameters: a center , a per-axis spread , an opacity, and a color. Its opacity at any query point falls off from its center with squared, sigma-scaled distance — the same bell curve as a 1D Gaussian, just axis-aligned in 2D:
Multiple Gaussians composite front-to-back — nearer ones painted first, each later one only allowed to paint whatever fraction of the pixel is still transparent:
- — a Gaussian's center and per-axis spread, both in the same 2D units as the query point.
- — the query point's offset from , before scaling by .
- — Gaussian 's own opacity at this point: its opacity parameter times its falloff weight.
- — transmittance: how much of the pixel survives every nearer Gaussian, before Gaussian gets its turn.
- Gaussian A (red, near): mu=(0,0), sigma=(2,2), opacity=0.6
Query pixel is at , so , . Weight , so .
- Gaussian B (blue, far): mu=(2,0), sigma=(2,2), opacity=0.8
The query pixel is exactly as far (in sigma units) from B's center as from A's: , same weight , so .
- Composite front-to-back: A first, then B through what A left transparent
- Gaussian A: , contribution (all red).
- Gaussian B: , contribution (all blue).
Total alpha — the pixel is mostly opaque, blended red-over-blue.
Turn up B's opacity and watch its bar (and its share of the composited color) grow — but never past what A's transmittance leaves behind. Widen B's spread instead: the shape of its falloff changes even though its center never moves, which is exactly why "covariance" is a real, separate parameter from position.
Two variations on the same scene:
- B's opacity pushed to 1.0 (fully opaque where it reaches)
. B's contribution becomes — nearly as much of the pixel as A itself, even though A is nearer and gets first pick.
- B's spread shrunk to sigma=0.5 (a tight, small blob)
The raw offset is unchanged, but now , weight , so — a much smaller alpha than the original sigma=2 case, even at the same opacity, because shrinking sigma makes the same raw distance count for far more standard deviations out into the tail.
Drag Gaussian B's opacity until the composited pixel's total alpha (coverage) reaches ≈ 0.85.
A Gaussian splat scene trades a neural network for a pile of simple, explicit blobs — each one's opacity at a point is just a scaled bell curve, and rendering is the same front-to-back alpha compositing NeRF uses for ray samples, with "distance from a Gaussian's center" standing in for "distance along a ray." The next chapter adds the two pieces still missing: how a 3D Gaussian's world-space spread projects onto the image plane, and how its parameters get optimized directly by gradient descent.