Tile-based differentiable rasterization in 3DGS

Drag the depth slider. The Gaussian's world-space spread never changes, but its footprint on screen shrinks the farther away it gets pushed — the same perspective shrink a camera applies to a point's position, just applied to a blob's size instead.

Approximating the camera as looking straight at each Gaussian, its projected screen-space spread is world spread scaled by focal length over depth — the exact same ratio that turns a 3D point's position into a pixel:

Gaussians must then be alpha-composited nearest-first, exactly like NeRF's ray samples. Once rendered, 3DGS drops the neural network entirely: every Gaussian's own parameters — opacity, position, color, covariance — get updated directly by gradient descent against a rendering loss:

• $\sigma_{\text{world}}, \sigma_{\text{2D}}$ — a Gaussian's spread before and after projection.
• $f$, depth — focal length and the Gaussian's distance from the camera.
• $\theta$ — any one of a Gaussian's own parameters (here, opacity); $\eta$ — the learning rate.
• $L$ — squared-error loss between the rendered pixel and its target color.

1. Two Gaussians, same screen position, different depths

   A near, half-opaque red Gaussian (depth 5) and a far, mostly-opaque blue one (depth 10), both centered exactly on the query pixel with the same projected spread. Composited nearest-first: red claims $0.5$ of the pixel, blue claims $0.5 \times 0.9 = 0.45$ of what's left — result $(0.5, 0, 0.45)$.

2. The same two Gaussians, composited in the wrong order

   Farthest first instead: blue claims $0.9$ of the pixel outright, red only gets $0.1 \times 0.5 = 0.05$ of the leftover — result $(0.05, 0, 0.9)$. Same scene, same Gaussians, a completely different pixel — because "nearest first" isn't a stylistic choice, it's what makes occlusion correct.

Step through a single Gaussian's opacity being optimized by gradient descent, starting far from the target color. No MLP, no ray march — just this one parameter (and, in a real scene, thousands like it) nudged directly against the rendering loss until the rendered pixel matches the photograph it's being fit to.

Two more depths for the same world spread ($\sigma_{\text{world}} = 0.5$, $f = 100$):

1. depth = 10

   $\sigma_{\text{2D}} = 0.5 \times 100 / 10 = 5$.

2. depth = 20 — twice as far

   $\sigma_{\text{2D}} = 0.5 \times 100 / 20 = 2.5$ — exactly half the previous spread. Doubling depth always halves projected spread, for any Gaussian.

And the first gradient step on opacity, starting at $0.5$ (color red, target $(1, 0.2, 0.2)$, learning rate $0.1$):

1. Loss at the start

   Rendered $= (1, 0.5, 0.5)$. Loss $= (0.5-0.2)^2 + (0.5-0.2)^2 = 0.18$ (red channel matches exactly, so it contributes nothing).

2. Gradient and one step

   Since rendered $=\text{opacity}\cdot\text{color}+(1-\text{opacity})\cdot\text{background}$, the chain rule gives $\partial L/\partial\text{opacity} = \sum_c 2(\text{rendered}_c-\text{target}_c)(\text{color}_c-\text{background}_c)$ per channel:

   • Red: $2(1-1)(1-1) = 0$
   • Green: $2(0.5-0.2)(0-1) = -0.6$
   • Blue: $2(0.5-0.2)(0-1) = -0.6$

   Summing: $0-0.6-0.6=-1.2$ exactly. One step: $0.5 - 0.1 \times (-1.2) = 0.62$ — closer to the target, and the loss at $0.62$ is lower than at $0.5$.