Saliency maps — Gradient

Hook

Chapter 3's minimal CNN — short for Convolutional Neural Network — predicted "Edge detected" for a striped image. It never said which pixels it actually used to get there — and a network with millions of weights can't be read off the way four dense weights can. A saliency map asks the network directly: nudge one pixel, and see how much the answer moves.

Intuition

the 6×6 input image (1 = light, 0 = dark)

This is the exact image and network from Part IV's minimal CNN chapter. Flip to the saliency map: the two darkest columns on the left get exactly zero weight — nothing about them affects the prediction at all — while the pixels near the dark-to-light boundary light up, because that's the evidence the kernel was built to find.

Formalize

Saliency at a pixel is how much the model's output would change if that one pixel moved, holding every other pixel fixed — the same numerical-gradient technique from the tiny-Transformer capstone, applied to a real input instead of a weight:

\text{saliency}(i,j) = \left| \frac{f(\text{image} + \epsilon\, e_{ij}) - f(\text{image} - \epsilon\, e_{ij})}{2\epsilon} \right|

$\text{saliency}(i,j)$ — the importance score assigned to pixel $(i,j)$ .
$f$ — the predicted class's logit, the model output being explained.
$\text{image}$ — the input image, as a grid of pixel values.
$\epsilon$ — the size of the small nudge applied to one pixel.
$e_{ij}$ — a perturbation that nudges only pixel $(i,j)$ , leaving every other pixel fixed.

A high score means an active path survived
A pixel earns a high score exactly when perturbing it moves the logit a lot — which, through a convolution, ReLU (short for Rectified Linear Unit), and max-pool, only happens for pixels that reach a surviving, active path all the way to the output.

Play

saliency for "Edge detected"

Switch between the "Edge detected" neuron and the "No edge" neuron. The saliency map doesn't change at all — both neurons are reading the exact same evidence from the same feature map, just weighting it with opposite signs. Saliency shows what mattered, not which way it pushed the decision.

Worked example

With $\epsilon = 0.01$ , on the same $6\times6$ striped image from Part IV:

A pixel with zero saliency
Every pixel in columns 0-1 (the left light block) scores exactly $0$ . Nudging it can only move the convolution's response near $-3$ , which ReLU already clips to $0$ — a small nudge can't cross that threshold, so it changes nothing downstream.
A pixel with moderate saliency
Pixel $(0, 2)$ — top row, right at the boundary — scores $0.5$ . It sits inside only one of the feature map's overlapping convolution windows that survives pooling.
The most salient pixels
Pixels $(2, 2)$ through $(3, 5)$ score exactly $1.0$ — twice pixel $(0,2)$ 's score. Rows 2 and 3 sit in the middle of the image, so they fall inside twice as many overlapping $3\times3$ convolution windows that make it through max-pooling as the outer rows do.

Checkpoint

Click any pixel the model gives exactly zero weight to — one it never uses to make this decision.

click a pixel

Click a pixel to try it

Summary

\text{saliency}(i,j) = \left| \frac{f(\text{image} + \epsilon\, e_{ij}) - f(\text{image} - \epsilon\, e_{ij})}{2\epsilon} \right|

A saliency map is the cheapest possible explanation: it needs nothing but the ability to query the model repeatedly, no access to its weights or architecture. That generality is also its weakness — it says where the model looked, pixel by pixel, but not how much each region of the image, as a whole, contributed. The next chapter builds a coarser, more targeted map that answers exactly that, using the convolution layer's own feature map instead of individual pixels.