Part XIX — Alignment, Mechanistic Interpretability, Safety & Red-Teaming · Chapter 5

Mechanistic interpretability fundamentals

Hook

Somewhere inside a trained network there's a number — one neuron's activation — that goes up on some inputs and down on others. Reading that number and guessing "it means large animals" is a hypothesis, not a fact. Mechanistic interpretability's job is turning "I think this neuron means X" into an experiment that can actually falsify it.

Intuition

Toggle each feature on and off. Neuron A only ever moves for "large AND not metal" — legs don't matter to it at all. Neuron B doesn't care about size or material; it fires for anything with legs. Two sharply different, cleanly separable jobs, found here by brute-force toggling because this network is small enough to try every combination by hand.

Formalize

Probing finds correlations: which inputs make a neuron fire. Activation patching tests causation instead: take a neuron's activation from one input (the source), splice it into an otherwise-normal forward pass on a different input (the target), and check whether the output moves toward the source's own output. If it does, that neuron was carrying at least part of the causal difference between the two inputs — not just correlated with it.

outputpatched=wA⋅apatch+wB⋅bpatch\text{output}_{\text{patched}} = w_A \cdot a_{\text{patch}} + w_B \cdot b_{\text{patch}}
  • apatch,bpatcha_{\text{patch}}, b_{\text{patch}} — each neuron's activation, taken from either the target (left alone) or the source (spliced in).
  • wA,wBw_A, w_B — the fixed output weights combining both neurons into the network's single output score.
  1. Run the target normally, note the output
    Baseline: both neurons take the target's own values.
  2. Swap in one neuron's source value
    Everything else about the forward pass — the other neuron, all the weights — stays exactly as it was for the target.
  3. Compare the patched output to both baselines
    If it lands on the source's actual output, that one neuron explains the entire target/source difference.
Play

Patch mouse's neuron A with elephant's value, and the output jumps all the way to elephant's own output — because mouse and elephant already agree on neuron B (both have legs), so neuron A alone was carrying the entire size difference. Patch neuron B instead, from the same pair, and nothing moves.

Worked example

Patching neuron A from "elephant" into "mouse"'s forward pass:

  1. Mouse's own values

    size=0, legs=1, metal=0:

    • Neuron A =ReLU(2⋅0−3⋅0−1)=ReLU(−1)=0=\text{ReLU}(2\cdot0 - 3\cdot0 - 1) = \text{ReLU}(-1) = 0
    • Neuron B =ReLU(3⋅1−1)=ReLU(2)=2=\text{ReLU}(3\cdot1 - 1) = \text{ReLU}(2) = 2
    • Output =2(0)−0.5(2)=−1= 2(0) - 0.5(2) = -1
  2. Elephant's own values

    size=1, legs=1, metal=0:

    • Neuron A =ReLU(2⋅1−3⋅0−1)=ReLU(1)=1=\text{ReLU}(2\cdot1 - 3\cdot0 - 1) = \text{ReLU}(1) = 1
    • Neuron B =ReLU(3⋅1−1)=ReLU(2)=2=\text{ReLU}(3\cdot1 - 1) = \text{ReLU}(2) = 2
    • Output =2(1)−0.5(2)=1= 2(1) - 0.5(2) = 1
  3. Patch: mouse's B, elephant's A

    Output = 2(1) − 0.5(2) = 2 − 1 = 1 — exactly elephant's output. Neuron A alone accounts for the entire −1 → 1 gap, confirmed causally, not just by correlation.

Checkpoint

Toggle the features until neuron A fires (activation > 0) while neuron B stays silent (activation = 0).

Toggle a feature to try it
Summary
output=wA⋅ReLU(… )+wB⋅ReLU(… )\text{output} = w_A \cdot \text{ReLU}(\dots) + w_B \cdot \text{ReLU}(\dots)

Toggling inputs finds correlations; patching activations between forward passes tests which of those correlations are actually load-bearing for the output. Real networks make both much harder — millions of neurons instead of two, and, as the next chapter shows, a single neuron often encodes more than one concept at once, so there may be no clean "this neuron means X" to find in the first place.