BERT & masked language modeling

Hook

Predicting the next word only ever looks left. But "the ___ sat" has a word missing from the middle — and the word after the blank, "sat", carries real information a left-to-right model can never use to fill it in.

Intuition

predicted x

2.00

predicted y

0.00

true x (cat)

0.00

true y (cat)

2.00

squared error = 8.00

The same masked word, the same attention mechanism from Part IV — the only thing that changes is which tokens are allowed to be seen. Toggle between a model that only looks left and one that looks both ways, and watch the reconstruction error change.

Formalize

Masked language modeling hides a token and asks a model to reconstruct it from context. A causal (left-to-right) model restricts attention to only the tokens before the mask:

\text{causal: attend over } \{t_j : j < i\} \qquad \text{bidirectional: attend over } \{t_j : j \neq i\}

$t_j$ — the token at position $j$ in the sequence.
$i$ — the position of the masked (currently predicted) token.
$\{t_j : j < i\}$ — the causal context: only tokens strictly before the mask.
$\{t_j : j \neq i\}$ — the bidirectional context: every token except the mask itself, on both sides.

BERT commits to the bidirectional form
BERT — short for Bidirectional Encoder Representations from Transformers — uses the second form. Since there's no autoregressive generation step to protect — the whole point is reconstructing a token that's already there — nothing stops the model from using tokens on both sides.
The only difference from GPT is who can see whom
That's the entire architectural difference from a GPT (short for Generative Pre-trained Transformer)-style model: which half of the sentence is allowed to inform each prediction.

Play

causal error (sees only 'the')

8.00

bidirectional error (sees 'the' + 'sat')

4.50

"the cat sat" — masking the middle word, reconstructing it from context alone

"the [MASK] sat" — reconstructing the hidden middle word using attention over the visible tokens. The causal version can only ever see "the"; the bidirectional version sees "the" and "sat" and blends both into its guess.

Worked example

Tokens "the" $=(2,0)$ , "cat" $=(0,2)$ (masked), "sat" $=(1,1)$ , with a generic mask query $(1,1)$ :

Causal: only the left context exists
With no visible token but "the", attention collapses to weight $1$ on it: the prediction is exactly "the"'s vector, $(2,0)$ . Squared error against the true "cat" vector $(0,2)$ : $2^2+2^2=8$ .
Bidirectional: both sides, scored equally
"the" and "sat" score identically against the mask query — $(1\!\cdot\!2+1\!\cdot\!0)/\sqrt2 = (1\!\cdot\!1+1\!\cdot\!1)/\sqrt2$ — so softmax splits attention exactly $0.5/0.5$ . The prediction becomes their average, $(1.5, 0.5)$ .
Using both directions nearly halves the error
Squared error for the bidirectional prediction: $(1.5-0)^2+(0.5-2)^2 = 2.25+2.25=4.5$ — worse than a perfect reconstruction, but a real improvement over the causal model's $8$ , purely from being allowed to see "sat" as well as "the".

Checkpoint

Find the attention mode that achieves the lower reconstruction error on the masked word.

Pick a mode to try it

Summary

\text{causal: only } j<i \qquad\Rightarrow\qquad \text{bidirectional: every } j\neq i \text{, error } 4.5 \text{ vs } 8

BERT's core idea isn't a new attention mechanism — it's the same self-attention from Part IV, just unrestricted in which tokens can talk to which. Removing the left-to-right constraint costs the ability to generate text one token at a time (there's no "next word" to predict when every position already sees everything), but wins a strictly richer context for tasks like this one, where filling in a blank is the whole point. The next chapter takes the same self-attention mechanism somewhere it was never designed for: an image, with no sequence to speak of at all.