Fine-tune a small pretrained model

Two raw inputs pass through the exact same frozen residual block from earlier in this part. The backbone itself never changes — only its output becomes the input to whatever small head gets trained next.

A "pretrained model" here is just the residual recursion from earlier — $h_i = h_{i-1} + w\cdot\sigma(w h_{i-1})$, weights fixed — reused as a frozen feature extractor:

• $\text{feature}$ — the frozen backbone's output for a given input: the representation the head trains on.
• $\text{Backbone}$ — the frozen, pretrained residual network reused as a fixed feature extractor.
• $w_{\text{head}}$ — the task-specific head's weight, applied to the backbone's feature.
• $b_{\text{head}}$ — the task-specific head's bias — in this example, the only parameter still being learned.

1. Fine-tuning leaves almost nothing to learn

   Fine-tuning freezes the backbone and, in this example, even freezes $w_{\text{head}}$ at its correct pretrained value — leaving only $b_{\text{head}}$ to learn.

2. From scratch means finding both head parameters together

   Training from scratch instead has to find $w_{\text{head}}$ and $b_{\text{head}}$ together, using the same two backbone features and the same tiny dataset.

Fine-tuning the head's one remaining parameter reaches an exact fit in a single step. Training both head parameters from scratch is still measurably behind after 10 steps, and only approaches — never exactly matches — that same result even after 50.

Backbone inputs $x_0=-1$ and $x_0=1$, one frozen residual layer, head target $y=2\cdot\text{feature}+5$:

1. The frozen backbone produces two well-separated features

   With one frozen layer and $w=3$, $h=x_0+3\sigma(3x_0)$:

   • $\text{Backbone}(-1) = -1+3\sigma(-3) \approx -1+3(0.0474) \approx -0.858$
   • $\text{Backbone}(1) = 1+3\sigma(3) \approx 1+3(0.9526) \approx 3.858$

   Different enough that the two-point target line is easy to recover, not a near-duplicate pair.

2. Fine-tuning the head's bias: solved in one step

   With $w_{\text{head}}=2$ frozen at its correct value, $\eta=0.5$ lands $b_{\text{head}}=5$ exactly. MSE (short for Mean Squared Error): $0$.

3. From scratch: the same ideal rate explodes; a safe one crawls

   That same $\eta=0.5$, applied to both head parameters jointly from $(0,0)$, explodes to $w\approx1059$ within 3 steps. A safe $\eta=0.05$ instead reaches MSE $\approx3.33$ after 10 steps and only MSE $\approx0.011$ after 50 — closer, but still not the exact $0$ that freezing the backbone got in a single step.