Transfer learning & feature extraction

Two points, one shared slope. One approach starts from a pretrained slope and only has to learn where to shift it; the other has to rediscover the slope and the shift together, from nothing, using the exact same two points.

Transfer learning freezes parameters already learned on a large source task and fine-tunes only the remainder on a small target task:

• $w_{\text{frozen}}$ — the pretrained weight, kept fixed during fine-tuning.
• $b_{\text{fine-tuned}}$ — the one remaining parameter, updated to fit the target task.

1. From scratch means every parameter, coupled together

   Training from scratch instead optimizes every parameter jointly, starting from nothing.

2. Freezing simplifies the optimization problem itself

   Freezing turns a coupled, multi-parameter optimization problem into a much simpler one — often, as here, a single parameter with an easy, well-behaved loss curve.

The learning rate that solves transfer's one-parameter problem exactly, in a single step, sends the joint two-parameter problem into an explosive spiral within 3 steps. The same underlying data, the same starting point conceptually — but coupling two unknowns together changes what "safe" even means.

Pretrained slope $w=2$; target task $y=2x+5$, observed at just two points, $(1,7)$ and $(3,11)$:

1. Freezing turns two unknowns into one

   With $w$ fixed at $2$, the loss in $b$ alone is exactly $2(b-5)^2$-shaped — the same quadratic form from the hyperparameter-tuning chapter, whose gradient with respect to $b$ is $2(b-5)$. Starting from $b=0$: gradient $=2(0-5)=-10$, so one step at $\eta=0.5$ gives $b \leftarrow 0 - 0.5(-10) = 5$ exactly. MSE (short for Mean Squared Error): $0$.

2. The same rate wrecks the joint problem

   Training $w$ and $b$ together from $(0,0)$ at that same $\eta=0.5$, using the joint gradient $(\partial\mathcal{L}/\partial w,\partial\mathcal{L}/\partial b)$ each step:

   • Step 1: $(w,b) \leftarrow (0,0) - 0.5(-40,-18) = (20, 9)$
   • Step 2: $(w,b) \leftarrow (20,9) - 0.5(196,80) = (-78, -31)$
   • Step 3: $(w,b) \leftarrow (-78,-31) - 0.5(-944,-392) = (394, 165)$

   Overshooting further each step, nowhere near the target $(2,5)$.

3. Even a safe rate is slow when there's more to learn

   At a stable $\eta=0.1$, from-scratch training reaches $(w,b)\approx(3.24, 2.01)$ after 5 steps — MSE $\approx1.80$. After 20 steps, 4x the budget, it's only improved to $(w,b)\approx(2.73, 3.23)$ — MSE $\approx0.63$, still far from the transfer model's exact $0$.