Feature scaling & transformation

Two features, two weights, one shared learning rate — exactly like every gradient descent so far in this course, except now $x_1$ ranges across $\pm 1$ and $x_2$ ranges across $\pm 100$. Step through learning rates and watch the faint curve ($w_2$, the huge-scale feature) and the bold curve ($w_1$, the tiny-scale feature) refuse to cooperate.

For linear regression, the curvature of the loss along each weight's axis scales with that feature's variance:

• $\mathcal{L}$ — the loss function being minimized.
• $w_j$ — the weight (parameter) for feature $j$.
• $x_{ij}$ — the value of feature $j$ for training example $i$.
• $j$ — index running over features.
• $i$ — index running over training examples.

1. Curvature scales with the square of a feature's range

   A feature on a 100x larger scale produces roughly $100^2 = 10{,}000\times$ more curvature along its own weight axis — small differences in scale become huge differences in curvature.

2. One shared rate must satisfy the worst axis

   A single learning rate must stay below $2/\text{curvature}$ everywhere to avoid diverging, so the large-scale feature forces the whole learning rate down — including for every feature that didn't need it to be that small.

Train on the raw features for 20 steps at the rate that's safe for the huge-scale feature, and $w_1$ has barely moved — its error is still nearly its full starting distance. Rescale $x_2$ down to $x_1$'s range first, and the same learning rate that solved the tiny-scale weight now solves both weights exactly, in two steps instead of twenty.

$y = w_1 x_1 + w_2 x_2$, true weights $w_1{=}3$, $w_2{=}0.02$, with $x_1 \in \{\pm 1\}$ and $x_2 \in \{\pm 100\}$:

1. The same starting point, wildly different gradients

   With four points $(x_1,x_2,y)$: $(1,100,5)$, $(1,-100,1)$, $(-1,100,-1)$, $(-1,-100,-5)$, all predictions are $0$ at $w_1{=}w_2{=}0$, so each residual is $0-y=-y$:

   • $\partial\mathcal{L}/\partial w_1 = \frac24\sum(\text{residual}\times x_1) = \frac24\big[(-5)(1)+(-1)(1)+(1)(-1)+(5)(-1)\big] = \frac24(-12) = -6$
   • $\partial\mathcal{L}/\partial w_2 = \frac24\sum(\text{residual}\times x_2) = \frac24\big[(-5)(100)+(-1)(-100)+(1)(100)+(5)(-100)\big] = \frac24(-800) = -400$

   Over 66x steeper for $w_2$, purely because $x_2$'s values are 100x larger. The loss surface is a long, narrow valley: gentle along $w_1$'s axis, brutally steep along $w_2$'s.

2. Safe for the steep axis, frozen on the gentle one

   • $\eta=0.00005$ is exactly the rate that lands $w_2$ on its true value in one step.
   • Using that same rate for $w_1$ over 20 steps leaves it at just $0.006$ — still $2.99$ away from its true value of $3$.
   • Double the rate to $\eta=0.0001$ and $w_2$ stops converging altogether, bouncing between $0$ and $0.04$ forever.
3. Scaling turns two curvatures into one

   Dividing $x_2$ by $100$ shrinks its range to match $x_1$'s, and its true weight becomes $0.02\times100=2$. Both features now produce identical curvature. $\eta=0.5$ — the same "magic" rate from the last chapter — lands both $w_1{=}3$ and $w_2{=}2$ exactly, in a single step.