Lasso regression (L1 regularization)

Same two nearly-identical predictors as the ridge chapter. Push λ up and, past a small threshold, $w_2$ doesn't just shrink toward zero the way ridge's would — it lands exactly on it and stays there.

Lasso swaps ridge's squared penalty for an absolute-value one:

• MSE($\mathbf{w}$) — mean squared error (lasso conventionally penalizes the averaged error, not the raw sum ridge uses — so the same numeric λ lands at a different effective strength for each).
• $\lambda$ — the penalty strength, same role as in ridge.
• $|w_j|$ — the absolute value of the $j$-th weight — the source of everything that follows.

Unlike ridge, this loss has no closed form — it's minimized by cyclic coordinate descent: repeatedly pick one weight, hold the others fixed, and solve for it via soft-thresholding, $\text{soft}(\rho, \gamma) = \text{sign}(\rho)\max(|\rho|-\gamma, 0)$.

1. A squared penalty is smooth at zero; an absolute-value one has a corner

   $w^2$ has zero slope at $w=0$, so shrinking a weight that's already small barely pulls it further. $|w|$ has a constant-slope kink at $w=0$ — that corner is what a coordinate update can land exactly on.

2. Soft-thresholding is the algebraic form of that corner

   Solving the 1D optimality condition for one weight at a time gives exactly the soft-threshold rule: if the weight's "signal" $\rho$ is smaller in magnitude than $\gamma$, the optimal weight is precisely $0$ — not close to it.

3. Correlated predictors can make lasso's choice unstable

   When two predictors carry almost the same signal, which one gets zeroed can depend on arbitrary details like update order — a real quirk of lasso under collinearity, and the reason ElasticNet exists (next chapter).

Same λ, fed to both penalties on the same data. Lasso routinely shows a flat $0.000$ for $w_2$; ridge, right below it, never does — its $w_2$ is always some nonzero decimal, however small.

Same toy setup as ridge: $x_1=[1,2,3,4]$, $x_2=[1,2,3,4.5]$ (correlation $\approx 0.998$), $y=[2,4,7,8]$.

1. Unregularized, lasso agrees with OLS

   At $\lambda=0$, coordinate descent converges to the same unstable fit as before: $w_1\approx3.929$, $w_2\approx-1.714$.

2. A tiny penalty already zeroes w2 (λ = 0.03)

   The soft-threshold cutoff is $\gamma=N\lambda=4(0.03)=0.12$. Starting from $w_1=w_2=0$:

   1. $\rho_1=\sum_i x_{1i}y_i=63$ (since $w_2=0$), so $w_1=\text{soft}(63,0.12)/30=(63-0.12)/30\approx2.096$
   2. $\rho_2 = \sum_i x_{2i}(y_i-w_1x_{1i}) \approx -0.072$ — inside the dead zone $[-0.12,0.12]$, so $w_2=0$ exactly

   Plugging $w_2=0$ back into step 1 reproduces the same $\rho_1$, so this is already the fixed point — $w_1\approx2.096$ absorbs essentially the whole fit.

3. It stays exactly zero across a wide range (λ = 0.5)

   The cutoff is now $\gamma=4(0.5)=2$ — much wider. Starting from $w_1=w_2=0$:

   1. $\rho_1=63$, so $w_1=\text{soft}(63,2)/30=(63-2)/30\approx2.033$
   2. $\rho_2 \approx 1.933$ — still inside the wider dead zone $[-2,2]$, so $w_2=0$

   Again a fixed point immediately: lasso has picked predictor 1 and permanently discarded predictor 2, a form of automatic feature selection ridge structurally cannot do.