ElasticNet regularization

Same two correlated predictors as the last two chapters, same total penalty strength — only the mix between L1 and L2 changes. At α=1 it's pure lasso: $w_2$ sits at exact zero. Slide α down and watch $w_2$ revive, smoothly, long before you reach pure ridge.

ElasticNet's penalty is a weighted blend of both:

• $\lambda$ — the total penalty strength, same role as in ridge and lasso.
• $\alpha$ — the mixing parameter, between 0 and 1: $\alpha=1$ is pure lasso, $\alpha=0$ is pure ridge.
• $\sum_j |w_j|$, $\sum_j w_j^2$ — the L1 and L2 penalty terms from the last two chapters, now combined.

1. Each coordinate update generalizes both single-penalty rules

   Coordinate descent still soft-thresholds by the L1 share of the penalty, then divides by the data term plus the L2 share — set $\alpha=1$ and the L2 share vanishes, recovering lasso's update exactly; set $\alpha=0$ and the soft-threshold cutoff vanishes, recovering a ridge-like update.

2. The L2 term restores ridge's grouping effect

   Even a small amount of L2 ($\alpha$ just under 1) is enough to stop lasso's arbitrary all-or-nothing choice between correlated predictors — both stay in the model, sharing the fit instead of one evicting the other.

3. The L1 term keeps sparsity available

   Push $\alpha$ close enough to 1 and the sparsity comes back — ElasticNet doesn't have to choose between "shrinks everything" and "zeros out weak predictors"; $\alpha$ dials between them.

Drag α down from 1. Somewhere before you reach 0, $w_2$ stops being exactly zero and starts climbing — that's the L2 term's grouping effect outweighing L1's corner.

Same toy setup as ridge and lasso: $x_1=[1,2,3,4]$, $x_2=[1,2,3,4.5]$, $y=[2,4,7,8]$, fixed $\lambda=0.5$.

1. α = 1 (pure lasso): sparse, as expected

   Pure L1: the L1 share is $N\lambda\alpha=4(0.5)(1)=2$, the L2 share is $0$. One coordinate-descent round from $w_1=w_2=0$:

   1. $\rho_1 = \sum_i x_{1i}(y_i-w_2x_{2i}) = \sum_i x_{1i}y_i = 63$ (since $w_2=0$), so soft-thresholding gives $w_1 = (63-2)/30 \approx 2.033$
   2. $\rho_2 = \sum_i x_{2i}(y_i-w_1x_{1i}) \approx 1.933$ — inside the dead zone $[-2,2]$, so $w_2=0$ exactly

   Plugging $w_2=0$ back into step 1 reproduces the same $\rho_1$, so this is already the fixed point — no further rounds change anything.

2. α = 0.5: both predictors survive, close together

   Now the L1 share is $4(0.5)(0.5)=1$ and the L2 share is also $4(0.5)(0.5)=1$. One round from $w_1=w_2=0$:

   1. $\rho_1=63$, so $w_1=(63-1)/(30+1)=62/31=2.000$
   2. $\rho_2 = \sum_i x_{2i}(y_i-w_1x_{1i}) = 3$, so $w_2=(3-1)/(34.25+1) \approx 0.057$

   Unlike $\alpha=1$, this isn't a fixed point yet — repeating the same two-line cycle (recompute $\rho_1$ with the latest $w_2$, then $\rho_2$ with the latest $w_1$) nudges both weights further each round, converging to $w_1\approx1.069$, $w_2\approx0.902$ — no longer sparse, but the two nearly-collinear predictors are now sharing credit almost the way ridge would.

3. α = 0 (pure ridge): fully grouped

   Now the L1 share is $0$ (no soft-thresholding) and the L2 share is $4(0.5)(1)=2$. One round from $w_1=w_2=0$:

   1. $\rho_1=63$, so $w_1=63/(30+2)=63/32\approx1.969$
   2. $\rho_2 = \sum_i x_{2i}(y_i-w_1x_{1i}) = 4$, so $w_2=4/(34.25+2)\approx0.110$

   Repeating that same cycle converges to $w_1\approx1.028$, $w_2\approx0.941$ — about as close together as ridge alone would produce (a differently-scaled λ than the closed-form ridge chapter, since this loss averages the squared error instead of summing it — but the qualitative grouping behavior matches).