Ridge regression (L2 regularization)

These two predictors are correlated at nearly 0.998 — practically the same signal, twice. At λ=0, watch the fitted weights: one strongly positive, one strongly negative, even though both features move with the target in the same direction. Push λ up and watch that flip resolve into two ordinary, positive weights instead.

Ridge regression adds a penalty on the size of the weights themselves to the squared-error loss being minimized:

• SSE($\mathbf{w}$) — the sum of squared errors of the fit, same as ordinary least squares.
• $\lambda$ — the penalty strength. $\lambda=0$ recovers plain OLS; larger $\lambda$ shrinks weights harder.
• $w_j$ — the model's $j$-th weight (the intercept, if there is one, is conventionally left unpenalized).

Minimizing this is still a quadratic problem, so it still has a closed form — just with $\lambda$ added to the diagonal of the normal equations: $(\mathbf{X}^\top\mathbf{X} + \lambda I)\mathbf{w} = \mathbf{X}^\top\mathbf{y}$.

1. Collinearity makes X^TX nearly singular

   When two predictors are highly correlated, $\mathbf{X}^\top\mathbf{X}$ is close to singular — its inverse blows up, and tiny changes in the data produce wildly different "optimal" weights.

2. Adding λ to the diagonal fixes that directly

   Adding $\lambda$ to every diagonal entry pulls $\mathbf{X}^\top\mathbf{X} + \lambda I$ away from singular, however close the raw predictors are to collinear — the matrix stays invertible and the solution stays finite and stable.

3. With correlated predictors, ridge splits credit evenly

   Ridge has no way to prefer one of two near-identical predictors over the other — its penalty treats them symmetrically — so as $\lambda$ grows it pushes their weights toward being equal, sharing the job between them instead of letting one dominate.

Watch the sum of squared errors (SSE) as you move λ: it barely changes, even while the two weights swing from wildly different to nearly identical. Near-perfect collinearity means many different weight combinations fit the data almost equally well — OLS just happens to land on an unstable one.

Two predictors, $x_1 = [1,2,3,4]$ and $x_2 = [1,2,3,4.5]$ — correlated at $\approx 0.998$ — fit to $y=[2,4,7,8]$ with no intercept.

1. Unregularized: an unstable, sign-flipped fit

   The normal equations $\mathbf{X}^\top\mathbf{X}\mathbf{w}=\mathbf{X}^\top\mathbf{y}$, with the numbers plugged in:

   $$\begin{pmatrix}30&32\\32&34.25\end{pmatrix}\begin{pmatrix}w_1\\w_2\end{pmatrix} = \begin{pmatrix}63\\67\end{pmatrix}$$

   Solving by Cramer's rule: the determinant is $30(34.25)-32(32)=1027.5-1024=3.5$, so:

   • $w_1=(63(34.25)-67(32))/3.5=13.75/3.5\approx3.929$
   • $w_2=(30(67)-32(63))/3.5=-6/3.5\approx-1.714$

   Both predictors rise together, yet one gets a strongly negative weight — a symptom of collinearity, not a real pattern in the data.

2. A modest penalty (λ = 1) already fixes the sign

   Adding $\lambda=1$ to the diagonal:

   $$\begin{pmatrix}31&32\\32&35.25\end{pmatrix}\begin{pmatrix}w_1\\w_2\end{pmatrix} = \begin{pmatrix}63\\67\end{pmatrix}$$

   Determinant: $31(35.25)-32(32)=1092.75-1024=68.75$. Solving:

   • $w_1=(63(35.25)-67(32))/68.75=76.75/68.75\approx1.116$
   • $w_2=(31(67)-32(63))/68.75=61/68.75\approx0.887$

   Both positive, and much closer together.

3. A strong penalty (λ = 5) makes them nearly equal

   At $\lambda=5$:

   $$\begin{pmatrix}35&32\\32&39.25\end{pmatrix}\begin{pmatrix}w_1\\w_2\end{pmatrix} = \begin{pmatrix}63\\67\end{pmatrix}$$

   Determinant: $35(39.25)-32(32)=1373.75-1024=349.75$. Solving:

   • $w_1=(63(39.25)-67(32))/349.75=328.75/349.75\approx0.940$
   • $w_2=(35(67)-32(63))/349.75=329/349.75\approx0.941$

   Ridge has split the credit between the two near-identical predictors almost exactly evenly, exactly as the theory predicts.