Low-rank matrix approximations

reconstruction error ‖A - A_k‖_F = 1.000

The rank-1 heatmap keeps only the SVD's largest term; rank-2 keeps both and reconstructs the original exactly. Watch how close rank-1 already gets, and exactly how large the leftover error is.

The Eckart-Young theorem says the best possible rank-$k$ approximation to $A$ — the one closest to it in Frobenius norm, among every rank-$k$ matrix — is obtained by keeping only the $k$ largest terms of $A$'s SVD:

• $\sigma_i$ — the $i$-th singular value of $A$, sorted largest to smallest.
• $u_i, v_i$ — the corresponding left and right singular vectors.
• $A_k$ — the best possible rank-$k$ approximation of $A$.
• $\|\cdot\|_F$ — the Frobenius norm: the square root of the sum of every entry squared.

1. Truncation error is exact, not a bound

   The right-hand equation isn't an upper bound — it's the exact error, computable directly from the singular values you dropped, without ever forming $A_k$ and subtracting.

2. Why the small singular values matter least

   Since the dropped singular values are squared and summed, a handful of small $\sigma_i$'s contribute far less error than their count alone would suggest — this is what makes low-rank compression work at all.

3. It's the best rank-k matrix, full stop

   No other rank-$k$ matrix — found by any method — can get closer to $A$ in Frobenius norm than $A_k$ does. Truncating the SVD isn't just convenient; it's provably optimal.

rank 1: error = 1.000, energy retained = 96.2%

rank 2: error = 0.000, energy retained = 100.0%

$\sigma_1=5$ carries far more of the squared-Frobenius "energy" than $\sigma_2=1$ — which is exactly why dropping $\sigma_2$ costs so little.

$A=\begin{pmatrix}3&2\\2&3\end{pmatrix}$ has singular values $\sigma_1=5,\ \sigma_2=1$, with singular vectors $u_1=v_1=(\tfrac{1}{\sqrt2},\tfrac{1}{\sqrt2})$ and $u_2=v_2=(\tfrac{1}{\sqrt2},-\tfrac{1}{\sqrt2})$:

1. Build the rank-1 approximation

   $A_1 = 5 u_1 v_1^\top = 5\cdot\tfrac12\begin{pmatrix}1&1\\1&1\end{pmatrix} = \begin{pmatrix}2.5&2.5\\2.5&2.5\end{pmatrix}$.

2. Compute the error directly

   $A - A_1 = \begin{pmatrix}0.5&-0.5\\-0.5&0.5\end{pmatrix}$, and $\|A-A_1\|_F=\sqrt{4\times0.5^2}=\sqrt{1}=1$.

3. Confirm it matches the formula

   Eckart-Young predicts $\|A-A_1\|_F = \sqrt{\sigma_2^2} = \sigma_2 = 1$ — exactly what direct subtraction gave. Retained energy: $\sigma_1^2/(\sigma_1^2+\sigma_2^2)=25/26\approx96.2\%$.