Image compression & latent extraction via SVD/PCA

reconstruction error = 8.944 (energy retained: 0%)

A 4x4 toy "image," reconstructed from 0, 1, or 2 of its singular components. Rank 0 is blank; rank 1 is already a recognizable (if blurred) sketch; rank 2 is the original back exactly.

Every image matrix $I$ (real or toy) has a singular value decomposition, and its best rank-$k$ reconstruction is the same truncated-SVD formula from the previous chapter:

• $I$ — the image, treated as a plain matrix of pixel intensities.
• $\sigma_i, u_i, v_i$ — the $i$-th singular value and left/right singular vectors of $I$.
• $I_k$ — the rank-$k$ compressed reconstruction of the image.

1. What compression actually means here

   Storing $I_k$ only requires $k$ singular values plus $k$ vectors of each length — for a large image, $k$ singular components can cost far fewer numbers than the raw pixel grid.

2. Where 'eigenfaces' comes from

   Applied to a collection of face images stacked into one matrix, the same $u_i$ patterns become eigenfaces: reusable visual building blocks that any face in the set can be approximated from.

3. The tradeoff is exact, not fuzzy

   Exactly as in the previous chapter, $\|I - I_k\|_F = \sqrt{\sum_{i>k}\sigma_i^2}$ — you can compute precisely how much reconstruction error a given compression ratio costs, before storing anything.

Pattern 1 (uniform across every row) alone captures 80% of the image's total energy — a single rank-1 term already does most of the work.

The toy image is built from two clean row-patterns: $u_1=(0.5,0.5,0.5,0.5)$ with $\sigma_1=8$, and $u_2=(0.5,0.5,-0.5,-0.5)$ with $\sigma_2=4$ (each pattern used as both its own left and right singular vector, since the image is symmetric):

1. Rank-1 reconstruction

   $I_1 = 8 \cdot u_1u_1^\top$: every entry of $u_1u_1^\top$ is $0.5\times0.5=0.25$, so $I_1$ is a flat image with every pixel equal to $8\times0.25=2$.

2. The error, by direct subtraction

   Reconstruct $I$ itself, since it was never written down: $I = \sigma_1u_1u_1^\top+\sigma_2u_2u_2^\top$.

   1. The second term has entries $4\times(\pm0.5)(\pm0.5)=\pm1$ — positive where $u_2$'s two entries share a sign, negative where they don't:

      $$\sigma_2u_2u_2^\top = 4\begin{pmatrix}0.25&0.25&-0.25&-0.25\\0.25&0.25&-0.25&-0.25\\-0.25&-0.25&0.25&0.25\\-0.25&-0.25&0.25&0.25\end{pmatrix} = \begin{pmatrix}1&1&-1&-1\\1&1&-1&-1\\-1&-1&1&1\\-1&-1&1&1\end{pmatrix}$$

   2. Adding that to $I_1$'s flat $2$s gives $I$ — a matrix of $1$s and $3$s, each exactly $1$ away from $I_1$'s $2$:

      $$I = \begin{pmatrix}3&3&1&1\\3&3&1&1\\1&1&3&3\\1&1&3&3\end{pmatrix}$$

   So $I - I_1$ is exactly the $\pm1$ matrix from step 1, in all 16 cells, and $\|I-I_1\|_F=\sqrt{16\times1^2}=4$.

3. Confirm it matches Eckart-Young

   The formula predicts $\|I-I_1\|_F=\sqrt{\sigma_2^2}=\sigma_2=4$ — exactly what direct subtraction gave. Energy retained: $8^2/(8^2+4^2)=64/80=80\%$.