Singular Value Decomposition (SVD)

Drag $v$ around. Most of the time, $Av$ points somewhere completely different — the angle between them swings all over the place. But two opposite directions are special: keep dragging near the diagonal and watch that rotation angle collapse toward $0°$.

A vector $v$ is an eigenvector of $A$ if $A$ doesn't rotate it at all — only scales it:

• $A$ — the matrix being analyzed.
• $v$ — the eigenvector: a direction $A$ only scales, never rotates.
• $\lambda$ — the eigenvalue: how much that direction gets stretched (or flipped, if negative).

1. What the eigenvalue means

   $\lambda$ is the corresponding eigenvalue: how much that direction gets stretched (or, if $\lambda<0$, stretched and flipped).

For a symmetric matrix, the eigenvectors are always perpendicular to each other, and $A$ can be rebuilt exactly from them:

• $Q$ — the matrix whose columns are the (unit) eigenvectors of $A$.
• $\Lambda$ — the diagonal matrix holding the eigenvalues, in the same order as $Q$'s columns.

1. Naming the eigendecomposition

   Here $Q$'s columns are the (unit) eigenvectors and $\Lambda$ is diagonal with the eigenvalues on it — this is the eigendecomposition.

2. Generalizing to the SVD

   The singular value decomposition (SVD), $A = U\Sigma V^\top$, is the same idea generalized to any matrix, square or not — but its diagonal $\Sigma$ is always non-negative, so a negative eigenvalue's direction gets its sign absorbed into $U$ instead.

Watch the stretch ratio $|Av|/|v|$ alongside the rotation angle. At the two special directions, the ratio locks onto a fixed number — that number is $|\lambda|$ for that direction, no matter how far out you drag $v$ along it.

For $A = \begin{pmatrix}1&2\\2&1\end{pmatrix}$:

1. Solve the characteristic equation

   $\det(A-\lambda I) = (1-\lambda)^2 - 4 = \lambda^2 - 2\lambda - 3 = (\lambda-3)(\lambda+1) = 0$, so $\lambda_1 = 3$ and $\lambda_2 = -1$.

2. Solve for each eigenvector

   • For $\lambda_1=3$: $(A-3I)v=0 \Rightarrow -2x+2y=0 \Rightarrow x=y$, giving direction $(1,1)$.
   • For $\lambda_2=-1$: $(A+I)v=0 \Rightarrow x+y=0$, giving direction $(1,-1)$.

   Normalized, these are $v_1=(\tfrac{1}{\sqrt2},\tfrac{1}{\sqrt2})$ and $v_2=(\tfrac{1}{\sqrt2},-\tfrac{1}{\sqrt2})$ — exactly the $45°$ and $-45°$ directions from the demo above.

3. Turn it into an SVD

   Singular values are $|\lambda|$, sorted:

   • $\sigma_1=3$
   • $\sigma_2=1$

   Since $\lambda_2$ is negative, its left singular vector flips sign:

   • $u_1=v_1=(\tfrac{1}{\sqrt2},\tfrac{1}{\sqrt2})$
   • $u_2=-v_2=(-\tfrac{1}{\sqrt2},\tfrac{1}{\sqrt2})$

   Each term is a rank-1 outer product, scaled by its singular value:

   • $\sigma_1 u_1v_1^\top = 3\begin{pmatrix}0.5&0.5\\0.5&0.5\end{pmatrix} = \begin{pmatrix}1.5&1.5\\1.5&1.5\end{pmatrix}$
   • $\sigma_2 u_2v_2^\top = 1\begin{pmatrix}-0.5&0.5\\0.5&-0.5\end{pmatrix}$

   Adding the two terms entrywise reconstructs $A$ exactly:

   $$\begin{pmatrix}1.5&1.5\\1.5&1.5\end{pmatrix} + \begin{pmatrix}-0.5&0.5\\0.5&-0.5\end{pmatrix} = \begin{pmatrix}1&2\\2&1\end{pmatrix}$$