PCA: directions of maximum variance

Drag the arrow around the cloud. The teal marks are each point's shadow cast straight onto the arrow's line — squeeze together when the arrow points across the cloud's short way, spread apart when it points along the long way.

That spread is variance — of the projections, not the raw points:

• $\hat u$ — the unit direction being tested; only its angle matters since it's normalized to length 1.
• $x_i$ — one data point.
• $\bar x$ — the mean of all the data points.
• $n$ — the number of points.

1. Definition: the direction of maximum variance

   The first principal component is whichever unit direction $\hat u$ makes this as large as possible.

2. Only the angle matters

   It only depends on the direction's angle — stretching the arrow longer or flipping it to point the opposite way changes nothing, since $\hat u$ is normalized before projecting.

Compare a horizontal arrow, a vertical one, and the angle in between. One of those is far closer to the true maximum than the other two — this cloud was built stretched along one specific diagonal, not along either axis.

This cloud was built by combining two independent directions: a wide spread along $(0.6, 0.8)$ and a narrow one along the perpendicular $(-0.8, 0.6)$.

1. Project onto the wide direction

   Projecting each of the 7 points onto the unit direction $(0.6,0.8)$ (dot product $0.6x+0.8y$):

   • $(-2.6,-1.8) \to -3$
   • $(-0.4,-2.2) \to -2$
   • $(-0.6,-0.8) \to -1$
   • $(0,0) \to 0$
   • $(0.6,0.8) \to 1$
   • $(2,1) \to 2$
   • $(1,3) \to 3$

   The mean projection is $0$, so the variance is just the mean of the squares: $(9+4+1+0+1+4+9)/7 = 28/7 = 4$.

2. Project onto the perpendicular

   Projecting onto the perpendicular unit direction $(-0.8,0.6)$ (dot product $-0.8x+0.6y$):

   • $(-2.6,-1.8) \to 1$
   • $(-0.4,-2.2) \to -1$
   • $(-0.6,-0.8) \to 0$
   • $(0,0) \to 0$
   • $(0.6,0.8) \to 0$
   • $(2,1) \to -1$
   • $(1,3) \to 1$

   Mean is again $0$, so variance $=(1+1+0+0+0+1+1)/7 = 4/7 \approx 0.57$ — seven times smaller.

3. Confirm it's the maximum

   No other direction beats $4$: that's the maximum, and $(0.6, 0.8)$ (about $53°$ from the x-axis) is the first principal component.