Linear Discriminant Analysis (LDA)

Class B's mean never moves. Shrink its spread and watch the dashed projection axis rotate — and the separation score climb — purely because the classes have gotten individually tighter, not because they've moved any further apart.

Fisher's LDA looks for the direction $\mathbf{w}$ that maximizes the ratio of between-class separation to within-class spread once every point is projected onto it:

• $\boldsymbol{\mu}_A, \boldsymbol{\mu}_B$ — the mean position of class A and class B.
• $S_W$ — the within-class scatter: how spread out each class is around its own mean, summed across both classes. Small $S_W$ means tight clusters.
• $S_W^{-1}$ — inverting it means directions where the classes are naturally tight get amplified more than directions where they're naturally spread out.
• $\mathbf{w}$ — the resulting direction: not just "toward the other class's mean," but rescaled by how reliable each axis is.

1. Between-class scatter is the mean gap, squared

   $S_B = (\boldsymbol{\mu}_A-\boldsymbol{\mu}_B)(\boldsymbol{\mu}_A-\boldsymbol{\mu}_B)^\top$ — large when the class means are far apart, independent of how tight or loose either cluster is.

2. Without S_W^-1, this would just be PCA on the mean difference

   Projecting onto the raw mean-difference vector alone ignores each class's own shape — a direction where both classes happen to be tightly packed is far more useful for separating them than a raw "toward the other mean" heading.

3. A degenerate S_W (perfectly collinear class spread) breaks the inverse

   If both classes' points fall on a single shared line, $S_W$ becomes singular and $S_W^{-1}$ doesn't exist — the same failure mode ridge regression exists to patch, one chapter over.

The raw Euclidean distance between the two class means never changes as you move the slider — only class B's own spread does. Yet the LDA separation score moves a lot: distance between means alone isn't what makes classes separable, distance relative to how tight each class is does.

Class A: $(1,2), (3,2)$, mean $(2,2)$. Class B: $(6,3), (6,5)$, mean $(6,4)$.

1. Within-class scatter for each class

   Each class's scatter is the sum of $(\text{point}-\text{mean})(\text{point}-\text{mean})^\top$ over its points.

   • Class A: deviations $(1,2)-(2,2)=(-1,0)$ and $(3,2)-(2,2)=(1,0)$. Both outer products give $\begin{bmatrix}1&0\\0&0\end{bmatrix}$, summing to $S_A = \begin{bmatrix}2&0\\0&0\end{bmatrix}$.
   • Class B: deviations $(6,3)-(6,4)=(0,-1)$ and $(6,5)-(6,4)=(0,1)$. Both outer products give $\begin{bmatrix}0&0\\0&1\end{bmatrix}$, summing to $S_B = \begin{bmatrix}0&0\\0&2\end{bmatrix}$.

   Summed: $S_W = S_A+S_B = \begin{bmatrix}2&0\\0&2\end{bmatrix}$ — a clean multiple of the identity, so $S_W^{-1} = \begin{bmatrix}0.5&0\\0&0.5\end{bmatrix}$.

2. The mean-difference vector

   $\boldsymbol{\mu}_A - \boldsymbol{\mu}_B = (2-6,\, 2-4) = (-4,-2)$.

3. The LDA direction, and the resulting separation

   $\mathbf{w} = S_W^{-1}(-4,-2) = (0.5(-4),\ 0.5(-2)) = (-2,-1)$. Projecting each point onto $\mathbf{w}$ (raw dot product):

   • Class A: $(-2)(1)+(-1)(2)=-4$ and $(-2)(3)+(-1)(2)=-8$ — mean $-6$
   • Class B: $(-2)(6)+(-1)(3)=-15$ and $(-2)(6)+(-1)(5)=-17$ — mean $-16$

   A gap of exactly $10$, with each class landing tightly around its own mean.