Loss landscapes in high dimensions

The gradient is exactly zero right where you started — by every test gradient descent has, this looks like a place to stop. Drag along $x$ and the value climbs. Drag along $y$ instead, and it falls. Zero gradient was never a guarantee of a minimum.

1. A true minimum curves up in every direction

   A flat point ($\nabla f = 0$) is a true local minimum only if the function curves upward in every direction through it.

2. One downward direction makes it a saddle

   If even one direction curves downward instead, it's a saddle point — and gradient descent can simply walk out along that direction.

3. The Hessian's eigenvalues decide which

   Whether a given direction curves up or down is governed by the sign of an eigenvalue of the Hessian (the matrix of second derivatives); a minimum needs all of them positive.

Switch between Bowl and Saddle. Same starting point, same exactly-zero gradient — one traps you, one doesn't. In two dimensions, a random flat point has to get lucky in only 2 independent directions to be a true minimum.

1. Model each dimension as a coin flip

   Treat each of $n$ dimensions at a flat point as an independent coin flip: does the function curve up or down along that direction? A true minimum needs every coin to land "up."

2. Compute the probability as n grows

   With fair coins, that probability is $(1/2)^n$:

   • $n=1$: $(1/2)^1=0.5$
   • $n=2$: $(1/2)^2=0.25$
   • $n=7$: $(1/2)^7=1/128\approx0.0078$ — already under $1\%$
   • $n=20$: $(1/2)^{20}=1/1{,}048{,}576\approx0.00000095$ — under one in a million
3. Apply it to real networks

   Real networks have millions of dimensions. Almost every flat point gradient descent finds is a saddle, not a trap — and saddles have at least one escape direction, which is exactly what noisy, non-exact gradients (Chapter 5 of Part II) are good at stumbling onto.