Momentum & Nesterov acceleration

Same ravine, same starting point, same β. The accent trail is plain Momentum; the ink trail is Nesterov. Step through it — the two are identical on step one, since velocity starts at zero either way, and then they visibly split.

Momentum evaluates the gradient at the current point. Nesterov accelerated gradient (NAG) evaluates it at a look-ahead point first:

• $\tilde\theta$ — the look-ahead point: where momentum alone would carry $\theta$ before this step's gradient is even computed.
• $v$ — the velocity carried over from the previous step.
• $v'$ — the new velocity, built from the gradient at $\tilde\theta$ instead of at $\theta$.
• $\theta, \theta'$ — the parameters before and after this step.
• $\eta, \beta$ — the learning rate and momentum coefficient, same roles as in plain Momentum.

1. Same bookkeeping, one different input

   Every quantity here — $v$, $\eta$, $\beta$ — means exactly what it meant in the Momentum chapter. The only change is where the gradient is evaluated.

2. Step one is always identical to Momentum

   With $v=0$ at the start, $\tilde\theta=\theta$, so the first step's gradient is evaluated at the same place either way — the two methods can only start splitting apart from step two onward.

3. Looking ahead corrects overshoot sooner

   If the velocity is about to carry the point past the minimum, the look-ahead gradient already points back the other way — Nesterov reacts to that correction one step earlier than plain Momentum can.

Push β high on Momentum and watch it overshoot and oscillate. Switch to Nesterov at the same β — the look-ahead gradient starts pulling back sooner, which is why NAG tends to tolerate higher β before it starts bouncing.

Starting at $(4,1)$ on $f(x,y)=x^2+10y^2$, with $\eta=0.045$, $\beta=0.7$, velocity initially zero:

1. Step 1 is identical to Momentum

   $\tilde\theta = (4,1) - 0.045(0.7)(0,0) = (4,1)$, so the gradient there is $(8,20)$, giving $v'=(8,20)$ and $\theta' = (4,1)-0.045(8,20) = (3.64, 0.1)$ — exactly Momentum's first step.

2. Step 2's look-ahead point is new

   $\tilde\theta = (3.64,0.1) - 0.045(0.7)(8,20) = (3.64,0.1) - (0.252, 0.63) = (3.388, -0.53)$

3. Evaluate the gradient there, not at (3.64, 0.1)

   $$\nabla f(3.388,-0.53) = (2 \times 3.388,\ 20\times(-0.53)) = (6.776,\ -10.6)$$
4. Finish the update

   $v' = 0.7(8,20)+(6.776,-10.6) = (12.376, 3.4)$, so $\theta' = (3.64,0.1) - 0.045(12.376,3.4) = (3.08308, -0.053)$ — already noticeably different from Momentum's second step, $(3.0604, -0.62)$, purely from evaluating the gradient one point ahead.