Optimizers: Momentum, RMSProp, Adam & AdamW

Same ravine as Part II's momentum chapter, same starting point, three optimizers stepping at once. Take a few steps and watch orange zigzag through the steep direction while teal barely wobbles at all.

RMSProp — short for Root Mean Square Propagation — keeps a running average of each parameter's squared gradient, then divides the step by its square root — a direction with a big, steep gradient gets its step shrunk automatically:

• $v$ — the running average of the squared gradient.
• $\beta$ — the decay rate controlling how much history that running average keeps.
• $g$ — the current gradient.
• $\theta$ — the parameter being updated.
• $\eta$ — the learning rate.
• $\epsilon$ — a tiny constant that keeps the division from blowing up when $v$ is near zero.

1. Adam combines RMSProp with momentum

   Adam — short for Adaptive Moment Estimation — keeps a running average of the gradient itself ($m$, momentum's idea) and of its square ($v$, RMSProp's idea).

2. Both averages get corrected for startup bias

   Adam corrects both $m$ and $v$ for their startup bias before combining them the same way RMSProp does.

Watch the distance-to-target numbers rather than the picture. Early on, RMSProp's per-direction scaling lets it beeline almost straight there while momentum is still fighting its own oscillation off the steep axis.

At the very first step from $(4,1)$ on $f(x,y)=x^2+10y^2$:

1. Find the gradient

   • $\partial f/\partial x = 2x = 2(4) = 8$
   • $\partial f/\partial y = 20y = 20(1) = 20$

   giving $(8,20)$ — $2.5\times$ larger in $y$ than $x$.

2. Compute RMSProp's first update

   Before its running average has accumulated any history, the update reduces to $\eta \cdot \text{sign}(g)/\sqrt{1-\beta}$ on each axis — with $\eta=0.3$, $\beta=0.9$, that's a step of $0.3/\sqrt{0.1}\approx0.949$ on both axes.

3. See what that means

   RMSProp actively cancels out the difference in steepness between directions, something plain gradient descent and momentum never do.