Gradient descent — Gradient

Hook

You know the slope at a point. How do you actually use it to walk downhill to the minimum — and what goes wrong if your steps are too big?

Intuition

Step 0

Each click takes one step: move a little in the direction that decreases $f$ , based on the slope right where you're standing. Watch the trail of dots — that's the path down.

Formalize

That's gradient descent. Starting from some $x_0$ , repeat:

x_{n+1} = x_n - \eta \, \nabla f(x_n)

$x_n$ — the current position, before this step.
$x_{n+1}$ — the new position, after taking the step.
$\eta$ — the learning rate: how big a step to take (read "eta").
$\nabla f(x_n)$ — the gradient evaluated at the current position, pointing uphill.

Eta is the learning rate
$\eta$ (eta) is the learning rate — how big a step you take.
Direction versus distance
The gradient tells you which way is uphill; $\eta$ decides how far you step away from it.

Play

Step 0 — x = 7.50, ∇f(x) = 11.00

η (learning rate)0.15

Drag $\eta$ up and take a few steps. Too small and it crawls; push it high enough and it overshoots the minimum, bouncing back and forth — or worse, flies further away with every step.

Worked example

From $x_0 = 5$ with $\eta = 0.1$ :

Take the first step
$x_1 = 5 - 0.1(2 \cdot 5 - 4) = 5 - 0.6 = 4.4$
Take the second step
$x_2 = 4.4 - 0.1(2 \cdot 4.4 - 4) = 4.4 - 0.48 = 3.92$
Read what happened
Each step shrinks the distance to the minimum at $x^* = 2$ by the same factor — that's what a stable learning rate looks like.

Checkpoint

Tune η and take steps until ∇f(x) is within 0.05 of zero — 15 steps or fewer. Too small and you’ll run out of steps; too large and you’ll overshoot.

Step 0 / 15 — ∇f(x) = 11.00

η (learning rate)0.10

0/15 steps used

Set a learning rate and take a step to try it

Summary

x_{n+1} = x_n - \eta \, \nabla f(x_n)

Gradient descent repeatedly steps against the gradient. The learning rate $\eta$ trades off speed against stability: too small converges slowly, too large overshoots or diverges entirely.