Hyperparameter tuning: Grid, Random & Bayesian search

Same starting point, same loss, same fixed budget of 20 steps — only the learning rate changes. Step through the candidates and watch identical mechanics produce a slow crawl, a perfect landing, an endless bounce, or a runaway explosion.

A hyperparameter is a setting chosen before training — not learned from data, but fixed in advance and left for you to pick. Grid search turns picking into search: define a set of candidate values, run the full training procedure once per candidate, and keep whichever one scores best on held-out data (the same evaluation machinery built in the last two chapters).

• $\eta$ — the hyperparameter being tuned; here, the learning rate.
• $\{\eta_1, \dots, \eta_n\}$ — the finite set of candidate values grid search tries.
• $\text{train}(\eta)$ — the model trained end-to-end using that candidate value.
• $\text{loss}(\cdot)$ — the held-out loss used to score each trained candidate.
• best — the candidate value that scored the lowest loss.

1. Exhaustive, not clever

   It's exhaustive and it's dumb — nothing here is smarter than you dragging the slider by hand.

2. But reliable in ways manual search isn't

   It never gets tired, never misses a candidate you forgot to try, and never fools itself into stopping at a "good enough" value it found first.

Six learning rates, one 20-step training run apiece. Most land somewhere between "still far off" and "essentially perfect." One lands on the minimum exactly. One never improves at all. One detonates. Grid search doesn't need to understand why — it just measures every outcome and reports the winner.

Minimizing $f(x) = (x-3)^2$ from $x_0=0$, 20 steps, six candidate learning rates $\eta$:

1. Too small crawls, symmetric rates tie

   • $\eta=0.01$: loss $\approx4.01$ after 20 steps — still visibly short of the minimum
   • $\eta=0.1$ and $\eta=0.9$: mirror images of each other around the ideal rate, landing on the exact same final loss $\approx0.0012$, by symmetry — $|1-2\eta|$ is $0.8$ either way
2. The exact rate lands in one step

   For this quadratic, $\eta=0.5=\tfrac{1}{2a}$ (where $a=1$ is the curve's steepness) satisfies $1-2\eta=0$ — the update jumps straight to $x=3$ on step one and stays there. Loss $=0$, not approximately, exactly.

3. Too large stalls, too much larger explodes

   • $\eta=1.0$ gives $1-2\eta=-1$: the update overshoots to $x=6$, back to $x=0$, forever — the loss never improves past its starting value of $9$.
   • $\eta=1.1$ pushes $|1-2\eta|$ past $1$: the point sails further from the minimum every step, ending 20 steps later with loss over $13{,}000$.