Part I — Calculus, Optimization & Gradients · Chapter 5

Taylor series approximations

Hook

A tangent line matches a curve's value and slope at one point — and nowhere else. How much of the actual curve can you recover just by adding more information at that same single point?

Intuition
f(x) = x³ — tangent line at x₀ = 1.0

The dashed curve is f(x)=x3f(x)=x^3. The solid line is its tangent at x0x_0 — drag the slider and watch the line pivot, matching the curve closely right at x0x_0 but drifting away fast on either side.

Formalize

A Taylor series keeps adding derivative terms at x0x_0 to build a better local approximation:

f(x)f(x0)+f(x0)(xx0)+f(x0)2(xx0)2+f(x) \approx f(x_0) + f'(x_0)(x-x_0) + \frac{f''(x_0)}{2}(x-x_0)^2 + \cdots
  • x0x_0 — the point the approximation is built around (the point of expansion).
  • f(x0)f(x_0) — the function's actual value at x0x_0; the zeroth-order term.
  • f(x0)(xx0)f'(x_0)(x-x_0) — the first-order (linear) term: the tangent line's contribution.
  • f(x0)2(xx0)2\dfrac{f''(x_0)}{2}(x-x_0)^2 — the second-order (quadratic) term: adds curvature.
  1. Truncating after one term is the tangent line

    Stopping after the linear term recovers exactly the tangent-line approximation from Chapter 1 — same formula, new name.

  2. Adding the quadratic term adds curvature

    The quadratic term is the first one that can bend — it uses f(x0)f''(x_0) to curve the approximation the same way the true function curves near x0x_0.

  3. Every term after that keeps closing the gap

    Each additional term corrects more of what the previous ones missed; for a well-behaved function, the approximation converges to f(x)f(x) itself as more terms are added.

  4. It only works locally

    All of this accuracy is local — no matter how many terms you add, expect the approximation to get worse the farther xx strays from x0x_0.

Play
at x=1.5: true=3.38, linear=2.50 (err 0.88), quadratic=3.25 (err 0.13)

Move x0x_0 and the evaluation point independently. Near x0x_0, both approximations track the curve almost perfectly; the quadratic one keeps tracking noticeably farther out, because it bends with the curve instead of running straight through it.

Worked example

Expand f(x)=x3f(x)=x^3 around x0=1x_0=1 and evaluate at x=1.5x=1.5:

  1. Compute the ingredients at x0=1
    f(1)=1,f(1)=3(1)2=3,f(1)=6(1)=6f(1)=1,\qquad f'(1)=3(1)^2=3,\qquad f''(1)=6(1)=6
  2. Build the linear approximation
    L(1.5)=1+3(0.5)=2.5L(1.5) = 1 + 3(0.5) = 2.5
  3. Build the quadratic approximation
    Q(1.5)=1+3(0.5)+62(0.5)2=1+1.5+0.75=3.25Q(1.5) = 1 + 3(0.5) + \frac{6}{2}(0.5)^2 = 1 + 1.5 + 0.75 = 3.25
  4. Compare against the true value

    f(1.5)=3.375f(1.5)=3.375.

    • The linear approximation misses by 0.8750.875.
    • The quadratic one misses by only 0.1250.125.

    Seven times closer, from one extra term.

Checkpoint

Expanding f(x) = x³ around x₀ = 2, compute the quadratic Taylor approximation's value at x = 2.5.

Compute Q(2.5), then pick a value
Summary
f(x)f(x0)+f(x0)(xx0)+f(x0)2(xx0)2f(x) \approx f(x_0) + f'(x_0)(x-x_0) + \frac{f''(x_0)}{2}(x-x_0)^2

Every "linear approximation" and "quadratic approximation" you'll meet later — a linearized loss landscape, a locally-quadratic optimizer step — is exactly this truncated Taylor series, keeping just enough terms to be cheap while still capturing the curvature that matters nearby. The next chapter picks the second-order term back up and asks what curvature means for an entire multivariable landscape at once.