Part I — Calculus, Optimization & Gradients · Chapter 3

The chain rule

Hook

If ff speeds up as xx grows, and gg speeds up as ff grows, how fast does g(f(x))g(f(x)) actually grow?

Intuition
u = f(x) = 3.00

↓ feeds into g(u)

y = g(u) = 9.00

Drag the point on the top curve. Whatever value it lands on, uu, feeds straight into the bottom curve — the point down there is never dragged directly, it's just wherever ff sent it.

Formalize

Call the whole composition h(x)=g(f(x))h(x) = g(f(x)). Its derivative is the chain rule

dhdx=dgdududx\frac{dh}{dx} = \frac{dg}{du} \cdot \frac{du}{dx}
  • h(x)h(x) — the whole composition, h(x)=g(f(x))h(x) = g(f(x)).
  • uu — the inner value, u=f(x)u = f(x), that feeds into gg.
  • dgdu\frac{dg}{du} — the outer function's slope, evaluated at the inner value uu.
  • dudx\frac{du}{dx} — the inner function's slope with respect to xx.
  1. Multiply the two local slopes

    In words: the combined rate of change is the outer function's slope (at the inner value uu) multiplied by the inner function's slope. Two local slopes, multiplied, give the slope of the whole chain.

Play
u = f(x) = 3.00

↓ feeds into g(u)

f'(x) = 2.00, g'(u) = 6.00 → dh/dx = 12.00

Drag the top point and watch both slopes update — then check that their product really does track dh/dxdh/dx as you move.

Worked example

Here f(x)=2x+1f(x) = 2x + 1 and g(u)=u2g(u) = u^2, so h(x)=(2x+1)2h(x) = (2x+1)^2.

  1. Find the inner value at x = 1
    u=f(1)=2(1)+1=3u = f(1) = 2(1) + 1 = 3
  2. Find both local slopes
    g(u)=2u=6,f(x)=2g'(u) = 2u = 6, \qquad f'(x) = 2
  3. Multiply them (the chain rule)
    dhdx=6×2=12\frac{dh}{dx} = 6 \times 2 = 12
  4. Check directly, a different route

    Expand the square first: (2x+1)2=(2x+1)(2x+1)=4x2+2x+2x+1=4x2+4x+1(2x+1)^2 = (2x+1)(2x+1) = 4x^2+2x+2x+1 = 4x^2+4x+1.

    Differentiate term by term:

    • ddx[4x2]=8x\frac{d}{dx}[4x^2]=8x
    • ddx[4x]=4\frac{d}{dx}[4x]=4
    • ddx[1]=0\frac{d}{dx}[1]=0

    Adding those up: h(x)=8x+4h'(x)=8x+4.

    Evaluating at x=1x=1: h(1)=8(1)+4=12h'(1) = 8(1)+4 = 12 — same answer, two routes.

Checkpoint

Drag the top curve’s point until dh/dx reads 20.

u = f(x) = 3.00

↓ feeds into g(u)

dh/dx = g'(u) · f'(x) = 12.00
Drag the point to try it
Summary
dhdx=dgdududx\frac{dh}{dx} = \frac{dg}{du} \cdot \frac{du}{dx}

To differentiate a composition, multiply the slopes of each stage. This is the entire mechanism behind backpropagation: a network is just a long chain of compositions, and training multiplies slopes all the way back through it.