Proximal Policy Optimization (PPO)

Drag the ratio $r$ — how much more (or less) likely the new policy makes the action the old policy took. The dashed line is the plain, unlimited objective; the bold line is PPO's version. They agree almost everywhere, then the bold line goes flat once $r$ crosses $1+\varepsilon$: past that point, moving further doesn't earn any more reward in the objective, no matter how far the policy actually moves.

With probability ratio $r(\theta) = \frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)}$ and advantage estimate $A$, PPO's clipped surrogate objective is:

• $r(\theta)$ — the probability ratio: how much more (or less) likely the new policy makes the action taken, relative to the policy that generated the data.
• $A$ — the advantage estimate for that action (e.g. the actor-critic TD error from last chapter).
• $\varepsilon$ — the clip range, typically around $0.2$.
• $\text{clip}(r, 1{-}\varepsilon, 1{+}\varepsilon)$ — $r$ clamped into $[1{-}\varepsilon,\ 1{+}\varepsilon]$.

1. r = 1 means 'no change yet'

   $r(\theta)=1$ exactly when the new policy assigns the same probability to that action as the old one did — the starting point before any update has moved anything.

2. The min makes it a pessimistic bound

   Taking the $\min$ of the raw and clipped terms means the objective can never be more generous about a large policy change than the honest, unclipped number — it can only be more conservative.

3. The clip only bites on one side

   For a positive advantage, increasing $r$ past $1+\varepsilon$ gets capped — no extra credit for over-committing to a good action. But decreasing $r$ below $1-\varepsilon$ isn't clipped at all: the $\min$ just picks the (already smaller) unclipped term, so nothing stops the policy from moving a good action's probability down as far as the raw gradient wants. The clip specifically prevents exploiting an advantage by moving too far in the rewarding direction, not the punishing one.

Step through ratios from $0.7$ up to $1.5$. Below $1+\varepsilon=1.2$, the bold and dashed lines sit exactly on top of each other — nothing is being clipped. Past $1.2$, the bold line stays flat at $1.2 \times A$ no matter how much further $r$ climbs.

With $A=1$ and $\varepsilon=0.2$ (so the clip range is $[0.8, 1.2]$):

1. r = 0.9 — inside the clip range, nothing happens

   $\text{clip}(0.9) = 0.9$, so both terms equal $0.9 \times 1 = 0.9$. The objective is just the plain policy gradient here — the clip hasn't engaged at all.

2. r = 0.7 — below 1-epsilon, but still not clipped

   $\text{clip}(0.7) = 0.8$, giving a clipped term of $0.8$. But the raw unclipped term is $0.7 \times 1 = 0.7$, which is smaller — so $\min(0.7, 0.8) = 0.7$ wins. The clip exists but never binds here, exactly the asymmetry described above.

3. r = 1.3 — past 1+epsilon, and now it bites

   $\text{clip}(1.3) = 1.2$, giving a clipped term of $1.2$. The unclipped term is $1.3 \times 1 = 1.3$. Now $\min(1.3, 1.2) = 1.2$ — the objective is held down to $1.2$, even though the policy actually moved the ratio to $1.3$. That gap is the clip doing its job: refusing to reward the update any further for moving past the trust region.