Group Relative Policy Optimization (GRPO / DeepSeek-R1)

Five sampled responses to one math problem, each with its own binary verifiable reward from the previous chapter. Toggle to the group-relative advantage and watch it appear — computed entirely from these five numbers, no separate network trained anywhere.

GRPO (short for Group Relative Policy Optimization) samples a GROUP of $G$ responses to the same prompt, then normalizes each one's reward against that group's own statistics:

• $A_i$ — the advantage assigned to sampled response $i$.
• $r_i$ — response $i$'s own reward (e.g. RLVR's binary verifiable reward from the previous chapter).
• $\text{mean}(r_1,\dots,r_G)$, $\text{std}(r_1,\dots,r_G)$ — the mean and standard deviation of all $G$ rewards in the group.

1. The baseline comes from the samples themselves

   PPO's baseline is a learned prediction of expected return. GRPO's baseline is just the group's own mean — no network, no separate training signal, no value-function bias to worry about.

2. The scale comes from the samples too

   Dividing by the group's standard deviation keeps the advantage's scale comparable across prompts of very different difficulty:

   • A group where everyone agrees produces small, tightly-scaled advantages
   • A group with wildly different rewards produces larger ones
3. No critic network, anywhere

   That's GRPO's entire pitch: PPO's clipped policy-gradient update, unchanged, but fed an advantage computed in closed form from one batch of samples instead of a trained critic's guess.

Same formula, two different groups. On the easy problem, four of five responses are correct — being the one that's wrong is expensive: advantage $-2.0$, far below the four correct responses' modest $+0.5$ each. On the hard problem, only one of five is correct — now being right is what's rare and valuable: advantage $+2.0$, while each of the four wrong responses only drops to $-0.5$. The exact same rewards, interpreted relative to two different groups, produce two very different training signals.

Five sampled responses to an easy problem, rewards $[1, 1, 1, 1, 0]$ — four correct, one wrong:

1. Group statistics

   • Mean $= (1+1+1+1+0)/5 = 4/5 = 0.8$
   • This is a Bernoulli group, so variance $= p(1-p) = 0.8 \times 0.2 = 0.16$
   • $\text{std} = \sqrt{0.16} = 0.4$
2. Advantage for a correct response

   $A = (1 - 0.8)/0.4 = 0.2/0.4 = 0.5$.

3. Advantage for the one wrong response

   $A = (0 - 0.8)/0.4 = -0.8/0.4 = -2.0$ — four times the magnitude of the reward for getting it right, purely because being wrong was rare in this group.