Direct Preference Optimization (DPO)

Apply each comparison. The policy shifts after every single one — no reward model sits in between the preference data and the policy update at all.

For an optimal policy under a KL (Kullback–Leibler)-regularized reward objective, the reward and the policy are related in closed form: $r(y) = \beta \log\big(\pi(y)/\pi_{\text{ref}}(y)\big) + \text{const}$. Substitute that directly into the Bradley-Terry preference loss from Part VII, and the reward model cancels out algebraically, leaving a loss purely in terms of the policy:

• $\mathcal{L}$ — the DPO loss being minimized for a single preference pair.
• $\beta$ — the KL-penalty strength inherited from the RLHF objective; controls how far the policy is allowed to drift from the reference.
• $\pi$ — the policy currently being trained.
• $\pi_{\text{ref}}$ — the frozen reference policy (typically the pre-RLHF model) the trained policy is measured against.
• $y_w$ — the winning (preferred) response in the comparison.
• $y_l$ — the losing (dispreferred) response in the comparison.

1. The reward model's term cancels in the gradient

   Differentiating this with respect to the policy's own parameters, the softmax's $\pi(a)$ term cancels exactly between the winner and loser gradients.

2. What's left is a single-stage update

   One gradient step per preference pair, straight on the policy — no reward model and no separate RL update rule required, simpler than RLHF's two-stage procedure.

Two completely different training procedures — fit-a-reward-model-then-run-RL versus one direct gradient loop — landing on the same ranking from the same three comparisons. Neither path is "more correct"; DPO — short for Direct Preference Optimization — is just a shortcut through algebra that happens to skip a stage RLHF needs.

The very first comparison, "B beats A", starting from a uniform policy:

1. At initialization, the policy IS the reference

   $\pi = \pi_{\text{ref}}$ everywhere, so $\log\pi(y_w)-\log\pi_{\text{ref}}(y_w) = 0$ for every arm. The bracketed term is $0 - 0 = 0$, so $h=\beta \cdot 0 = 0$ and $\sigma(0)=0.5$.

2. The gradient step

   The update is $\theta \to \theta + \eta\,(1-\sigma(h))\,\beta\,(\mathbb{1}[a{=}y_w]-\mathbb{1}[a{=}y_l])$, with learning rate $\eta=0.5$ and $\beta=0.5$. With $\sigma(h)=0.5$, the update factor $(1-\sigma(h))=0.5$ pushes $\theta_B$ up and $\theta_A$ down by the same amount:

   • $\theta_B \to 0 + \underbrace{0.5}_{\eta}\underbrace{(0.5)}_{1-\sigma(h)}\underbrace{(0.5)}_{\beta}(1) = 0.125$
   • $\theta_A \to 0 - 0.125 = -0.125$
3. Nothing else moved

   $\theta_C$ is untouched — this comparison never mentioned C, and DPO's update (like RLHF's) only touches the two arms actually being compared.