Advanced preference alignment (KTO & SimPO)

The same 3 comparisons DPO's chapter used — "B beats A," "B beats C," "C beats A" — but relabeled as 6 completely independent judgments: each response is just "desirable" or "undesirable," on its own, with no memory of which other response it was compared against. Step through them one at a time.

KTO (short for Kahneman-Tversky Optimization) keeps DPO's reference-relative log-ratio and its logistic loss shape, but scores ONE response at a time instead of a pair:

SimPO (short for Simple Preference Optimization) keeps DPO's pairing, but drops the reference model entirely — the implicit reward is a response's OWN length-normalized log-probability, compared against a fixed target margin $\gamma$:

• $\lambda$ — KTO's sign flip: $+1$ for a desirable response, $-1$ for an undesirable one — the only place pairing information used to live.
• $y$ — the single response being judged, in KTO; $y_w$, $y_l$ — the winning and losing responses of a pair, in SimPO.
• $\pi_{\text{ref}}$ — the frozen reference model DPO and KTO both compare against; SimPO has no such term at all.
• $|y|$ — the response's length in tokens, used to normalize its log-probability so longer responses aren't penalized just for being longer.
• $\gamma$ — SimPO's fixed target reward margin, replacing the role a reference model's baseline would otherwise play.

1. KTO drops PAIRING

   Each of DPO's comparisons becomes two independent examples: the winner labeled desirable, the loser labeled undesirable. Only one response's log-ratio is touched per update — there's no "loser" to simultaneously push down in the same step, because pairing information was thrown away.

2. SimPO drops the REFERENCE MODEL

   Instead of $\log\pi(y) - \log\pi_{\text{ref}}(y)$, SimPO uses $\log\pi(y)$ alone, divided by length. No second frozen copy of the model needs to be kept in memory during training at all.

3. Both keep DPO's core trick

   Neither method needs a separately trained reward model — like DPO, the loss is computed directly on the policy's own log-probabilities.

Both variants still recover roughly DPO's ranking — $B$ preferred, then $C$, then $A$ — using strictly less machinery than DPO started with: KTO drops the paired structure, SimPO drops the reference model. Neither is strictly "better" than DPO; each is a different cost DPO's original formulation was willing to pay, removed.

KTO's very first example, "$B$ is desirable," starting from a uniform policy:

1. At initialization, the policy IS the reference

   $\pi(B) = \pi_{\text{ref}}(B) = 1/3$, so the log-ratio is $0$. With $\lambda=+1$ for "desirable," $z = \beta \cdot 0 = 0$, so $\sigma(z) = 0.5$.

2. The update touches ONLY B

   $\theta_B \to 0 + 0.5 \times (1-0.5) \times 0.5 \times 1 = 0.125$ — identical in magnitude to DPO's very first update, but $\theta_A$ and $\theta_C$ are untouched here; DPO's paired update would have pushed a loser DOWN by the same amount in the same step.

3. After all 6 examples

   The same one-response-at-a-time update, run through the remaining 5 examples in order:

   • Example 2, "$A$ undesirable": $\pi(A)\approx0.319$ (softmax has already shifted after $B$'s update), log-ratio $=\ln(0.319)-\ln(1/3)\approx-0.043$, $z=(-1)(0.5)(-0.043)\approx0.022$, $\sigma(z)\approx0.505$, update $=0.5(1-0.505)(0.5)(-1)\approx-0.124 \Rightarrow \theta_A\approx-0.124$
   • Example 3, "$B$ desirable" again: $\pi(B)\approx0.376$, log-ratio $\approx0.119$, $z\approx0.060$, $\sigma(z)\approx0.515$, update $\approx+0.121 \Rightarrow \theta_B\approx0.246$
   • Example 4, "$C$ undesirable": $\pi(C)\approx0.316$, log-ratio $\approx-0.053$, $z\approx0.026$, $\sigma(z)\approx0.507$, update $\approx-0.123 \Rightarrow \theta_C\approx-0.123$
   • Example 5, "$C$ desirable": $\pi(C)\approx0.290$, log-ratio $\approx-0.139$, $z\approx-0.069$, $\sigma(z)\approx0.483$, update $\approx+0.129 \Rightarrow \theta_C\approx0.006$
   • Example 6, "$A$ undesirable" again: $\pi(A)\approx0.279$, log-ratio $\approx-0.178$, $z\approx0.089$, $\sigma(z)\approx0.522$, update $\approx-0.119 \Rightarrow \theta_A\approx-0.243$

   Final values:

   • $\theta_B \approx 0.246$
   • $\theta_C \approx 0.006$
   • $\theta_A \approx -0.243$

   The same $B > C > A$ ordering DPO found, recovered without ever using a preference pair.