RLHF: Bradley-Terry reward modeling

Three candidate responses, no numeric reward for any of them — only three comparisons: "B beats A," "B beats C," "C beats A." Step through fitting a reward score to each response from nothing but those three judgments, watching the scores separate as the fit proceeds.

The Bradley-Terry model turns a pile of pairwise comparisons into a reward score per item: the probability that $i$ beats $j$ is a sigmoid of the difference in their scores,

• $P(i \succ j)$ — the probability that response $i$ beats response $j$ in a head-to-head comparison.
• $i$, $j$ — the two candidate responses being compared.
• $r_i$, $r_j$ — the fitted reward scores for responses $i$ and $j$.

1. Fitting r is ordinary logistic regression

   Fitting $r$ from observed comparisons is exactly Part II's logistic regression, with each comparison contributing one cross-entropy term.

2. The fitted score stands in for the reward

   Once a reward model is fit this way, it can score a response the same way any other trained reward function would — and that score is what stands in for the reward Chapter 4's policy gradient needs but nobody could hand it directly.

The policy starts uniform, with no idea any response is better than any other. After training on the fitted reward model — not on any ground truth, since none exists here — it clearly favors the response humans actually preferred, using the exact same update rule from Chapter 4.

With $A$ fixed at reward $0$ as a reference point (Bradley-Terry scores are only meaningful relative to each other, exactly like softmax logits):

1. First fitting step

   All three scores start at $0$, so every comparison's predicted probability is $\sigma(0)=0.5$ — completely uncertain. Each comparison pushes the winner's gradient down by $(1-p)$ and the loser's up by $(1-p)$, i.e. by $0.5$ each:

   • $B$ beats $A$: pushes $B$ by $-0.5$
   • $B$ beats $C$: pushes $B$ by another $-0.5$ (total $-1.0$), pushes $C$ by $+0.5$
   • $C$ beats $A$: pushes $C$ by $-0.5$ (net $0$ for $C$, since its two pushes cancel)

   Descending: $r_B \leftarrow 0 - 0.5(-1.0) = 0.5$; $r_C \leftarrow 0 - 0.5(0) = 0$ — $B$ wins two comparisons and its score rises, while $C$'s one win and one loss (against parties still at $0$) exactly cancel.

2. After three fitting steps

   Repeating the same gradient step, using the updated rewards each time:

   • After step 2 (starting from $r_B=0.5, r_C=0$): $r_B\to0.8775$, $r_C\to0.061$
   • After step 3 (starting from those): $r_B\to1.18$, $r_C\to0.15$

   $r_A=0$ throughout (fixed as the reference) — the fitted scores recover the true order the comparisons implied ($B$ beat both, $C$ split, $A$ lost both), entirely from three yes/no judgments.

3. Training the policy on that fitted reward

   Feeding these three scores through Chapter 4's exact policy-gradient update, one response at a time, moves the policy from uniform to favoring $B$ with about $46\%$ of the probability mass — the response people actually preferred.