Math Foundations — Lecture 10

Bradley–Terry preference model, RM loss, RLHF/PPO objective with KL penalty, clipped surrogate, DPO loss derivation.

Notation

\(x\)Input prompt.

\(y\)Model completion (sequence of tokens).

\(y_w \succ y_l\)"Winner beats loser": human ranks completion \(y_w\) above \(y_l\) for the same prompt.

\(\pi_{\text{SFT}}\)Supervised fine-tuned reference policy (frozen anchor).

\(\pi_\theta\)Trainable policy being optimised.

\(r_\phi(x,y)\)Scalar reward model with parameters \(\phi\).

\(\sigma(z)\)Logistic sigmoid \(1/(1+e^{-z})\).

\(\beta\)KL coefficient — fluency vs reward trade-off.

Bradley–Terry preference model

Model the probability that \(y_w\) is preferred to \(y_l\) by a sigmoid of the reward gap:

\[P(y_w \succ y_l \mid x) = \sigma\!\bigl(r_\phi(x,y_w) - r_\phi(x,y_l)\bigr) = \frac{1}{1+\exp\!\bigl(r_\phi(x,y_l)-r_\phi(x,y_w)\bigr)}.\]

Only score differences are identifiable: \(r' = r + c\) produces an identical likelihood, so the reward model is unique only up to an additive constant.

Reward model loss (negative log-likelihood)

\[\mathcal{L}_{\text{RM}}(\phi) = -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}_{\text{pref}}}\!\left[\log\sigma\!\bigl(r_\phi(x,y_w) - r_\phi(x,y_l)\bigr)\right].\]

Equivalent to logistic regression on the score gap with labels always "winner". Minimising this NLL fits \(r_\phi\) so that preferred completions score higher.

Quick Check: If you shift every reward by a constant \(c\), the Bradley–Terry loss…

is unchanged — only differences matter

scales by \(e^c\)

becomes undefined

RLHF objective with KL anchor

After the reward model is trained, freeze \(r_\phi\) and optimise the policy:

\[\max_{\theta}\;\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_\theta(\cdot\mid x)}\!\left[r_\phi(x,y)\;-\;\beta\,\mathrm{KL}\!\bigl(\pi_\theta(\cdot\mid x)\,\|\,\pi_{\text{SFT}}(\cdot\mid x)\bigr)\right].\]

The KL term is implemented in practice as a per-token penalty on the log-ratio \(\log\pi_\theta(y_t\mid x,y_{<t}) - \log\pi_{\text{SFT}}(y_t\mid x,y_{<t})\), folded into the per-step reward used by PPO.

Why the KL term?

Reward hacking: with no anchor, \(\pi_\theta\) can drift into degenerate text that the imperfect \(r_\phi\) over-scores.
Mode collapse: a pure reward-max objective concentrates probability on one or two completions; KL keeps the distribution spread.
Fluency anchor: \(\pi_{\text{SFT}}\) already speaks like a normal LM; we only want a small corrective shift toward preferred behaviour.

The optimal solution to the KL-regularised reward objective is a tilted reference policy:

\[\pi^*(y\mid x) = \frac{1}{Z(x)}\,\pi_{\text{SFT}}(y\mid x)\,\exp\!\left(\frac{1}{\beta}\,r_\phi(x,y)\right).\]

Reward acts as a log-density tilt of strength \(1/\beta\); large \(\beta\) → stay close to SFT, small \(\beta\) → chase reward.

PPO clipped surrogate objective

Define the importance ratio between the current policy and the policy used to collect rollouts:

\[\rho_t(\theta) = \frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)},\qquad \hat A_t = \text{advantage estimate}.\]

Vanilla policy gradient maximises \(\mathbb{E}[\rho_t \hat A_t]\) but can blow up if \(\rho_t\) drifts far from 1. PPO clips:

\[L^{\text{CLIP}}(\theta) = \mathbb{E}_t\!\left[\min\!\bigl(\rho_t(\theta)\,\hat A_t,\;\mathrm{clip}\!\bigl(\rho_t(\theta),\,1-\epsilon,\,1+\epsilon\bigr)\,\hat A_t\bigr)\right].\]

With \(\epsilon\) typically 0.1–0.2. The \(\min\) picks the more conservative of the two terms, so once the ratio leaves \([1-\epsilon, 1+\epsilon]\) in the helpful direction, the gradient is clipped — limiting any single update step. This is what makes PPO "proximal".

Quick Check: The clip in PPO's surrogate exists to…

stop any one update from moving the policy too far from \(\pi_{\theta_{\text{old}}}\)

replace the value function

make sampling differentiable

Direct Preference Optimisation (DPO)

Start from the closed-form optimum of the KL-regularised reward objective:

\[\pi^*(y\mid x) = \frac{1}{Z(x)}\,\pi_{\text{SFT}}(y\mid x)\,\exp\!\left(\frac{1}{\beta} r(x,y)\right).\]

Invert to express the implicit reward in terms of the policy:

\[r(x,y) = \beta\,\log\frac{\pi^*(y\mid x)}{\pi_{\text{SFT}}(y\mid x)} + \beta\,\log Z(x).\]

Plug into the Bradley–Terry likelihood. The intractable \(Z(x)\) cancels because it appears for both \(y_w\) and \(y_l\) at the same prompt, leaving the DPO loss:

\[\mathcal{L}_{\text{DPO}}(\theta) = -\mathbb{E}_{(x,y_w,y_l)}\!\left[\log\sigma\!\left(\beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\text{SFT}}(y_w\mid x)} - \beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\text{SFT}}(y_l\mid x)}\right)\right].\]

Train \(\pi_\theta\) directly on preference pairs — no reward model, no PPO sampling loop. Two model passes per pair: score \(y_w\) and \(y_l\) under both \(\pi_\theta\) and \(\pi_{\text{SFT}}\).

Quick Check: In DPO, the reward model is…

replaced by an implicit reward expressed through \(\pi_\theta\) and \(\pi_{\text{SFT}}\)

trained jointly with the policy

still used, but on different data

RLHF (PPO) vs DPO at a glance

RLHF + PPOTrain RM → sample from \(\pi_\theta\) → score with RM → KL-penalised PPO step. On-policy, exploration-friendly, infrastructure-heavy.

DPOOne classification-style loss on preference pairs; no RM, no sampling loop. Off-policy, simpler, no on-policy exploration.

Both share the Bradley–Terry preference model and a KL-style anchor to \(\pi_{\text{SFT}}\) controlled by \(\beta\). Both inherit the limitations and biases of the preference data, and Lee et al. (2024) show that DPO de-toxification can be brittle — internal toxic feature vectors persist even when outputs no longer expose them.