Lecture 10: Training — RLHF, Reward Models, Alignment

The gap

Why SFT alone is not enough

• SFT objective = imitate reference tokens: next-token cross-entropy on instruction/response pairs.
• Token-level prediction does not measure task success: "very delicious" and "somewhat delicious" get similar scores even if one is much better.
• Many useful tasks have no single best answer (open-ended writing, advice, summarisation) — graders only know which of two outputs they prefer.
• SFT models may stay helpful but drift from being honest / harmless — that gap is "alignment".
• Solution: optimise directly against human preferences via reinforcement learning (RLHF).

RecallName two reasons next-token loss is a poor stand-in for "human-preferred output".

Three stages

The RLHF pipeline

• (1) Instruction tuning (SFT): supervised fine-tune base LLM on (prompt, response) pairs — gives a competent starting policy \(\pi_{\text{SFT}}\).
• (2) Reward model (RM): collect preference pairs \((y_w, y_l)\) (winner / loser) ranked by human labellers; train a small model \(r_\phi\) that scores completions.
• (3) RL fine-tune: optimise \(\pi_\theta\) with PPO to maximise \(r_\phi\) while staying close to \(\pi_{\text{SFT}}\) via a KL penalty.
• This is what InstructGPT / ChatGPT (Ouyang et al., 2022) and Llama-2 chat (Touvron et al., 2023) actually use.

RecallList the three stages in order and what artefact each one produces.

Easier for humans

Collecting preference data

• Sample two completions \(y_A, y_B\) from \(\pi_{\text{SFT}}\) for the same prompt \(x\); ask a labeller "which is better?". This is much easier than asking them to write the gold answer.
• OpenAI sourcing: plain (labeller-invented tasks), few-shot (labeller writes an instruction + example pairs), user-based (real waitlist use-cases).
• Llama-2 collected on the order of 1M+ human preference comparisons across multiple rounds.
• Data quality dominates quantity — but who the labellers are (demographics, training, instructions) bakes biases into the final model.

RecallWhy ask "A or B?" instead of "write the ideal answer"?

Bradley–Terry

Training the reward model

• Architecture: a small LLM with a scalar head; \(r_\phi(x,y)\in\mathbb{R}\) scores a completion \(y\) under prompt \(x\).
• Bradley–Terry preference model: \[P(y_w \succ y_l \mid x) = \sigma\!\bigl(r_\phi(x,y_w) - r_\phi(x,y_l)\bigr).\]
• Loss: negative log-likelihood of observed preferences, \[\mathcal{L}_{\text{RM}}(\phi) = -\mathbb{E}_{(x,y_w,y_l)}\!\left[\log\sigma\bigl(r_\phi(x,y_w) - r_\phi(x,y_l)\bigr)\right].\]
• Only differences matter — reward is identified up to an additive constant.
• Outputs are scalars, not probabilities of "correct"; the RM tells PPO relative quality.

RecallWhy is the reward model defined only up to a constant?

RL step

PPO fine-tuning with a KL anchor

• Treat each generated token as an RL action; the reward for a full response \(y\) is \(r_\phi(x,y)\), distributed back to tokens.
• Objective: \[\max_\theta\;\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_\theta}\!\left[r_\phi(x,y) - \beta\,\mathrm{KL}\!\bigl(\pi_\theta(\cdot\mid x)\,\|\,\pi_{\text{SFT}}(\cdot\mid x)\bigr)\right].\]
• PPO clipped surrogate: let \(\rho_t(\theta)=\pi_\theta(a_t\mid s_t)/\pi_{\theta_{\text{old}}}(a_t\mid s_t)\); update with \[L^{\text{CLIP}}(\theta)=\mathbb{E}\!\left[\min\bigl(\rho_t \hat A_t,\;\mathrm{clip}(\rho_t,1-\epsilon,1+\epsilon)\hat A_t\bigr)\right].\] Clipping stops over-large policy updates that could collapse the policy.
• Why the KL penalty against \(\pi_{\text{SFT}}\)?
  • Prevents reward hacking — drifting into high-reward gibberish the RM mis-scores.
  • Prevents mode collapse — the policy starving its output distribution onto one answer.
  • Keeps language fluent — \(\pi_{\text{SFT}}\) is the linguistic anchor.
• Note: this is not the textbook RL loop with an environment; the "environment" is just the prompt distribution and a frozen RM.

RecallWhat single term in the PPO objective stops the policy from hacking the reward model?

Skip the RM

Direct Preference Optimisation (DPO)

• Insight (Rafailov et al., 2023): under the Bradley–Terry assumption, the optimal RLHF policy has a closed-form relationship to the reward, \(r(x,y) = \beta\log\frac{\pi^*(y\mid x)}{\pi_{\text{SFT}}(y\mid x)} + \text{const}\).
• Substitute back — the reward model disappears. Train \(\pi_\theta\) directly with a classification-style loss on preference pairs: \[\mathcal{L}_{\text{DPO}} = -\mathbb{E}\!\left[\log\sigma\!\left(\beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\text{SFT}}(y_w\mid x)} - \beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\text{SFT}}(y_l\mid x)}\right)\right].\]
• No reward model, no PPO, no sampling loop — just supervised-style training.
• Trade-off: avoids an unreliable RM, but inherits the same brittle preference data; cannot easily exploit on-policy exploration.

RecallWhat does DPO replace from the standard RLHF pipeline?

Failure modes

Risks: reward hacking, brittleness, biases

• Reward hacking: policy finds prompts/styles the RM over-scores but humans dislike — KL anchor mitigates, doesn't eliminate.
• Sycophancy: labellers prefer agreeable answers, so the RM rewards them; model learns to agree even when wrong.
• Mode collapse: reward-maximisation can drain output diversity onto a few high-reward templates.
• Preference-data bias: who labels matters — demographics, training, English-only data → biased "alignment".
• Brittle alignment (Lee et al., 2024): DPO de-toxified models still contain the toxic feature vectors internally — the objective only enforces the minimal change needed to flip output behaviour.
• RLVR (verifiable rewards) sidesteps human labels for code/maths, but "verifiable" \(\ne\) "correct"; hard to extend to open-ended tasks.

RecallDistinguish reward hacking from sycophancy with a one-line example each.

Lab

Workshop preview & materials

• Workshop preview (next session): Pinecone vector database and RAG set-up; collect API keys ahead of time.
• Lecture slides: chapters/chapter10/materials/lecture-10.pdf.
• Further reading: Ouyang et al. (InstructGPT, 2022); Touvron et al. (Llama-2, 2023); Rafailov et al. (DPO, 2023); Lee et al. (toxicity-vector brittleness, 2024); the RLHF Book, chapter on RLVR.

Quick practice

Problem cards