RLHF and preference tuning, explained

What is RLHF?

Reinforcement learning from human feedback, or RLHF, trains a model to behave the way people want. It uses human preference judgments instead of a hand-written reward function. Rather than scoring outputs on an absolute scale, people compare pairs of outputs and say which one they prefer. A separate reward model learns to predict those preferences, and the main model is then optimised against that reward.

The idea began outside language entirely. Christiano et al. (2017) trained simulated robots and Atari-playing agents by showing contractors short video clips and asking which clip looked more like the intended behaviour. That paper showed a policy could learn to backflip or drive with traffic from about an hour of human comparisons. Nobody wrote a reward function for 'backflip' at all.

How does a reward model work?

A reward model is a network trained to output a single score for a given input and response. It is fit so that preferred responses score higher than rejected ones. Christiano et al. (2017) framed this with the Bradley-Terry model. They treated the gap in predicted reward between two trajectory segments like an Elo rating gap in chess: the bigger the predicted gap, the more confidently one should be preferred.

Ouyang et al. (2022), who built InstructGPT, applied the same idea to language. They started from a 175-billion-parameter GPT-3 model and collected labeler demonstrations for supervised fine-tuning. Labelers then ranked four to nine model outputs per prompt. A 6-billion-parameter reward model was trained on those rankings using a cross-entropy loss over all pairwise comparisons within a ranked set. They found this far more stable than training on shuffled individual comparisons.

How is the policy optimised against the reward model?

Once a reward model exists, a second model, the 'policy', is fine-tuned to produce outputs that score well on it. Ouyang et al. (2022) did this with proximal policy optimization, or PPO. They treated each prompt-response pair as a one-step bandit problem: generate a response, score it with the reward model, update the policy, done.

A plain reward-maximising policy will drift toward whatever fools the reward model. So Ouyang et al. added a per-token penalty based on how far the policy's output distribution strayed from the original supervised model, measured by KL divergence. They called the result PPO-ptx when pretraining-data gradients were mixed in too. This fixed performance drops on public NLP benchmarks without hurting labeler preference scores. Outputs from their 1.3-billion-parameter InstructGPT were preferred to the 175-billion-parameter GPT-3's outputs by labelers, despite having over 100 times fewer parameters.

RLHF vs DPO: why skip the reward model?

Direct Preference Optimization, or DPO, was introduced by Rafailov et al. (2023). It reaches the same goal without training a separate reward model or running reinforcement learning at all. The authors show that the reward-maximising policy implied by the standard RLHF objective has a closed-form solution. That means the preference data can be used to train the language model directly, with a simple binary classification loss.

Rafailov et al. (2023) call this 'your language model is secretly a reward model': the policy itself becomes the implicit reward. There is no sampling from the model during training, and little hyperparameter tuning is needed. In their experiments, DPO matched or beat PPO-based RLHF on controlled sentiment generation, summarisation and single-turn dialogue, using models up to 6 billion parameters. On Reddit TL;DR summarisation, judged by GPT-4, DPO reached a 61% win rate against reference summaries at temperature zero, versus PPO's best of 57%. DPO also stayed more stable as sampling temperature rose.

Where does AI feedback fit in?

Rafailov et al. (2023) note a related line of work: using language models already fine-tuned on human feedback to generate additional synthetic preference data for targeted attributes such as safety. Only weak supervision from a human-written rubric guides the labeling. This is sometimes called RLAIF, feedback from AI rather than people. None of the documents here report its own benchmark results in detail.

What does post-training actually change in the model?

Ouyang et al. (2022) report concrete shifts from applying RLHF to GPT-3. On TruthfulQA, InstructGPT gave truthful and informative answers roughly twice as often as GPT-3. On closed-domain tasks, where the answer should draw only on the given input, InstructGPT invented unsupported information about half as often: a 21% hallucination rate versus GPT-3's 41%. InstructGPT also generated about 25% fewer toxic outputs than GPT-3 when prompted to be respectful. It showed no significant improvement on the Winogender and CrowS-Pairs bias benchmarks, though.

These gains were not free. Ouyang et al. (2022) call the regressions seen on public NLP datasets such as SQuAD, DROP, HellaSwag and WMT 2015 French-to-English translation an 'alignment tax'. Fine-tuning for human preference cost performance elsewhere. Mixing pretraining gradients back into the PPO updates largely closed that gap.