What is alignment in AI, and how is it done?

What does 'alignment' mean?

Alignment is the practice of training a language model so its behaviour matches what users actually intend. That's different from matching what the training objective merely rewards. The InstructGPT paper (Ouyang et al., 2022) points out that a model trained only to predict the next word on internet text has a different goal from one trained to follow instructions helpfully and safely. It calls that gap misalignment.

Ouyang et al. borrow a three-part test from Askell et al. (2021). An aligned model should be helpful, meaning it solves the user's task. It should be honest, meaning it does not fabricate or mislead. And it should be harmless, meaning it avoids physical, psychological or social harm. The Constitutional AI paper (Bai et al., 2022) adopts the same helpful-honest-harmless framing for its own work on reducing harm without human labels.

How does instruction following get trained?

Ouyang et al. built InstructGPT from GPT-3 in three steps. First, about 40 hired contractors wrote demonstrations of desired responses to prompts. These were used to fine-tune GPT-3 by supervised learning. Second, labelers ranked sets of four to nine model outputs per prompt. These rankings trained a reward model to predict which output a person would prefer.

Third, the supervised model was fine-tuned further with reinforcement learning, using the proximal policy optimization (PPO) algorithm, to maximise the reward model's score. A per-token penalty kept the policy close to the original model so it would not over-optimise against the reward model. The paper also tried mixing in gradients from the original pretraining data, a variant it calls PPO-ptx. That variant recovered most of the performance the model otherwise lost on public NLP benchmarks during fine-tuning.

What are constitutional methods?

Constitutional AI, described by Bai et al. (2022), trains a model to be harmless using a short written list of principles instead of human labels marking harmful outputs. In a supervised stage, a helpful model critiques its own response against a randomly chosen principle, then revises it. Repeating this against red-team prompts yields a dataset used to fine-tune the model, producing what the paper calls an SL-CAI model.

In the reinforcement learning stage, the same model generates pairs of responses. An AI feedback model, not a human, picks the more harmless one according to a principle, formatted as a multiple-choice question. Those AI-generated preferences train a preference model, mixed with human helpfulness labels. The policy is then optimised against it with reinforcement learning, a method the paper calls reinforcement learning from AI feedback (RLAIF). Bai et al. report that chain-of-thought reasoning, where the feedback model reasons step by step before answering, improved its judgments.

How is alignment tested?

Ouyang et al. relied mainly on human preference ratings. Labelers judged which of two model outputs they preferred, and judged overall quality on a 1-7 scale. On the firm's API prompt distribution, outputs from the 175-billion-parameter InstructGPT were preferred to those of the 175-billion-parameter GPT-3 85% of the time. Outputs from a 1.3-billion-parameter InstructGPT, over 100 times smaller, were still preferred to the full-size GPT-3.

The same paper measured truthfulness on the TruthfulQA benchmark, where InstructGPT gave truthful and informative answers about twice as often as GPT-3. It measured hallucination on closed-domain tasks, where InstructGPT invented unsupported information 21% of the time versus 41% for GPT-3. It measured toxicity on RealToxicityPrompts, finding about 25% fewer toxic outputs when prompted to be respectful. But it found no improvement on the Winogender and CrowS-Pairs bias benchmarks.

Bai et al. tested their models differently, using crowdworker comparisons converted into Elo scores for helpfulness and harmlessness separately. They report that RL-CAI models reached harmlessness at least equal to models trained on human harm labels, without a large cost to helpfulness. On a set of 438 binary comparison questions testing helpful, honest and harmless judgment, large pretrained models using chain-of-thought reasoning approached the accuracy of a preference model trained on human feedback.

What are the trade-offs?

Ouyang et al. call the drop in performance on certain public NLP datasets — including SQuAD, DROP, HellaSwag and French-to-English translation — an "alignment tax." That's the cost the fine-tuning procedure imposes on tasks the model was not directly optimised for. Mixing in pretraining gradients reduced, but did not eliminate, that tax.

Bai et al. describe a related tension in their earlier human-feedback work. Training a model to be harmless tended to make it evasive, refusing to engage even with borderline questions, which crowdworkers judged as less helpful. Their constitutional approach, with crowdworkers instructed to prefer non-evasive answers, aimed to reduce that tension rather than eliminate it. SL-CAI models were still less helpful than the RL-trained ones, even as they were more harmless than a purely helpful model.

Does alignment generalise beyond its training data?

Ouyang et al. found InstructGPT's preference held up with held-out labelers, who did not contribute training data, at roughly the same rate as the labelers who did. The model also showed what the paper calls promising generalisation. It could follow instructions for summarising and answering questions about code, and sometimes followed instructions in other languages, despite very little such data in fine-tuning.

Both papers caution that their results describe a specific group of labelers and a specific set of written principles, not a universal standard. Ouyang et al. state plainly that their procedure aligns GPT-3 to the preferences of their labelers and researchers, not to any broader notion of human values.