What is hallucination in language models?

What is hallucination?

Hallucination is when a language model generates text that is nonsensical or unfaithful to the source content it was given, or simply false. The term comes from psychology, where a hallucination is an unreal perception that feels real. The survey by Ji et al. (2022) borrows it because hallucinated text feels fluent and grounded even though it isn't. The problem spans summarization, dialogue, question answering, data-to-text generation, translation and image captioning.

Ji et al. (2022) split hallucinations into two kinds. Intrinsic hallucination contradicts the source directly — a summary that changes a date the source actually gives. Extrinsic hallucination adds content the source neither confirms nor denies. Extrinsic hallucination isn't always wrong; it can add true background knowledge. But most researchers still treat it with caution, since unverifiable claims carry factual risk even when they happen to be accurate.

Why do models state false things confidently?

Ji et al. (2022) trace hallucination to two broad sources: the training data and the model itself. On the data side, many datasets are built by heuristically pairing a source with a target that actually contains extra information. The survey notes that 62% of WikiBio's first sentences include facts absent from the matching infobox. That mismatch trains models to add unsupported content.

On the model side, encoders can learn wrong correlations. Decoders can attend to the wrong part of the input, mixing up facts between similar entities. Decoding strategies that boost diversity, such as top-k sampling, are positively correlated with more hallucination, per Dziri et al. (2021), cited in the survey. Teacher-forced training also creates exposure bias: the model never practises conditioning on its own, possibly erroneous, past outputs the way it must at inference.

A further contributor is parametric knowledge bias. Large pretrained models store knowledge in their weights. Per Longpre et al. (2021), as cited in the survey, they can favour that memorised knowledge over the input actually given to them. TruthfulQA (Lin, Hilton and Evans, 2021) gives this a sharper edge. They show models can produce imitative falsehoods — false answers that are high-likelihood under the training distribution because they mimic common human misconceptions, not because the model failed to learn.

How is hallucination measured?

Ji et al. (2022) describe three families of metrics. Statistical metrics compare word overlap between generated text and source or reference, such as PARENT, which checks entailment against both a table and target text in data-to-text generation. Model-based metrics use a second neural system. These extract facts as subject-relation-object triples and check them, generate questions and compare answers against the source, or use natural language inference to see whether the source entails the generated claim. Human evaluation, scoring or comparing outputs, remains common because automatic metrics correlate poorly with human judgment; Falke et al. (2019), cited in the survey, found hallucinated content in 25% of summaries from systems that scored well on ROUGE, BLEU and METEOR.

TruthfulQA (Lin, Hilton and Evans, 2021) takes a different, benchmark-based approach: 817 questions across 38 categories, written so that some humans would answer them falsely due to a misconception. Answers are human-scored for truthfulness and informativeness. The authors also built an automated judge, a GPT-3-6.7B model finetuned on human-labelled answers, that matched human evaluation 90-96% of the time on held-out models.

Does scale fix hallucination?

No, not by itself. Lin, Hilton and Evans (2021) tested GPT-3, GPT-Neo/J, GPT-2 and a T5-based UnifiedQA model on TruthfulQA. They found the largest models in each family were generally the least truthful, the opposite of the usual pattern where bigger models do better. The largest GPT-Neo/J, for instance, was 17% less truthful than a model sixty times smaller.

The explanation offered is that imitative falsehoods are learned from the training distribution itself, and larger models are better at learning that distribution, misconceptions included. Scaling laws predict that bigger models fit the training distribution more closely. That should reduce falsehoods caused by poor learning but increase ones that are faithfully imitated from human text. The paper's authors therefore argue that scaling up alone is less promising than finetuning with objectives other than pure imitation of web text.

What reduces hallucination?

Ji et al. (2022) group mitigation into two kinds. Data-related methods clean or filter training data to reduce source-target divergence. Modelling-and-inference methods change decoding strategies or how the encoder and decoder attend to the input. TruthfulQA's authors found that prompting GPT-3 with a 'helpful' instruction meaningfully improved truthfulness over a default prompt, though it did not change the rate of true-and-informative answers as much.

Lin, Hilton and Evans (2021) suggest finetuning on examples chosen to demonstrate truthfulness, or reinforcement learning from human feedback. They see these as more promising than scale alone, potentially combined with retrieval from reliable sources. Even their best model, GPT-3-175B with a helpful prompt, was truthful on only 58% of questions against a 94% human baseline. It gave answers that were both false and informative 42% of the time, versus 6% for the human.