What is retrieval-augmented generation (RAG)?

What is retrieval-augmented generation?

Retrieval-augmented generation, or RAG, is a method that gives a language model documents to read at the moment it answers a question. It does not rely only on facts baked into its parameters during training. The term comes from Lewis et al. (2020), who built RAG models combining a pretrained sequence-to-sequence generator with a dense vector index of Wikipedia. That index is accessed through a neural retriever.

The idea is to split knowledge into two stores. One is parametric memory: whatever the model learned during training, encoded in its weights. The other is non-parametric memory: an external corpus of text, such as a Wikipedia dump, that can be searched at answer time. Lewis et al. (2020) found this combination let their models generate more specific, diverse and factual answers than a generator working from parameters alone.

How does the retriever find documents?

The retriever in Lewis et al. (2020) is Dense Passage Retriever, or DPR, a bi-encoder. One BERT-based model turns the query into a vector, another turns each document into a vector, and relevance is the inner product of the two. Finding the best-matching documents among millions is a Maximum Inner Product Search problem. It is solved approximately and in sub-linear time using FAISS, a vector search library, with a Hierarchical Navigable Small World index for speed.

Wikipedia was split into 21 million 100-word chunks, each embedded once to build the index. REALM (Guu et al., 2020) uses the same inner-product retrieval idea but trains its retriever during language model pretraining itself. It rewards a retrieved document whenever it improves the model's ability to predict a masked word. Both papers call this vector-based matching dense retrieval, to distinguish it from older sparse, word-overlap methods.

Dense retrieval vs. BM25

BM25 is a classic sparse retrieval method that scores documents by word overlap with the query, with no learning involved. Lewis et al. (2020) tested replacing their dense DPR retriever with a fixed BM25 system. Dense retrieval won on most tasks, especially open-domain question answering, where it was, in their words, crucial.

BM25 had one exception: on FEVER, a fact-verification task, it performed best of all. The authors attribute this to FEVER claims being heavily entity-centric and so well suited to word overlap. That split result is itself a useful rule of thumb: dense retrieval tends to win when relevance depends on meaning rather than shared vocabulary.

RAG-Sequence vs. RAG-Token

Lewis et al. (2020) proposed two ways to combine retrieved documents with generation. RAG-Sequence picks one retrieved document and uses it for every token of the generated answer, marginalising, or averaging, over the top-k documents only at the sequence level. RAG-Token can draw a different document for each token it generates, letting a single answer blend content from several sources.

On open-domain question answering, RAG-Sequence scored slightly higher: 44.5 exact match on Natural Questions versus 44.1 for RAG-Token, both state of the art at the time. But RAG-Token did better at Jeopardy-style question generation, a task where answers often combine two separate facts. It could pull from more than one document within a single output.

When does RAG beat a longer context?

Neither paper compares RAG against simply stuffing documents into a longer context window, since long-context models of that scale were not yet common. But their results point to RAG's specific advantages. Knowledge in a retrieval index can be swapped without retraining. Lewis et al. (2020) rebuilt their index from a 2016 Wikipedia dump and a 2018 one.

Their model answered 'who is the president of X' questions correctly 70% and 68% of the time respectively, when the index matched the question's year. That dropped to 12% and 4% when it did not. Hot-swapping is the core case for retrieval over a longer prompt: updating a document index is far cheaper than retraining or re-prompting with an ever-larger context. It also keeps answers traceable to a specific passage rather than to opaque weights.

REALM's authors make a related point about scale. Their 330-million-parameter model beat an 11-billion-parameter closed-book T5 model on open-domain question answering. They argue that external, searchable text can substitute for parameters a model would otherwise need to memorise facts.

What are the trade-offs and limits?

RAG still depends on what the corpus contains. Lewis et al. (2020) note that some MS-MARCO questions, such as asking for the weather in a specific town, cannot be answered from Wikipedia alone. The model then falls back on parametric knowledge or guesses. Retrieval quality also matters: freezing the retriever during training, rather than learning it jointly with the generator, hurt results on every task tested.

REALM's authors solve a related engineering problem. A retriever's document index goes stale the moment its embedding parameters update, since the index was built from the old embeddings. Their fix is to rebuild, or refresh, the index asynchronously every several hundred training steps. Letting it go 30 times staler than that measurably hurt accuracy and retrieval recall in their ablations.

Where is RAG used today?

Lewis et al. (2020) evaluated RAG on open-domain question answering, where it set a new state of the art on Natural Questions, TriviaQA, WebQuestions and CuratedTrec. They also tested abstractive question answering using MS-MARCO, Jeopardy-style question generation, and FEVER fact verification, where it scored within 4.3% of specialised pipeline systems despite using no retrieval supervision. REALM was evaluated specifically on open-domain question answering, outperforming prior systems by 4 to 16 percentage points of accuracy across three benchmarks.

Lewis et al. (2020) released their code through the HuggingFace Transformers library, with an interactive demo. That helped the RAG pattern spread into general-purpose question-answering and knowledge-grounded generation systems beyond the original paper's experiments.