What is a context window? RoPE and the "lost in the middle" effect

What is a context window?

A context window is the number of tokens a language model can take as input at once. It is the text the model can actually see when it generates a response. Transformer language models were often trained with context windows of just 512 to 2,048 tokens, according to the Lost in the Middle paper (Liu et al., 2023). Faster GPUs and algorithmic advances have since pushed that to 4,096, 32,000, and even 100,000 tokens, the paper notes.

The limit exists because the standard transformer's self-attention mechanism requires memory and compute that grow quadratically with sequence length, per Liu et al. (2023). Every extra token a model must hold in context makes every other token's attention computation more expensive. That cost is why context windows did not simply start large. Extending them has required new techniques, not just more hardware.

How does a model know where each token is?

Self-attention on its own is position-agnostic. It has no built-in sense of which word came first, according to the RoFormer paper (Su et al., 2021), which cites Yun et al. (2020) for this claim. So transformers need a separate mechanism, called position encoding, to tell the model where each token sits in the sequence.

The original transformer paper (Vaswani et al., 2017) added a fixed sinusoidal vector to each token's embedding based on its position, the RoFormer paper explains. Other approaches instead encode position as a relative distance between pairs of tokens. This includes work by Shaw et al. (2018), Dai et al. (2019), and Raffel et al. (2020). Their method builds the distance into the attention calculation itself, rather than adding it to the embedding beforehand.

What is RoPE, and how does it differ?

Rotary Position Embedding, or RoPE, is a method introduced in the RoFormer paper (Su et al., 2021). It encodes a token's position by rotating its query and key vectors by an angle proportional to that position, rather than adding a position vector to it. The rotation is multiplicative. When two rotated vectors are compared in attention, the result depends only on the relative distance between their positions, not on either one's absolute position.

Su et al. (2021) report three properties that follow from this design. RoPE works at any sequence length, since a rotation angle can be computed for any position. It gives a long-term decay: the inner product between two tokens' vectors shrinks as the distance between them grows, matching the intuition that distant tokens should be less tightly linked. And because the rotation preserves vector norms, RoPE can be combined with efficient linear-attention variants, which the paper demonstrates alongside the Performer architecture.

Why do models lose information in the middle of long inputs?

Having a large context window does not mean a model uses it evenly. Liu et al. (2023) tested models including GPT-3.5-Turbo, Claude-1.3, MPT-30B-Instruct and LongChat-13B (16K) on tasks requiring them to find one relevant document or key-value pair hidden among distractors. Accuracy traced a U-shaped curve. Performance was highest when the relevant information sat at the very start of the input, a primacy bias, or the very end, a recency bias, and fell sharply when it sat in the middle.

The drop was substantial. GPT-3.5-Turbo's multi-document question-answering accuracy, when the answer was buried in the middle of 20 or 30 documents, fell below its closed-book accuracy of 56.1%. The model did worse with the answer in front of it than guessing from memory alone, Liu et al. (2023) found. The same U-shape showed up even in a synthetic key-value retrieval task built from random identifiers, with no natural language to confuse the model. So the effect is not just about meaning; it shows up in raw lookup too.

Liu et al. (2023) traced part of the cause to architecture. Encoder-decoder models such as Flan-T5-XXL and Flan-UL2 were far more robust to the position of relevant information, but only within the sequence lengths they were trained on. Beyond that, they showed the same U-shaped drop. Placing the query both before and after the data, a technique the paper calls query-aware contextualization, nearly eliminated the effect on the key-value task, but barely changed results on multi-document question answering.

Does a bigger context window always help?

Not reliably, according to Liu et al. (2023). Giving GPT-3.5-Turbo or Claude-1.3 access to documents in a 16K- or 100K-token window produced performance nearly identical to their shorter-context versions, when the shorter window could already fit the input. In an open-domain question-answering case study, reader accuracy saturated long before retrieval recall did. Pulling in 50 documents instead of 20 improved GPT-3.5-Turbo's accuracy by only about 1.5 percentage points, while adding to the length, latency and cost of each query.

The paper's conclusion is a caution for anyone building retrieval-augmented systems: stuffing more retrieved passages into the prompt is not a free upgrade. Reranking to push the most relevant passage toward the start of the context may do more for accuracy than widening the window. So may simply retrieving fewer documents, Liu et al. (2023) suggest.