Study maps where language models settle on an answer
Internal states cluster by answer, not by question, a new paper finds
The facts
- Researcher Shailesh Rana tested four open models — Llama-3.1-8B, Qwen2.5-7B, Gemma-2-9B and Llama-3.2-1B — on 2,256 recurring factual questions
- In every model, the share of questions whose internal state sits nearest its answer's centroid jumps sharply at one layer, between 47% and 71% through the network
- Centroids track the answer rather than the question: a country's centroid learned from citizenship questions also identifies that country in questions about towns and films
- After the jump, moving a wrongly answered question's state to the correct centroid flips the model to the right answer 83%-97% of the time, versus 2%-4% for a random move of equal size
A new paper by Shailesh Rana tracks where a language model's internal state lands once it has settled on a factual answer. It was posted to arXiv and revised in October. Rana tested four open models on 2,256 questions with recurring answers, such as the country a person holds citizenship in. He recorded the model's state at every layer. Questions sharing an answer converge on a common region of that state space, which the paper calls an answer centroid.
The convergence is not gradual. In every model tested, the proportion of questions whose state sits closest to its own answer's centroid jumps sharply at a single layer. That jump lands between 47% and 71% of the way through the network, depending on the model. The centroids organise around the answer itself rather than the question that produced it. A centroid built from citizenship questions about people also identifies the same country when the model is asked about towns or films. That suggests a shared representation, reused across question types.
The paper also shows the model actually uses these centroids, rather than merely passing near them. Past the convergence layer, forcing a wrongly answered question's state toward the correct answer's centroid produces the right answer in 83% to 97% of cases. A random vector of the same size works in just 2% to 4% of cases. Before the jump, neither kind of move works reliably. The paper further reports that near misses are detectable from the very layer where centroids first form; these are cases where a model answers incorrectly but would pick the right option if given multiple choices. The correct centroid is already among the nearest at that point. Rana says this early signal is roughly as informative as the right answer's rank in the model's final output, and more informative than the model's own confidence score.
Why it matters
For anyone building interpretability or steering tools, this gives a concrete, causal lever: answer centroids can be identified and edited layer by layer rather than guessed at post hoc. It also hints that hallucination detection could move earlier in the forward pass than output logits or confidence scores allow. That could cut inference cost for safety checks.
Primary source
arXiv cs.AI — arxiv.orgWritten from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.