What is interpretability in neural networks?
What is interpretability in neural networks?
What is mechanistic interpretability?
Interpretability is the effort to look inside a trained neural network and work out what it is actually computing. It treats the network as something to be read, not a black box that merely produces outputs. Mechanistic interpretability pushes this further. It tries to find the specific internal parts, individual neurons or combinations of them, and the circuits they form, that implement a given behaviour.
The obstacle this runs into early is that single neurons rarely correspond to single, clean ideas. Toy Models of Superposition (Elhage et al., 2022) calls this problem polysemanticity: a neuron that fires for several unrelated concepts at once. That makes reading meaning off any one neuron unreliable.
What is superposition?
Superposition is the phenomenon the paper proposes as the cause of polysemanticity. A network stores more distinct features than it has neurons by packing them into overlapping combinations of neurons, rather than giving each its own. The paper builds a toy model small enough to analyse fully. In it, superposition can be shown to arise when the features a network needs to represent are sparse, meaning most of them are absent or irrelevant on any given input.
Elhage et al. report a phase change in their toy model: as conditions shift, the model switches between representing features individually and packing them into superposition. They also find a surprising link to the geometry of uniform polytopes, regular geometric shapes. This suggests the way features get arranged in superposition is not arbitrary but follows specific geometric structures. The paper further reports evidence connecting superposition to adversarial examples, inputs crafted to fool a model.
Why does this matter for reading a model?
If features are stored in superposition, looking at individual neurons will not reveal the model's true concepts. Any one neuron may be a blend of several unrelated ones. Elhage et al. discuss this as a direct implication for mechanistic interpretability: the field cannot simply inspect neurons and expect them to map onto human-understandable ideas.
This is why interpretability research looks for features and circuits at a different level of description than the raw neuron. A feature, in this framing, is the underlying concept a network represents, which may be spread across many neurons rather than confined to one. A circuit is a set of such features connected together, wired through the network's weights, that together carries out a specific computation.
What reports polysemanticity and superposition, and when?
The account of polysemanticity and superposition described here comes from Toy Models of Superposition (Elhage et al., 2022), published on arXiv in September 2022. It works with small, fully analysable toy models built specifically to isolate the phenomenon, rather than with large production language models.
That scope matters for anyone reading its claims. The phase change, the polytope geometry and the adversarial-example link are all results reported in a controlled, simplified setting. Whether they hold in the same form inside a large deployed model is a separate question. The paper does not claim to answer it.
What is outside this page
Sparse autoencoders, a technique that later interpretability work uses to try to separate superposed neurons back into individual, readable features, are named in this topic's scope. But they are not described in the document this page draws from. So are the practicalities of running interpretability tools against real, large-scale models in production.
A reader deciding whether to invest in interpretability tooling should treat the toy-model results as a theory of why neurons are hard to read. They are not yet a proven method for fixing it.
Key terms
- Polysemanticity
- A neuron that responds to several unrelated concepts at once, making its individual activity hard to interpret.
- Superposition
- A network storing more features than it has neurons by representing them as overlapping combinations of neurons rather than one feature per neuron.
- Feature
- An underlying concept a network represents, which in superposition may be spread across many neurons rather than confined to one.
- Circuit
- A set of connected features, wired through a network's weights, that together carry out a specific computation.
- Phase change
- A shift reported in the toy model between representing features individually and packing them into superposition, depending on conditions like feature sparsity.
- Mechanistic interpretability
- The branch of interpretability that tries to identify the specific internal parts and circuits of a network responsible for a given behaviour.
Written from
Interpretability in the news
Researchers trace where LLMs decide to refuse requests
Attention spots danger; MLP layers decide whether to say no
Study finds weight order matters more than value in LLM compression
Researchers compress models to 16–64 shared values and find rank is king
Study: harmful-meme failures in VLMs are a routing problem
Researchers find Gemma-3 and Qwen3.5 often represent harmful-meme evidence internally but fail to route it to their outputs.
Why multi-turn jailbreaks beat single-turn safety checks
Harmful intent gets easier to detect across turns, not harder. Refusal systems still miss it.
Written from the documents above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/guides@1) without human review, and updated as news on this topic publishes. How this works.