Researchers trace where LLMs decide to refuse requests
Attention spots danger; MLP layers decide whether to say no
The facts
- Study examined 18 Base-Instruct model pairs across Qwen, Llama and Gemma families
- Attention layer updates carry harmfulness information readily, especially in early layers; MLP updates drive the actual refusal response
- Post-training strengthens MLP refusal writing in all 18 pairs but does not systematically improve how well harmfulness is decoded
- In six reasoning-enabled models, harmfulness stays decodable throughout reasoning while refusal writing concentrates at answer onset, even with reasoning disabled
- Authors introduce Sentinel, combining an attention-derived perception score and MLP-derived decision score; conditional routing on both signals beat single-signal routing and fixed reminders across eight models
Researchers led by Zixuan Weng have mapped the internal pathway that separates recognising a harmful request from refusing it. The paper, posted to arXiv on 9 October, analyses residual-stream updates inside decoder-only transformers. It covers 18 Base-Instruct model pairs spanning the Qwen, Llama and Gemma families.
Attention layers, particularly early ones, carry information about whether a prompt is harmful more readily than MLP layers do. But it is the MLP updates that drive the model's actual refusal output in aligned models. Safety post-training strengthens this MLP refusal-writing behaviour across all 18 model pairs tested, the authors report. That gain comes without a corresponding improvement in how well the model tells harmful requests apart at the attention stage.
Writing of positive refusal signal in the MLP also becomes more concentrated in deeper layers after post-training, the paper says. Causal tests back the pathway: resetting downstream MLP updates to clean values substantially reduces the refusal shift that upstream attention interventions would otherwise cause. In six reasoning-enabled models, harmfulness stays decodable throughout a model's reasoning trace, the authors find. Strong refusal signal only shows up once the model begins its answer, a pattern that holds even when reasoning is switched off.
Building on these findings, the authors introduce Sentinel, a detection method. It combines an attention-derived 'perception' score with an MLP-derived 'decision' score to flag jailbreaks and over-refusals. Routing decisions conditioned on both signals together produce a lower aggregate error rate than routing on either signal alone, they report. The combined approach also beats fixed reminder prompts, across eight models tested.
Why it matters
The finding implies alignment training tunes the "decision" layer to refuse more often without sharpening the model's ability to tell harmful prompts apart. That gap explains why jailbreaks evading early detection still slip through refusal logic. Teams building safety filters may get more mileage from monitoring both signal types jointly, as Sentinel does, than from reinforcing refusal behaviour alone.
Primary source
arXiv cs.AI — arxiv.orgWritten from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.