Study finds LLaMA-2 safety hinges on 0.19% of its weights
Researchers say a tiny, localised edit to one MLP layer strips safety guardrails while leaving accuracy almost untouched.
The facts
- Researchers studied LLaMA-2-7B-Chat to see if safety behaviour is concentrated in a sparse subset of parameters.
- Two localisation methods both pointed to the MLP down_proj layer as the most safety-sensitive component, with o_proj contributing less.
- Modifying just 0.19% of weights in down_proj produced 53% attack success rate (ASR) on a basic jailbreak test and 56% ASR under the GCG attack method.
- tinyBenchmarks accuracy barely moved, from a 52.2% unmodified baseline to 51.6% after the edit.
- The paper was accepted at a NeurIPS 2026 workshop and is authored by Muhammad Zeeshan Karamat and Christiana Chamon Garcia.
Researchers Muhammad Zeeshan Karamat and Christiana Chamon Garcia have found that the safety behaviour of LLaMA-2-7B-Chat lives in a startlingly small corner of the model. They used two complementary techniques: low-rank subspace analysis and parameter-level safety-utility importance filtering. Safety sensitivity, they say, is distributed non-uniformly across the network. One component, the MLP down_proj layer, consistently stood out as the most sensitive, with the o_proj layer offering a smaller secondary signal.
The practical result is blunt. Changing only 0.19% of the weights in down_proj raised the model's attack success rate to 53% on a basic jailbreak test. Under the GCG attack, a gradient-based method for finding adversarial prompts, it rose to 56%. General capability barely moved: tinyBenchmarks accuracy fell from a 52.2% unmodified baseline to 51.6%. An attacker who can touch a sliver of the weight file can strip out refusal behaviour. The model still looks, by standard benchmarks, almost identical to its safe original.
The authors frame this as a fault-tolerance problem specific to on-device and agentic deployment, where model weights sit on local storage rather than behind an API. Safety, on this account, is concentrated in a fraction of a percent of parameters. That fraction becomes a small, well-defined attack surface. The paper argues it is a natural target for integrity protection, rather than blanket defences applied to the whole model.
The paper, accepted at a NeurIPS 2026 workshop, stops at demonstrating the vulnerability on one 7-billion-parameter chat model. It does not report testing on other model families or sizes. Nor does it propose a deployed defence beyond recommending targeted integrity protection for the identified components.
Why it matters
Anyone shipping open-weight models on-device or in agentic pipelines now has a concrete, narrow target to defend. Checksum or sign the down_proj weights specifically, rather than the whole file. But the same finding is a blueprint for an attacker with local write access to silently disable refusals.
Primary source
arXiv cs.CL — arxiv.orgWritten from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.