Study: LLM safety refusals flip 18-28% across seeds, temperature
One prompt, many answers — single-shot safety tests may be measuring noise
The facts
- Erik Larsen tested four instruction-tuned models (Llama 3.1 8B, Qwen 2.5 7B, Qwen 3 8B, Gemma 3 12B) on 876 harmful prompts across 20 sampling configurations (4 temperatures x 5 seeds)
- 18-28% of prompts showed a decision flip between REFUSE/PARTIAL/COMPLY labels depending on the model, per the paper
- Mean within-temperature Safety Stability Index fell from 0.977 at temperature 0.0 to 0.942 at temperature 1.0 (Friedman chi-squared = 396.81, p < 0.001)
- For Llama 3.1 8B, single-shot evaluation agreed with multi-sample ground truth only 92.5% of the time pooled across temperatures, ranging from 98.7% at greedy decoding to 90.3% at temperature 1.0
- Paper recommends scaling sample counts with temperature: one sample at greedy, three at low temperature, more at higher temperatures, and ten when pooling across temperatures
A paper posted to arXiv by Erik Larsen finds that large language models do not refuse harmful prompts consistently. They can give different answers even when asked the identical question twice. The author tested four instruction-tuned models from three families against 876 harmful prompts. These were run across 20 combinations of temperature and random seed. Between 18% and 28% of prompts flip their classification between refuse, partial comply and fully comply. That flip depends on the model and the sampling configuration alone, not the prompt.
The instability gets worse as temperature rises, the parameter that controls how randomly a model samples its next token. Larsen's Safety Stability Index measures how consistent a model's refusal decision is across repeated runs. It drops from 0.977 at temperature 0.0 to 0.942 at temperature 1.0. The paper reports this difference as statistically significant, with a Friedman chi-squared of 396.81 and p < 0.001. Models also waver most on borderline requests. Prompts with higher overall compliance rates show lower stability, a correlation the author reports at Spearman rho between -0.47 and -0.70 across models.
The practical stakes are laid out through Llama 3.1 8B. Single-shot evaluation is the dominant method used today. It agreed with the ground truth established by multi-sample testing only 92.5% of the time when pooling across temperatures. That agreement slipped from 98.7% at greedy decoding to 90.3% at temperature 1.0. The results were cross-checked using Claude 3.5 Haiku as an external judge. It reported 89.1% agreement with a Llama 70B judge on the two models both evaluated, a Cohen's kappa of 0.62. Larsen recommends replacing single-shot tests with sample counts that scale with temperature. He suggests one sample at greedy, three at low temperature, and more at higher settings, since three samples reach only about 95% agreement. Results pooled across temperatures need ten samples, he says.
Why it matters
Anyone benchmarking model safety on a single generation per prompt is measuring sampling noise as much as alignment. Red-teaming pipelines and safety leaderboards that score with one completion per prompt should move to multi-sample protocols. A model certified safe may just have drawn a lucky seed.
Primary source
arXiv cs.CL — arxiv.orgWritten from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.