Ai2's FairStress finds LLMs flip fairer for minority candidates

A new benchmark shows models abandon correct answers more readily when it helps a minority-coded candidate

Allen Institute for AI has released FairStress, a benchmark built to catch a failure mode that standard fairness tests miss. Models often pick the same candidate regardless of demographic swap. Yet they abandon that correct pick far more readily when a persuasive argument favours one group over another. The benchmark tests open-weight and frontier models, including Olmo, Llama, Qwen, Gemma, Claude, and the GPT family. It covers hiring, lending, healthcare, education, and housing scenarios.

The core finding is what Ai2 calls anti-stereotypical over-correction. When a clearly stronger candidate carried a majority-coded identity, such as white, male, or heterosexual, models were more likely to switch away from that correct choice under a persuasive prefix favouring the weaker, minority-coded candidate than the reverse. Ai2 reports this gap at 6.9 percentage points when identity was only implied, and 9.3 points when it was stated outright, averaged across 15 models. On the trimmed FairStress-Core set, frontier models Claude Opus 5 and GPT-5.5 switched toward minority-coded candidates 8.8 to 42.4 percentage points more often than toward majority-coded ones.

The asymmetry extends beneath the final answer. Explanations for minority-coded selections leaned on words like resilience and diversity. Explanations for majority-coded picks used more procedural language, such as evidence and criteria. That means a model can favour a candidate in its choice while crediting their qualifications less in its reasoning. In a narrower probe of Gemma-3-12B-it and Qwen2.5-7B-Instruct, internal activations predicted the selected demographic side with about 73% accuracy. That is far above the 57-59% reached by reading the model's own text explanations. The stated reasoning, in other words, does not fully reflect what the model has represented internally.

Ai2 also ran its own method against BBQ, a widely used bias benchmark, and found the two can point in opposite directions. BBQ's standard setup scores fixed answer options by probability, rather than having the model generate a response. Ai2's controlled tests on four models found that switching to a chat template with generated yes/no answers moved three of them from stereotype-consistent behaviour toward minority-favouring behaviour instead.

Written from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.