Paper finds math benchmark rankings flip under paraphrasing

A $10 audit shows model leaderboards can hinge on which rewordings get asked

Researchers Nishal Thomas and Noel Thomas audited MathCheck, a math reasoning benchmark accepted at ICLR 2025. They say the routine check turned up enough bad test items to change which model looks best.

Checking 129 question groups, they found 4 semantically incorrect paraphrases, a 3.1% error rate. Stripping those items out drops GPT-4o from second place to fourth, the paper reports. Claude Haiku and DeepSeek V3 move above it as a result.

The authors say the errors were catchable cheaply. A cross-model unanimity check, where models agree a paraphrase changed a question's meaning, found them automatically for under $10. Applying the same check to their own dataset, they found 47% of auto-generated connective-variation paraphrases were in fact semantically wrong, though meant to preserve meaning. That is the shallow half of the problem. The deeper one, the paper argues, is that benchmarks built from imperfect paraphrases measure something they were never designed to isolate.

Even on theorems every model already answers correctly in canonical form, a rewording that should not change the answer still flips it. The flip rate ranges from 5% to 42% depending on the model, a metric the authors call the Conditional Inconsistency Rate. Claude Haiku 4.5 flips a known-correct answer roughly nine times as often as DeepSeek V3, 41.9% versus 4.7%, on the 43 theorems both models were tested against. That ranking holds up on a separate, independent set of 103 theorems.

Raw Semantic Consistency Rates range from 50% to 82% across the 9 models tested. But the authors say that spread mostly reflects accuracy compounding over roughly six paraphrases per theorem, not genuine robustness. No model dominates across every family of paraphrase, they argue. So the specific paraphrase types a benchmark happens to include quietly fix which model comes out on top, what they term a No-Free-Benchmark corollary. The paper evaluates 9 models on 366 to 811 items, with canonical statements verified in the Lean4 proof assistant, and introduces a tool called FormInvSelector for picking models by the regime a deployment actually needs.

Written from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.