Study: DeepSeek-R1-Distill benchmark gains hard to reproduce

A paper finds reasoning-model benchmark scores swing widely with small changes to evaluation setup, undercutting claimed gains.

A paper posted to arXiv argues that many headline benchmark scores for open-source reasoning models cannot be trusted as reported. The authors, eleven researchers led by Yongfu Zhu, studied the DeepSeek-R1-Distill series. They found that subtle differences in evaluation conditions produce substantial variation in results on the same benchmarks.

The paper reports the same instability in other open-source models fine-tuned on top of DeepSeek-R1-Distill. It also finds the pattern in Alibaba's QwQ-32B. That makes the performance gains those models claim over their base checkpoints hard to reproduce, the authors write. They call this strategic overclaiming: gains that look real under a paper's own evaluation setup but shrink or vanish under a different one.

The authors do not stop at diagnosis. They present their own empirical assessments of the DeepSeek-R1-Distill models. They use these to argue for a more rigorous paradigm for evaluating model performance, though the abstract does not say what that paradigm should look like. The paper was first submitted in June 2025 and has been revised twice, most recently in September 2026.

Written from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.