Reasoning models RLHF and preference tuning·2 Oct 2026

Study: post-training expands, not just compresses, hard reasoning

A diagnostic probe finds post-training raises the solvable ceiling on hard math problems, not just the odds of sampling a known answer.

The facts

Researchers compared three post-training paths: their own off-policy distillation, released Qwen3 off-policy-plus-on-policy distillation endpoints, and a released DeepSeek-Math endpoint trained with GRPO (Group Relative Policy Optimisation)
On easier AMC problems, large-K pass@K ceilings are near saturation, so post-training mainly compresses sample cost rather than unlocking new solutions
On harder AIME problems, post-training expands the large-K ceiling over the base model, with Qwen3 released endpoints raising it furthest and DeepSeek-Math GRPO not dominating sufficient off-policy distillation at large K
English-dominant distillation improves non-English reasoning but preserves gaps between language tiers; a controlled-overfit audit finds limited sensitivity in current SFT-membership probes
The paper is 18 pages with 17 figures and 10 tables, submitted 29 September 2026 and currently under review

A team led by Hongyang Li, with co-authors including Pascal Bouvry, has published a diagnostic study on post-training gains in math reasoning. The common explanation is that models are simply getting better at sampling solutions they could already find. The authors argue this only holds some of the time.

The paper, posted to arXiv on 29 September 2026, compares three post-training approaches under a shared evaluation method. These are the authors' own off-policy distillation runs, released Qwen3 endpoints that combine off-policy and on-policy distillation, and a released DeepSeek-Math endpoint trained with GRPO. GRPO is a reinforcement-learning method that scores a batch of sampled responses against each other instead of a learned value function.

The method goes beyond pass@1, the standard measure of whether a model's first attempt is correct. The authors instead measure pass@K across large sample budgets, across verbatim prompts, paraphrases, numerical variants and translations. They pair that with checks for answer consistency, distribution shape, and whether supervised fine-tuning data has leaked into the test set.

On easier AMC-level problems, they report that large-K ceilings are already near saturation across all three methods. Post-training's main effect there, in other words, is to make correct answers cheaper to sample, not to make more of them reachable.

On harder AIME-level problems the picture reverses. Post-training raises the large-K ceiling above the base model's, with sufficient off-policy distillation already doing so. Qwen3's released endpoints raise it further still. The DeepSeek-Math GRPO endpoint fails to beat off-policy distillation at large sample counts.

The paper also reports that distillation dominated by English data improves reasoning in other languages. But it does not close the gap between language tiers — models stay measurably worse in lower-resource languages even after the English-centric training helps.

A separate audit, designed to detect overfitting to supervised fine-tuning data, found that current membership-detection probes have limited sensitivity. Claims that a model's gains are mostly memorisation are therefore harder to verify or rule out than the existing tools suggest.

Why it matters

For teams evaluating post-trained math models, pass@1 leaderboard gains mean different things depending on problem difficulty. On easy benchmarks a score bump may just reflect cheaper sampling; on hard benchmarks it can reflect genuinely new capability. Anyone choosing between GRPO-style RL and distillation for a reasoning model should test at large K on the hardest problems in their domain, not trust a single pass@1 number. Current SFT-contamination checks are weak evidence, not proof of clean data.

Primary source

arXiv cs.LG — arxiv.org

Written from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.