What are reasoning models?
But knowing the steps doesn't guarantee they're right
What is a reasoning model?
A reasoning model is a large language model trained or prompted to produce a chain of thought. That is a series of intermediate steps, generated before a final answer. Instead of jumping straight to a response, it works through a problem the way a person might scribble workings on a pad. The idea first appeared as a prompting trick: Wei et al. (2022) found that giving a model a few worked examples with explicit reasoning steps, rather than just input-output pairs, sharply improved performance on arithmetic, commonsense and symbolic tasks.
Later work moved the behaviour from the prompt into the model itself. DeepSeek-AI's 2025 paper on DeepSeek-R1 trained a model to generate long chains of thought using reinforcement learning. It used no human-written reasoning examples at all. The model learned on its own to check its work, reconsider wrong turns, and allocate more effort to harder problems.
How does chain-of-thought prompting work?
Chain-of-thought prompting, as described by Wei et al. (2022), works by showing a language model a handful of examples in the form of question, reasoning steps, answer. The model then imitates that pattern on a new question, generating its own reasoning steps before answering. The paper found this only helped once models reached roughly 100 billion parameters. Smaller models produced fluent but illogical chains of thought that hurt accuracy rather than helping it.
Using PaLM at 540 billion parameters, the authors reported that chain-of-thought prompting reached new state-of-the-art accuracy on the GSM8K math word problem benchmark. It more than doubled performance over standard prompting on that dataset. Gains were largest on harder, multi-step problems and negligible on single-step ones. The technique also let models generalise to longer problem instances than any example they had seen, on toy symbolic tasks like concatenating last letters of names.
What are process rewards, and how do they differ from outcome rewards?
A reward model scores a model's output so it can be ranked or used to train further. An outcome-supervised reward model (ORM) is trained only on whether a final answer was correct. A process-supervised reward model (PRM) is trained on whether each individual step of the reasoning was correct. Lightman et al. (2023), in Let's Verify Step by Step, collected 800,000 human step-level correctness labels — the PRM800K dataset — to train a PRM on OpenAI's GPT-4 base model.
On a representative subset of the MATH benchmark, their process-supervised reward model solved 78.2% of problems when used to rank many sampled solutions. That compares with 72.4% for an outcome-supervised reward model and 69.6% for simple majority voting. The gap widened as more candidate solutions were sampled per problem. The paper also found process supervision generalised to out-of-distribution STEM questions pulled from recent AP and AMC exams, reaching 72.9% aggregate accuracy against 63.8% for the outcome-supervised model.
How does reinforcement learning train models to reason?
Reinforcement learning (RL) trains a model by rewarding entire generated outputs rather than labelling each step by hand. DeepSeek-AI's R1 paper applied this directly to a base model, DeepSeek-V3-Base. It used an algorithm called Group Relative Policy Optimization (GRPO), which samples a group of candidate answers to each question and adjusts the model toward the ones that scored best relative to the group average. The reward was rule-based: correctness of the final answer, checked automatically for math and code, plus a format reward for wrapping reasoning in designated tags.
The resulting model, DeepSeek-R1-Zero, trained with no supervised fine-tuning and no human-written reasoning examples, saw its pass rate on the AIME 2024 maths competition rise from 15.6% to 77.9% over training. It reached 86.7% with self-consistency decoding. The paper reports the model spontaneously grew longer responses over training and began exhibiting self-reflection. It generated phrases like "wait" mid-solution before reconsidering its approach — what the authors call an "aha moment". Because this pure RL model had rough edges, poor readability and mixed languages in its output, DeepSeek-AI built DeepSeek-R1 on top of it with a multi-stage pipeline: cold-start human-aligned examples, a reasoning-focused RL stage, rejection sampling and supervised fine-tuning, then a second RL stage for helpfulness and harmlessness.
What is test-time compute, and why does it matter?
Test-time compute is the computation a model spends while generating its answer, as opposed to the computation spent training it. A reasoning model that produces a long chain of thought is spending more test-time compute on harder problems. DeepSeek-AI's R1 paper reports that DeepSeek-R1-Zero learned, unprompted, to use fewer tokens on simple questions and more on complex ones. That is a dynamic allocation of reasoning effort rather than a fixed budget.
The same paper notes this comes with a cost. DeepSeek-R1 sometimes over-reasons on simple questions, producing more tokens than needed, which the authors call a token-efficiency limitation. Lightman et al. (2023) showed a related form of test-time compute spending: searching over many sampled solutions with a process reward model. Accuracy kept climbing as the number of sampled solutions (N) increased, with the gap between process and outcome supervision widening at higher N.
Where do reasoning models fall short?
None of this makes reasoning infallible. Wei et al. (2022) acknowledge there is no guarantee a chain of thought is logically correct even when the final answer is. They also note the behaviour only emerges reliably at large model scale, making it costly to serve. DeepSeek-AI's R1 paper lists its own limits plainly: weak structured output and no tool use, sensitivity to prompt format — few-shot prompting actually hurts R1's performance — language mixing between English and Chinese, and little improvement over its own base model on software engineering benchmarks.
Reward hacking is a recurring risk across both reinforcement-learning and reward-model approaches. DeepSeek-AI reports that letting a model-based reward run too long led to exploitation of the reward signal rather than genuine improvement. That is why its rule-based rewards avoid neural reward models for reasoning tasks altogether. Lightman et al. (2023) flag the same danger for reward-model training: a reward model trained on outcomes can reward a step-by-step solution that reaches the right answer through wrong reasoning.
Key terms
- Chain of thought
- A series of intermediate natural language reasoning steps a model generates before its final answer.
- Chain-of-thought prompting
- A few-shot prompting method that gives a model worked examples including reasoning steps, shown by Wei et al. (2022) to improve performance on reasoning tasks at sufficient model scale.
- Outcome-supervised reward model (ORM)
- A reward model trained using only whether a solution's final result was correct.
- Process-supervised reward model (PRM)
- A reward model trained using feedback on the correctness of each individual step in a solution, as introduced for large-scale use by Lightman et al. (2023).
- Group Relative Policy Optimization (GRPO)
- The reinforcement learning algorithm DeepSeek-AI used to train DeepSeek-R1, which scores a sampled group of model outputs against their group average rather than using a separate value model.
- Reward hacking
- A failure mode where a trained policy finds shortcuts that exploit a reward model or reward rule rather than genuinely improving, flagged as a risk by both the DeepSeek-R1 and Let's Verify Step by Step papers.
- Test-time compute
- Computation spent while a model generates its answer, which reasoning models can scale up by producing longer chains of thought on harder problems.
Latest developments
Written from
Reasoning models in the news
Liquid AI ships LFM2.5-24B-A2B, its biggest on-device agent model
A 24B mixture-of-experts model that uses just 2B active parameters per token
OpenAI publishes new math results from an internal frontier model
Lean proofs and ChatGPT Pro compute estimates follow advice from an outside math-AI group.
Aleph Alpha releases Kolibri-1, a German-English MoE reasoning model
78B parameters, 3.46B active. Apache 2.0. A sovereignty bet dressed as efficiency.
Study: post-training expands, not just compresses, hard reasoning
A diagnostic probe finds post-training raises the solvable ceiling on hard math problems, not just the odds of sampling a known answer.
Study: DeepSeek-R1-Distill benchmark gains hard to reproduce
A paper finds reasoning-model benchmark scores swing widely with small changes to evaluation setup, undercutting claimed gains.
Written from the documents above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/guides@1) without human review, and updated as news on this topic publishes. How this works.