A process reward model (PRM) is a learned evaluator that assigns a score to every intermediate step of a multi-step solution, rather than a single score to the final answer. In artificial intelligence, and especially in the training of large language models for mathematics, code and logical reasoning, a PRM tells the system where a chain of thought went wrong, not merely that it went wrong.
PRMs are a specialized form of reward model. They are used in two main ways: as a dense reward signal during reinforcement learning and as a verifier at inference time that ranks or prunes candidate reasoning paths. Their counterpart is the outcome reward model (ORM), which judges only the result.
How a process reward model scores reasoning steps
A process reward model works by reading a problem together with a partial solution and predicting how likely the latest step is correct. The solution is first split into steps, usually by line breaks or explicit step markers, and the model emits one score per step, typically a probability between 0 and 1.
Architecturally, most PRMs are language models with a scoring head. The base model is often the same family as the policy being trained, fine-tuned on labeled reasoning traces. At each step boundary, the PRM either predicts a special token such as positive, negative or neutral, or a linear head outputs a scalar. The per-step scores are then aggregated into a solution score, most commonly by taking the minimum or the product, because a single wrong step usually invalidates everything after it.
The table below lists the main design choices that shape how a PRM behaves.
| Design choice | Common options | What it affects |
|---|---|---|
| Step segmentation | Line breaks, numbered steps, sentence boundaries | Granularity and labeling cost |
| Label type | Binary correct or incorrect, three-way with neutral, soft value | Calibration and nuance of feedback |
| Scoring head | Token classification or scalar regression head | Ease of training and interpretation |
| Aggregation | Minimum, product, last step, mean | How one weak step influences the total |
| Base model size | Same as policy or smaller | Inference cost and judgment quality |
Where step-level labels come from
The quality of a process reward model depends almost entirely on its step-level training labels, which are far harder to obtain than outcome labels. Checking a final numeric answer is cheap; deciding whether step seven of a proof is valid requires expertise or an automated substitute.
Three families of labeling methods exist, each with different cost and noise profiles.
| Labeling method | How it works | Main strength | Main weakness |
|---|---|---|---|
| Human annotation | Experts mark each step as correct, incorrect or neutral | Highest precision on subtle errors | Slow and expensive at scale |
| Monte Carlo rollouts | Many completions are sampled from each step; the share reaching a correct answer becomes the label | Fully automatic and scalable | Noisy; rewards lucky recoveries |
| LLM-as-a-judge labeling | A strong model critiques each step | Cheap and flexible | Inherits the judge’s blind spots |
| Formal verification | Steps are checked by a proof assistant or executed code | Exact correctness where applicable | Limited to formalizable domains |
Monte Carlo estimation is popular because it scales, but it measures whether a step can still lead to success, which is a value estimate rather than a correctness judgment. A flawed step followed by a lucky correction can receive a high score, so many teams combine automatic labels with a smaller human-verified set for calibration and evaluation.
Process reward models compared with outcome reward models
The central difference is granularity of feedback: an outcome reward model rates only the final answer, while a process reward model rates the path. This matters because correct answers can be reached by invalid reasoning, and wrong answers can contain mostly valid reasoning.
| Property | Process reward model | Outcome reward model |
|---|---|---|
| Feedback unit | Each reasoning step | Whole solution |
| Credit assignment | Localizes the first error | Sparse; blames the entire trace |
| Labeling cost | High per example | Low; often automatic |
| Detects right answer, wrong reasoning | Yes, in principle | No |
| Use in tree search | Natural fit for pruning partial paths | Only after completion |
| Reward hacking risk | Step padding and style gaming | Answer guessing and format gaming |
Published comparisons on mathematical benchmarks have generally found that process supervision produces more reliable verifiers than outcome supervision for best-of-N selection, although the margin depends on labeling quality and task difficulty. Related training methods such as RLHF, DPO and RLAIF typically rely on outcome-level or response-level preferences, which is why PRMs are discussed separately in the reasoning literature.
Using a process reward model in training and inference
A process reward model serves two distinct roles that are often combined in one system. During post-training, it supplies dense rewards to a reinforcement learning algorithm so the policy learns which intermediate moves help. At inference time, it acts as a verifier that guides search over many sampled reasoning paths.
| Use case | Mechanism | Typical benefit |
|---|---|---|
| Best-of-N reranking | Sample N full solutions, keep the highest aggregated score | Accuracy gains with no policy changes |
| Beam or tree search | Expand partial paths and prune low-scoring steps early | Better compute efficiency than sampling alone |
| Dense RL reward | Add step scores to the policy gradient objective | Faster, more stable credit assignment |
| Rejection sampling fine-tuning | Keep only traces whose every step passes a threshold | Cleaner supervised fine-tuning data |
| Error localization | Report the first low-scoring step to a user or tool | Debuggable and auditable reasoning |
In practice, inference-time search is the lower-risk entry point, because the policy weights stay unchanged and the PRM can be evaluated offline. Using a PRM as an RL reward is more powerful but exposes the system to reward hacking, so practitioners usually cap the weight of step rewards and keep an outcome check in the loop.
Limitations and failure modes
Process reward models are imperfect proxies for correctness, and their errors compound when a policy is optimized against them. The most frequently reported problems are listed below.
- Reward hacking: the policy learns to write many short, safe-looking steps or to repeat obvious statements that the PRM rates highly.
- Distribution shift: a PRM trained on one policy’s outputs can misjudge the style of a stronger or different model.
- Ambiguous step boundaries: reasoning that does not split cleanly into steps produces inconsistent scores.
- Label noise from rollouts: Monte Carlo labels confuse recoverable mistakes with correct steps.
- Domain narrowness: PRMs trained on mathematics transfer poorly to open-ended writing, planning or ethical reasoning.
- Inference cost: scoring every step of many candidate paths can multiply compute by an order of magnitude.
Because of these limits, a PRM score should be treated as evidence rather than ground truth. Mature pipelines monitor the gap between PRM scores and verified outcomes and retrain the PRM periodically on fresh outputs of the current policy.
The idea in brief
A process reward model is like a teacher who marks every line of a math solution, not just the final answer at the bottom of the page. If a student makes a mistake in line three, the teacher circles line three, so the student knows exactly what to fix.
AI systems that solve problems step by step benefit from the same kind of feedback. A process reward model gives each step a grade, which helps the AI learn better habits and helps it pick the most trustworthy solution when it has written several attempts. The catch is that the grader can also be fooled, so its grades need regular checking.
A practical scenario
A research team builds a tutoring assistant for high school algebra. The base model answers about 62 percent of competition-style problems correctly on a held-out set, and many correct answers rest on flawed algebra that would confuse students. The team trains a process reward model on about 40,000 solutions, using automatic rollout labels for most traces and a teacher-verified subset of 3,000 solutions for calibration.
At inference, the assistant now samples 16 solutions per problem and uses beam search guided by the PRM, discarding paths whose minimum step score falls below 0.4. Accuracy rises into the mid-70 percent range, and spot checks show that far fewer accepted answers contain invalid intermediate steps. When the PRM flags a low-scoring step in a student’s own work, the assistant highlights that line, which teachers find more useful than a simple right-or-wrong verdict.