Process reward model

Also known as: PRM

On this page 7

A process reward model (PRM) is a learned evaluator that assigns a score to every intermediate step of a multi-step solution, rather than a single score to the final answer. In artificial intelligence, and especially in the training of large language models for mathematics, code and logical reasoning, a PRM tells the system where a chain of thought went wrong, not merely that it went wrong.

PRMs are a specialized form of reward model. They are used in two main ways: as a dense reward signal during reinforcement learning and as a verifier at inference time that ranks or prunes candidate reasoning paths. Their counterpart is the outcome reward model (ORM), which judges only the result.

How a process reward model scores reasoning steps

A process reward model works by reading a problem together with a partial solution and predicting how likely the latest step is correct. The solution is first split into steps, usually by line breaks or explicit step markers, and the model emits one score per step, typically a probability between 0 and 1.

Architecturally, most PRMs are language models with a scoring head. The base model is often the same family as the policy being trained, fine-tuned on labeled reasoning traces. At each step boundary, the PRM either predicts a special token such as positive, negative or neutral, or a linear head outputs a scalar. The per-step scores are then aggregated into a solution score, most commonly by taking the minimum or the product, because a single wrong step usually invalidates everything after it.

The table below lists the main design choices that shape how a PRM behaves.

Design choiceCommon optionsWhat it affects
Step segmentationLine breaks, numbered steps, sentence boundariesGranularity and labeling cost
Label typeBinary correct or incorrect, three-way with neutral, soft valueCalibration and nuance of feedback
Scoring headToken classification or scalar regression headEase of training and interpretation
AggregationMinimum, product, last step, meanHow one weak step influences the total
Base model sizeSame as policy or smallerInference cost and judgment quality
Core design choices in a process reward model

Where step-level labels come from

The quality of a process reward model depends almost entirely on its step-level training labels, which are far harder to obtain than outcome labels. Checking a final numeric answer is cheap; deciding whether step seven of a proof is valid requires expertise or an automated substitute.

Three families of labeling methods exist, each with different cost and noise profiles.

Labeling methodHow it worksMain strengthMain weakness
Human annotationExperts mark each step as correct, incorrect or neutralHighest precision on subtle errorsSlow and expensive at scale
Monte Carlo rolloutsMany completions are sampled from each step; the share reaching a correct answer becomes the labelFully automatic and scalableNoisy; rewards lucky recoveries
LLM-as-a-judge labelingA strong model critiques each stepCheap and flexibleInherits the judge’s blind spots
Formal verificationSteps are checked by a proof assistant or executed codeExact correctness where applicableLimited to formalizable domains
Sources of step-level supervision

Monte Carlo estimation is popular because it scales, but it measures whether a step can still lead to success, which is a value estimate rather than a correctness judgment. A flawed step followed by a lucky correction can receive a high score, so many teams combine automatic labels with a smaller human-verified set for calibration and evaluation.

Process reward models compared with outcome reward models

The central difference is granularity of feedback: an outcome reward model rates only the final answer, while a process reward model rates the path. This matters because correct answers can be reached by invalid reasoning, and wrong answers can contain mostly valid reasoning.

PropertyProcess reward modelOutcome reward model
Feedback unitEach reasoning stepWhole solution
Credit assignmentLocalizes the first errorSparse; blames the entire trace
Labeling costHigh per exampleLow; often automatic
Detects right answer, wrong reasoningYes, in principleNo
Use in tree searchNatural fit for pruning partial pathsOnly after completion
Reward hacking riskStep padding and style gamingAnswer guessing and format gaming
PRM and ORM side by side

Published comparisons on mathematical benchmarks have generally found that process supervision produces more reliable verifiers than outcome supervision for best-of-N selection, although the margin depends on labeling quality and task difficulty. Related training methods such as RLHF, DPO and RLAIF typically rely on outcome-level or response-level preferences, which is why PRMs are discussed separately in the reasoning literature.

Using a process reward model in training and inference

A process reward model serves two distinct roles that are often combined in one system. During post-training, it supplies dense rewards to a reinforcement learning algorithm so the policy learns which intermediate moves help. At inference time, it acts as a verifier that guides search over many sampled reasoning paths.

Use caseMechanismTypical benefit
Best-of-N rerankingSample N full solutions, keep the highest aggregated scoreAccuracy gains with no policy changes
Beam or tree searchExpand partial paths and prune low-scoring steps earlyBetter compute efficiency than sampling alone
Dense RL rewardAdd step scores to the policy gradient objectiveFaster, more stable credit assignment
Rejection sampling fine-tuningKeep only traces whose every step passes a thresholdCleaner supervised fine-tuning data
Error localizationReport the first low-scoring step to a user or toolDebuggable and auditable reasoning
Common ways to deploy a PRM

In practice, inference-time search is the lower-risk entry point, because the policy weights stay unchanged and the PRM can be evaluated offline. Using a PRM as an RL reward is more powerful but exposes the system to reward hacking, so practitioners usually cap the weight of step rewards and keep an outcome check in the loop.

Limitations and failure modes

Process reward models are imperfect proxies for correctness, and their errors compound when a policy is optimized against them. The most frequently reported problems are listed below.

  • Reward hacking: the policy learns to write many short, safe-looking steps or to repeat obvious statements that the PRM rates highly.
  • Distribution shift: a PRM trained on one policy’s outputs can misjudge the style of a stronger or different model.
  • Ambiguous step boundaries: reasoning that does not split cleanly into steps produces inconsistent scores.
  • Label noise from rollouts: Monte Carlo labels confuse recoverable mistakes with correct steps.
  • Domain narrowness: PRMs trained on mathematics transfer poorly to open-ended writing, planning or ethical reasoning.
  • Inference cost: scoring every step of many candidate paths can multiply compute by an order of magnitude.

Because of these limits, a PRM score should be treated as evidence rather than ground truth. Mature pipelines monitor the gap between PRM scores and verified outcomes and retrain the PRM periodically on fresh outputs of the current policy.

The idea in brief

A process reward model is like a teacher who marks every line of a math solution, not just the final answer at the bottom of the page. If a student makes a mistake in line three, the teacher circles line three, so the student knows exactly what to fix.

AI systems that solve problems step by step benefit from the same kind of feedback. A process reward model gives each step a grade, which helps the AI learn better habits and helps it pick the most trustworthy solution when it has written several attempts. The catch is that the grader can also be fooled, so its grades need regular checking.

A practical scenario

A research team builds a tutoring assistant for high school algebra. The base model answers about 62 percent of competition-style problems correctly on a held-out set, and many correct answers rest on flawed algebra that would confuse students. The team trains a process reward model on about 40,000 solutions, using automatic rollout labels for most traces and a teacher-verified subset of 3,000 solutions for calibration.

At inference, the assistant now samples 16 solutions per problem and uses beam search guided by the PRM, discarding paths whose minimum step score falls below 0.4. Accuracy rises into the mid-70 percent range, and spot checks show that far fewer accepted answers contain invalid intermediate steps. When the PRM flags a low-scoring step in a student’s own work, the assistant highlights that line, which teachers find more useful than a simple right-or-wrong verdict.