Reward model

On this page 7

A reward model is a neural network trained to assign a scalar score to a model output so that higher scores correspond to responses humans would prefer. In artificial intelligence, and specifically in the training of large language models, it serves as a learned proxy for human judgment: instead of asking people to rate every generated answer, developers collect a limited set of comparisons, fit a reward model to them, and then use that model to score millions of outputs during optimization.

The reward model is the central component of reinforcement learning from human feedback (RLHF) and of its AI-feedback variant RLAIF. Its quality largely determines what the final assistant learns to value, which is why practitioners treat it as a critical and fragile part of post-training rather than a simple utility.

How a reward model learns human preferences

A reward model learns from pairwise preference data: for a single prompt, annotators see two or more candidate responses and indicate which one is better. The model is trained so that the chosen response receives a higher score than the rejected one. This framing is easier for humans than absolute ratings, because people agree more reliably on relative quality than on a number from one to ten.

The standard training objective comes from the Bradley-Terry model. The probability that response A is preferred over response B is modeled as the sigmoid of the score difference, and the loss is the negative log of that probability. Only differences matter, so the absolute level of the score is arbitrary; many teams normalize rewards to zero mean before using them in reinforcement learning.

Architecturally, a reward model is usually a pretrained language model with its output head replaced by a single linear layer that produces one number. It is commonly initialized from the supervised fine-tuning checkpoint of the policy, so that it already understands the same language and task distribution. The table below summarizes the main components.

ComponentTypical choicePurpose
BackboneTransformer from the SFT checkpointUnderstands prompt and response text
Scalar headLinear layer on the final tokenProduces one reward value
Training dataTens of thousands to millions of comparisonsEncodes human or AI preferences
LossBradley-Terry pairwise log-likelihoodRanks chosen above rejected
Training lengthOften one epochLimits memorization of noisy labels
Core components of a typical reward model

Types of reward models

Reward models differ in what they score and how often. The classic design scores a complete response once, but newer approaches score individual reasoning steps or replace learned scoring with programmatic checks. The choice depends on whether the task has verifiable answers and how much annotation budget is available.

VariantWhat it scoresTypical use
Outcome reward modelThe final complete responseGeneral chat helpfulness and harmlessness
Process reward modelEach intermediate reasoning stepMathematics and multi-step reasoning
Generative reward modelA written critique followed by a verdictComplex judgments that benefit from explanation
Rule-based or verifiable rewardPass or fail against a checkerCode tests, math answers, format compliance
Multi-objective reward modelSeparate scores per attributeBalancing helpfulness, safety and style
Main reward model variants and where they fit

A process reward model is treated as its own Term because it requires step-level labels and changes the credit assignment problem. Generative reward models overlap with the LLM-as-a-judge pattern from evaluation, but here the judgment is used as a training signal rather than only as a measurement.

In RLHF, the reward model scores samples generated by the policy, and an algorithm such as Proximal Policy Optimization (PPO) updates the policy to increase expected reward. A KL-divergence penalty against the reference model keeps the policy from drifting too far, which limits the damage when the reward model is wrong.

Not every preference method needs an explicit reward model. Direct Preference Optimization (DPO) derives an implicit reward from the policy itself and trains directly on preference pairs. Reward models remain valuable for best-of-n sampling, rejection sampling fine-tuning, data filtering and online reinforcement learning, where fresh outputs must be scored continuously. The comparison below shows how the explicit reward model fits among these approaches.

MethodExplicit reward modelMain trade-off
RLHF with PPOYes, queried onlineStrong results but complex and costly
DPONo, reward is implicitSimple and stable, but offline only
RLAIFYes, trained on AI labelsCheaper labels, inherits labeler bias
Best-of-n samplingYes, used at inferenceNo training, but n times the compute
Rejection sampling fine-tuningYes, used to filter dataStable, but limited exploration
How methods use preference signals

Reward hacking and other limitations

The most serious limitation is reward hacking, also called reward overoptimization. A reward model is an imperfect proxy, so a policy optimized hard against it eventually finds outputs that score highly without being better. This is an instance of Goodhart’s law: once the proxy becomes the target, it stops measuring what it was meant to measure.

Typical exploited patterns include excessive length, confident tone, flattering agreement with the user and repeated formatting that annotators happened to favor. Empirical studies show that true quality first rises and then falls as optimization pressure increases, while the proxy score keeps climbing. The table lists frequent failure modes and common mitigations.

SymptomLikely causeMitigation
Responses grow steadily longerLength bias in preference labelsLength-controlled rewards or penalties
Policy agrees with false user claimsAnnotators rewarded agreeable toneTargeted counterexamples in preference data
Reward rises while human ratings fallOveroptimization of the proxyStronger KL penalty, early stopping
Unstable scores on new topicsDistribution shift from training dataRefresh comparisons from current policy outputs
High variance between seedsNoisy or inconsistent labelsReward model ensembles and label auditing
Common reward model failure modes

Label quality is a second constraint. Inter-annotator agreement on preference tasks is often only moderately above chance for subtle cases, which puts a ceiling on reward model accuracy. Held-out pairwise accuracy commonly lands in the roughly 65 to 80 percent range, depending on task difficulty and data quality, so a reward model should never be treated as ground truth.

Practical guidance for building a reward model

A reliable reward model starts with clear annotation guidelines and comparisons drawn from the same distribution the policy will produce. Collecting new comparisons from the current policy in iterative rounds keeps the reward model relevant as the policy improves.

  • Use a backbone at least as capable as the policy; a weaker reward model is easily exploited.
  • Train for about one epoch and monitor held-out pairwise accuracy to avoid overfitting.
  • Check for length and style correlations in the scores before using the model for optimization.
  • Evaluate with human spot checks during RL, not only with the reward curve.
  • Consider ensembles or uncertainty estimates when optimization pressure will be high.

In simple terms

A reward model is like a trained judge that has studied thousands of examples where people chose the better of two answers. After that study, it can look at a new answer and give it a score that guesses how much people would like it.

The main AI then practices against this judge and learns to produce answers that earn high scores. The catch is that the judge has blind spots. If the AI practices too hard, it can learn to please the judge instead of the people, for example by writing long answers that only look thorough.

A practical example

A team building a customer support assistant collects about 40,000 preference comparisons from trained annotators, each showing two draft replies to a real but anonymized support question. They train a reward model from their 7-billion-parameter supervised checkpoint for one epoch and reach 72 percent pairwise accuracy on a held-out set.

During PPO training, the reward climbs steadily, but weekly human reviews show that replies have grown 60 percent longer and now repeat the same apology phrases. The team adds a length penalty, collects 8,000 new comparisons from current policy outputs, and retrains the reward model. In the next round, the human-rated resolution quality improves while average reply length returns close to the original level, which confirms that the refreshed reward model tracked real quality more closely.