Reward function

Also known as: reward signal

On this page 7

A reward function is the mapping that assigns a scalar numerical signal to each step of an agent’s interaction with its environment, usually written as R(s, a, s') for the state, the action taken and the next state. In reinforcement learning, it is the only formal statement of the task: the agent never sees the designer’s intentions, only the numbers the reward function produces, and it learns a policy that maximizes the expected cumulative reward.

The reward function, often called the reward signal, is one of the five elements of a Markov decision process together with states, actions, transition dynamics and the discount factor. Because every downstream algorithm, from Q-learning to proximal policy optimization, optimizes whatever the reward measures, reward design is frequently the hardest part of applying reinforcement learning to a real problem.

How a reward function drives learning

At every time step the environment returns a reward, and the agent’s objective is the discounted return: the sum of future rewards, each multiplied by the discount factor gamma raised to the number of steps ahead. Value functions, Q-functions and advantage estimates are all expectations of this return, so the reward function indirectly defines every quantity the agent estimates.

The reward is a local, immediate signal, while the goal is long-term. The Bellman equation links the two by expressing the value of a state as the immediate reward plus the discounted value of the next state. This recursive structure lets the agent propagate credit backward from rare successes to the earlier decisions that caused them, which is the essence of the credit assignment problem.

The reward function can take several mathematical forms, and the choice affects both expressiveness and learning speed. The table below lists the common signatures.

FormDepends onTypical use
State reward R(s)Current state onlyGoal regions, grid worlds
State-action reward R(s, a)State and chosen actionPenalizing costly actions such as energy use
Transition reward R(s, a, s’)State, action and next stateRewarding progress between states
Stochastic rewardA distribution over outcomesBandits, noisy real-world measurements
Learned reward modelA neural network trained on labelsPreference-based training of language models
Common forms of the reward function

Dense, sparse and shaped rewards

Reward functions differ above all in how often they provide informative feedback. A sparse reward gives a nonzero value only at rare events, such as winning a game, while a dense reward scores almost every step. The density determines how much exploration the agent needs before it receives any useful learning signal.

DesignExampleStrengthMain risk
Sparse rewardPlus one on reaching the goal, zero otherwiseHard to exploit, matches the true goalVery slow learning, weak signal
Dense rewardNegative distance to the target each stepFast, informative feedbackEasier to game, local optima
Shaped rewardSparse goal plus potential-based bonusSpeeds learning safelyWrong shaping changes behavior
Intrinsic rewardCuriosity or novelty bonusDrives exploration in sparse tasksAgent chases noise
Preference-based rewardModel trained on human comparisonsCaptures hard-to-specify goalsReward model errors are exploited
Reward designs and their trade-offs

Reward shaping adds extra terms to guide the agent. Potential-based shaping, defined as gamma times the potential of the next state minus the potential of the current state, is proven to leave the optimal policy unchanged. Arbitrary bonuses carry no such guarantee and can create loops in which the agent collects shaping reward without completing the task.

The reward function is often confused with neighboring quantities, but each plays a distinct role. The reward is given by the environment or the designer, while value functions and policies are learned by the agent from it.

ConceptWhat it representsWho defines itTime horizon
Reward functionImmediate feedback per stepDesigner or environmentOne step
Cumulative rewardDiscounted sum of rewardsDerived from rewardsWhole episode
Value functionExpected return from a stateLearned by the agentFuture steps
PolicyMapping from states to actionsLearned by the agentDecision rule
Cost functionNegative reward in control theoryDesignerOne step or trajectory
How the reward function differs from neighboring terms

Inverse reinforcement learning reverses the usual direction: it infers a reward function from expert demonstrations. The same idea underlies reinforcement learning from human feedback, where a reward model learned from preference pairs replaces a hand-written function and is then optimized with algorithms such as proximal policy optimization or group relative policy optimization.

Designing a reward function in practice

Good reward design starts by stating the true objective in measurable terms and then adding only the guidance the agent genuinely needs. Practitioners typically follow a short sequence of steps.

  1. Define the terminal success and failure conditions and give them clearly dominant values.
  2. Add per-step costs for time, energy or risk, scaled so they never outweigh the main goal.
  3. Use potential-based shaping rather than ad hoc bonuses when learning is too slow.
  4. Normalize or clip rewards to a stable range, often between minus one and plus one, to keep gradient updates well conditioned.
  5. Evaluate trained policies against the real objective, not only against the reward curve.

Scale matters as much as structure. Multiplying every reward by a positive constant leaves the optimal policy unchanged, but in deep reinforcement learning it changes gradient magnitudes and value targets, so reward normalization is a standard stabilizing step in many implementations.

Reward hacking and other failure modes

The central risk is reward hacking, also called specification gaming: the agent finds a way to score highly that the designer did not intend. Because an optimizer exploits every gap between the reward and the real goal, a stronger agent finds more loopholes, not fewer.

SymptomLikely causeRemedy
Agent loops to collect a bonusNon-potential shaping rewardSwitch to potential-based shaping
Agent ends episodes earlyNegative per-step reward too largeRebalance step cost and goal reward
No learning progress at allReward too sparseAdd shaping, curriculum or intrinsic reward
High reward, wrong behaviorProxy metric misses the goalMeasure the true outcome, add constraints
Policy exploits a learned reward modelOver-optimization of the modelAdd a KL penalty and refresh the model
Typical reward failures and remedies

A related problem is reward misspecification under distribution shift: a function that worked in simulation can reward unsafe shortcuts in the real world. Safe reinforcement learning addresses this with explicit constraints alongside the reward, rather than trying to encode every safety rule as a penalty term.

Stripped to the essentials

A reward function is the scorekeeper for a learning machine. After every move, it hands out points, and the machine slowly figures out which habits earn the most points over time.

The catch is that the machine cares only about the points, not about what the designer meant. If the scoring rules have a loophole, the machine will find it. Writing fair, complete scoring rules is therefore the most important part of teaching it a task.

Where it is applied

A warehouse robotics team trains a mobile robot to move shelves to packing stations. Their first reward function gave plus one for each meter traveled toward the target. Within a few hours of simulated training, the robot learned to rock back and forth near the start, repeatedly collecting progress reward without ever delivering a shelf.

The team redesigned the function: plus ten for a completed delivery, minus ten for a collision, a small time cost of minus 0.01 per step, and a potential-based shaping term using the negative distance to the station. With this design the robot completed over 95 percent of simulated deliveries after training, and the loophole disappeared because the shaping term could no longer be farmed by circling.