A reward function is the mapping that assigns a scalar numerical signal to each step of an agent’s interaction with its environment, usually written as R(s, a, s') for the state, the action taken and the next state. In reinforcement learning, it is the only formal statement of the task: the agent never sees the designer’s intentions, only the numbers the reward function produces, and it learns a policy that maximizes the expected cumulative reward.
The reward function, often called the reward signal, is one of the five elements of a Markov decision process together with states, actions, transition dynamics and the discount factor. Because every downstream algorithm, from Q-learning to proximal policy optimization, optimizes whatever the reward measures, reward design is frequently the hardest part of applying reinforcement learning to a real problem.
How a reward function drives learning
At every time step the environment returns a reward, and the agent’s objective is the discounted return: the sum of future rewards, each multiplied by the discount factor gamma raised to the number of steps ahead. Value functions, Q-functions and advantage estimates are all expectations of this return, so the reward function indirectly defines every quantity the agent estimates.
The reward is a local, immediate signal, while the goal is long-term. The Bellman equation links the two by expressing the value of a state as the immediate reward plus the discounted value of the next state. This recursive structure lets the agent propagate credit backward from rare successes to the earlier decisions that caused them, which is the essence of the credit assignment problem.
The reward function can take several mathematical forms, and the choice affects both expressiveness and learning speed. The table below lists the common signatures.
| Form | Depends on | Typical use |
|---|---|---|
| State reward R(s) | Current state only | Goal regions, grid worlds |
| State-action reward R(s, a) | State and chosen action | Penalizing costly actions such as energy use |
| Transition reward R(s, a, s’) | State, action and next state | Rewarding progress between states |
| Stochastic reward | A distribution over outcomes | Bandits, noisy real-world measurements |
| Learned reward model | A neural network trained on labels | Preference-based training of language models |
Dense, sparse and shaped rewards
Reward functions differ above all in how often they provide informative feedback. A sparse reward gives a nonzero value only at rare events, such as winning a game, while a dense reward scores almost every step. The density determines how much exploration the agent needs before it receives any useful learning signal.
| Design | Example | Strength | Main risk |
|---|---|---|---|
| Sparse reward | Plus one on reaching the goal, zero otherwise | Hard to exploit, matches the true goal | Very slow learning, weak signal |
| Dense reward | Negative distance to the target each step | Fast, informative feedback | Easier to game, local optima |
| Shaped reward | Sparse goal plus potential-based bonus | Speeds learning safely | Wrong shaping changes behavior |
| Intrinsic reward | Curiosity or novelty bonus | Drives exploration in sparse tasks | Agent chases noise |
| Preference-based reward | Model trained on human comparisons | Captures hard-to-specify goals | Reward model errors are exploited |
Reward shaping adds extra terms to guide the agent. Potential-based shaping, defined as gamma times the potential of the next state minus the potential of the current state, is proven to leave the optimal policy unchanged. Arbitrary bonuses carry no such guarantee and can create loops in which the agent collects shaping reward without completing the task.
Reward function compared with related concepts
The reward function is often confused with neighboring quantities, but each plays a distinct role. The reward is given by the environment or the designer, while value functions and policies are learned by the agent from it.
| Concept | What it represents | Who defines it | Time horizon |
|---|---|---|---|
| Reward function | Immediate feedback per step | Designer or environment | One step |
| Cumulative reward | Discounted sum of rewards | Derived from rewards | Whole episode |
| Value function | Expected return from a state | Learned by the agent | Future steps |
| Policy | Mapping from states to actions | Learned by the agent | Decision rule |
| Cost function | Negative reward in control theory | Designer | One step or trajectory |
Inverse reinforcement learning reverses the usual direction: it infers a reward function from expert demonstrations. The same idea underlies reinforcement learning from human feedback, where a reward model learned from preference pairs replaces a hand-written function and is then optimized with algorithms such as proximal policy optimization or group relative policy optimization.
Designing a reward function in practice
Good reward design starts by stating the true objective in measurable terms and then adding only the guidance the agent genuinely needs. Practitioners typically follow a short sequence of steps.
- Define the terminal success and failure conditions and give them clearly dominant values.
- Add per-step costs for time, energy or risk, scaled so they never outweigh the main goal.
- Use potential-based shaping rather than ad hoc bonuses when learning is too slow.
- Normalize or clip rewards to a stable range, often between minus one and plus one, to keep gradient updates well conditioned.
- Evaluate trained policies against the real objective, not only against the reward curve.
Scale matters as much as structure. Multiplying every reward by a positive constant leaves the optimal policy unchanged, but in deep reinforcement learning it changes gradient magnitudes and value targets, so reward normalization is a standard stabilizing step in many implementations.
Reward hacking and other failure modes
The central risk is reward hacking, also called specification gaming: the agent finds a way to score highly that the designer did not intend. Because an optimizer exploits every gap between the reward and the real goal, a stronger agent finds more loopholes, not fewer.
| Symptom | Likely cause | Remedy |
|---|---|---|
| Agent loops to collect a bonus | Non-potential shaping reward | Switch to potential-based shaping |
| Agent ends episodes early | Negative per-step reward too large | Rebalance step cost and goal reward |
| No learning progress at all | Reward too sparse | Add shaping, curriculum or intrinsic reward |
| High reward, wrong behavior | Proxy metric misses the goal | Measure the true outcome, add constraints |
| Policy exploits a learned reward model | Over-optimization of the model | Add a KL penalty and refresh the model |
A related problem is reward misspecification under distribution shift: a function that worked in simulation can reward unsafe shortcuts in the real world. Safe reinforcement learning addresses this with explicit constraints alongside the reward, rather than trying to encode every safety rule as a penalty term.
Stripped to the essentials
A reward function is the scorekeeper for a learning machine. After every move, it hands out points, and the machine slowly figures out which habits earn the most points over time.
The catch is that the machine cares only about the points, not about what the designer meant. If the scoring rules have a loophole, the machine will find it. Writing fair, complete scoring rules is therefore the most important part of teaching it a task.
Where it is applied
A warehouse robotics team trains a mobile robot to move shelves to packing stations. Their first reward function gave plus one for each meter traveled toward the target. Within a few hours of simulated training, the robot learned to rock back and forth near the start, repeatedly collecting progress reward without ever delivering a shelf.
The team redesigned the function: plus ten for a completed delivery, minus ten for a collision, a small time cost of minus 0.01 per step, and a potential-based shaping term using the negative distance to the station. With this design the robot completed over 95 percent of simulated deliveries after training, and the loophole disappeared because the shaping term could no longer be farmed by circling.