Reward shaping

On this page 7

Reward shaping is the technique of adding supplementary reward signals to a task so that an agent receives more frequent and informative feedback than the original reward provides. In reinforcement learning it is the standard response to sparse or delayed rewards, where an agent might act for thousands of steps before learning whether its behavior was useful.

Done correctly, reward shaping speeds up learning without changing the optimal policy. Done carelessly, it teaches the agent to chase the bonus instead of the goal, a failure known as reward hacking. The difference between the two outcomes is well understood in theory and is the core of the technique.

Why sparse rewards make shaping necessary

Reward shaping exists because many tasks give a nonzero reward only at the end. A robot arm that earns +1 only when a peg enters a hole, or a maze agent rewarded only at the exit, receives no gradient of progress. Random exploration may almost never reach the goal, so value estimates stay flat and learning stalls.

Shaping also eases the credit assignment problem. When a single terminal reward must be propagated back through hundreds of actions by temporal difference learning, each update carries little information. Intermediate rewards place useful signal closer to the actions that caused it, which shortens the effective horizon the agent must reason over.

How potential-based reward shaping works

Potential-based reward shaping is the form that provably preserves the optimal policy. The designer defines a potential function Φ(s) over states that estimates how promising each state is, and the shaping term is F(s, s') = γΦ(s') − Φ(s), where γ is the discount factor. The agent then learns from the original reward plus F.

Ng, Harada and Russell showed in 1999 that this form is both necessary and sufficient for policy invariance in a Markov decision process without further assumptions. The intuition is that shaping terms telescope along any trajectory: the bonus earned for approaching a good state is paid back when the agent leaves it, so no loop of states can generate free reward.

A useful corollary is that the ideal potential is the optimal value function V*(s). With that choice the shaped problem becomes almost trivial, which explains why a good heuristic estimate of progress makes such an effective potential.

Shaping familyForm of the extra rewardPreserves optimal policy
Potential-based (PBRS)γΦ(s′) − Φ(s)Yes, guaranteed
Potential-based advicePotential over state-action pairsYes, with correct form
Dynamic potential-basedPotential that changes over timeYes, under stated conditions
Ad hoc bonusHand-picked rewards for subgoalsNo guarantee
Intrinsic motivationNovelty or curiosity bonusNo, unless it decays
Main families of reward shaping

Designing a potential function

A good potential function encodes domain knowledge about progress while staying cheap to compute. Typical sources are geometric distance to a goal, the number of completed subtasks, or a value estimate from a simpler related task. The potential should be monotonic in true progress, because a potential that rises in dead ends actively misleads exploration.

Scale matters as much as shape. If the shaping term dominates the environment reward, value estimates become noisy and slow to correct; if it is negligible, it has no effect. Practitioners therefore normalize Φ to a range comparable with the task reward and tune its weight like any other hyperparameter.

Task typeExample potentialMain caution
NavigationNegative distance to goalWalls make straight-line distance misleading
Robotic manipulationGripper-to-object distanceIgnores grasp quality
Multi-step puzzlesCount of solved subgoalsSubgoal order may matter
GamesMaterial or score advantageEncourages greedy short-term play
Typical potential functions by task type

Reward shaping is one of several tools that change what the agent learns from, and it is often confused with its neighbors. It modifies the reward function directly, whereas curriculum learning modifies the task sequence and imitation methods use expert demonstrations instead of reward.

ApproachWhat it changesRelation to reward shaping
Reward shapingAdds terms to the rewardThe technique itself
Intrinsic motivationAdds task-agnostic exploration bonusGeneric shaping without domain knowledge
Inverse reinforcement learningInfers the reward from demonstrationsCan supply a learned potential
Curriculum learningOrders tasks from easy to hardComplements shaping, does not alter reward
Hierarchical reinforcement learningAdds subgoals and optionsSubgoal rewards are a form of shaping
Reward shaping and neighboring approaches

Risks and failure modes of reward shaping

The main risk of reward shaping is that the agent optimizes the proxy instead of the intended goal. Shaping terms that are not potential-based can create positive reward cycles, and a capable learner will find and exploit them. These failures are especially common with powerful optimizers such as deep reinforcement learning and proximal policy optimization.

SymptomLikely causeFix
Agent circles near the goalBonus for approaching, no penalty for leavingConvert to potential-based form
High shaped return, task unsolvedShaping term dominates rewardLower its weight or anneal it
Agent stops exploringPotential traps in local optimumRevise Φ or add exploration bonus
Good training, poor deploymentPotential uses unavailable informationRestrict Φ to observable state
Symptoms, causes and fixes

A well-known illustration is the boat-racing agent that learned to spin in circles collecting bonus targets rather than finish the race. The lesson is to evaluate on the unshaped objective: shaped reward helps training, but success must always be measured against the original task.

The short version

Reward shaping is like giving hints during a treasure hunt. If the only reward is finding the treasure, a beginner may wander for a very long time without learning anything. Saying “warmer” and “colder” along the way helps much faster.

The hints must be fair, though. If someone earns points just for moving toward the treasure but loses nothing for walking away, they may pace back and forth forever. Good hints guide without changing the real goal, and that is exactly what careful reward shaping does for a learning program.

Where it makes a difference

A robotics team trains a simulated arm to insert a peg into a hole with a tolerance of about one millimeter. The original reward is +1 on successful insertion and 0 otherwise. After five million environment steps with soft actor-critic, the agent has almost never succeeded, so it has learned essentially nothing.

The engineers add a potential-based shaping term with Φ equal to the negative distance between the peg tip and the hole, scaled to stay below the success reward, plus a second potential for correct peg orientation. Because both terms follow the γΦ(s′) − Φ(s) form, the optimal insertion policy is unchanged.

With shaping, the agent reaches a success rate above 90% in under one million steps. The team still reports results only on the original sparse reward, which confirms that the policy learned the task itself rather than the hints.