On-policy learning

On this page 7

On-policy learning is a family of reinforcement learning methods in which the agent evaluates and improves the same policy that it uses to choose actions and gather experience. The policy that generates data, called the behavior policy, is identical to the policy being learned, called the target policy, so every update reflects the agent’s current way of acting.

This design makes on-policy algorithms such as SARSA, REINFORCE, actor-critic variants and proximal policy optimization comparatively stable and easy to reason about. The price is lower sample efficiency: experience collected under an older policy becomes outdated after each update and is usually discarded rather than reused.

How on-policy learning works

On-policy learning works as a repeating cycle of collecting fresh experience with the current policy and then updating that policy using only this experience. Because the data distribution always matches the policy being improved, estimates of value functions and policy gradients are unbiased with respect to current behavior, without correction terms.

A typical iteration contains four steps, and the loop continues until performance plateaus or a training budget is exhausted.

  1. Run the current policy in the environment to collect trajectories of states, actions and rewards.
  2. Estimate returns or advantages for the visited states, often with generalized advantage estimation.
  3. Update the policy and, if present, the value function using that batch.
  4. Discard or retire the batch, because it no longer matches the updated policy.

Exploration must therefore come from the policy itself. Value-based on-policy methods use rules such as epsilon-greedy action selection, while policy gradient methods rely on stochastic policies and often add an entropy bonus so that the agent does not collapse prematurely to a single action.

On-policy versus off-policy learning

The difference between on-policy and off-policy learning lies in whether the data-generating policy and the learned policy are the same. Off-policy methods such as Q-learning learn about one policy, typically the greedy optimal one, while following another, which allows them to reuse old data through experience replay.

PropertyOn-policy learningOff-policy learning
Data sourceCurrent policy onlyAny behavior policy, including old data
Sample efficiencyLower, data used brieflyHigher, data reused many times
Training stabilityGenerally more stableProne to divergence with function approximation
Experience replayNot used or very limitedCentral component
ExplorationBuilt into the learned policyCan be separated from the target policy
Typical algorithmsSARSA, REINFORCE, A3C, PPOQ-learning, DQN, DDPG, SAC
Key differences between on-policy and off-policy learning

The classic illustration is the contrast between SARSA and Q-learning. SARSA updates with the action actually chosen next, Q(s,a) ← Q(s,a) + α[r + γQ(s′,a′) − Q(s,a)], whereas Q-learning substitutes the maximum over next actions. In the well-known cliff-walking task with epsilon-greedy exploration, SARSA learns a safer path away from the cliff, because it accounts for its own occasional random moves, while Q-learning learns the shorter but riskier optimal path.

Main on-policy algorithms

The main on-policy algorithms span both value-based and policy-based approaches, and several of the most widely used deep reinforcement learning methods belong to this group. The table summarizes their core mechanisms.

AlgorithmTypeCore mechanismData reuse
SARSAValue-based, tabular or approximateTemporal difference update with the next chosen actionSingle step
REINFORCEPolicy gradientMonte Carlo returns weight log-probability gradientsOne episode, one update
A2C and A3CActor-criticCritic estimates advantages for the actorShort rollouts, one update
TRPOPolicy gradientConstrains each update by KL divergenceOne batch per update
PPOPolicy gradientClipped probability ratio limits policy changeSeveral epochs on one batch
GRPOPolicy gradient for language modelsGroup-relative advantages without a criticSeveral epochs on one batch
Representative on-policy algorithms

PPO and TRPO are often described as nearly on-policy. They reuse a batch for a few optimization epochs and correct for the small mismatch with an importance sampling ratio, but clipping or a trust region keeps the updated policy close enough that the data remains approximately on-policy.

When to choose on-policy learning

On-policy learning is the better choice when environment interaction is cheap and stability matters more than the number of samples. Fast simulators, games and massively parallel environments make it easy to generate fresh data, which removes the main drawback.

SituationRecommended approachReason
Fast simulator with parallel workersOn-policy, such as PPOFresh data is cheap and training is robust
Physical robot with costly trialsOff-policy, such as SACEvery sample must be reused
Fine-tuning language models from rewardsOn-policy, such as PPO or GRPOUpdates must track current model outputs
Fixed historical dataset onlyOffline reinforcement learningNo new interaction is possible
Safety-sensitive explorationOn-policy with conservative updatesLearned values reflect actual behavior
Decision guide for on-policy and off-policy methods

Limitations and common misconceptions

The limitations of on-policy learning stem mostly from its strict data requirements, and several misunderstandings appear repeatedly in practice.

  • Sample inefficiency can make training require millions or billions of environment steps for complex continuous control tasks.
  • High-variance gradient estimates in pure Monte Carlo methods such as REINFORCE demand baselines or critics to train reliably.
  • Adding an experience replay buffer to an on-policy algorithm silently breaks its assumptions unless importance weights correct the mismatch.
  • Calling PPO fully off-policy is incorrect; its data reuse is deliberately limited to a few epochs per batch.
  • On-policy does not mean exploration-free; a deterministic policy without exploration noise stops discovering better actions.

Understanding it simply

On-policy learning means learning only from what you just did. Picture someone learning to cook by trying a recipe, tasting the result and adjusting the recipe. The next attempt uses the adjusted recipe, and notes from older versions are set aside because they describe a cook who no longer exists.

This keeps the lessons honest and up to date, but it is slow, because each meal is tasted once and then forgotten. Off-policy learning is more like reading other people’s cooking notes as well: more information, but some of it no longer fits the way you cook.

Applied in practice

A robotics research group trains a four-legged robot to walk over rough terrain entirely in simulation before deploying it on hardware. The team runs 4,096 simulated robots in parallel on a single GPU, each collecting 24 steps per iteration, which yields about 98,000 fresh transitions every update cycle.

They use proximal policy optimization, reuse each batch for five epochs with a clipping range of 0.2 and then discard it. Because fresh simulated experience is nearly free, the poor sample efficiency matters little, while the stable updates let the policy reach a reliable walking gait within a few hours of training. The trained policy is then transferred to the physical robot, where randomized terrain and sensor noise during training help it handle conditions it never saw exactly in simulation.