On-policy learning is a family of reinforcement learning methods in which the agent evaluates and improves the same policy that it uses to choose actions and gather experience. The policy that generates data, called the behavior policy, is identical to the policy being learned, called the target policy, so every update reflects the agent’s current way of acting.
This design makes on-policy algorithms such as SARSA, REINFORCE, actor-critic variants and proximal policy optimization comparatively stable and easy to reason about. The price is lower sample efficiency: experience collected under an older policy becomes outdated after each update and is usually discarded rather than reused.
How on-policy learning works
On-policy learning works as a repeating cycle of collecting fresh experience with the current policy and then updating that policy using only this experience. Because the data distribution always matches the policy being improved, estimates of value functions and policy gradients are unbiased with respect to current behavior, without correction terms.
A typical iteration contains four steps, and the loop continues until performance plateaus or a training budget is exhausted.
- Run the current policy in the environment to collect trajectories of states, actions and rewards.
- Estimate returns or advantages for the visited states, often with generalized advantage estimation.
- Update the policy and, if present, the value function using that batch.
- Discard or retire the batch, because it no longer matches the updated policy.
Exploration must therefore come from the policy itself. Value-based on-policy methods use rules such as epsilon-greedy action selection, while policy gradient methods rely on stochastic policies and often add an entropy bonus so that the agent does not collapse prematurely to a single action.
On-policy versus off-policy learning
The difference between on-policy and off-policy learning lies in whether the data-generating policy and the learned policy are the same. Off-policy methods such as Q-learning learn about one policy, typically the greedy optimal one, while following another, which allows them to reuse old data through experience replay.
| Property | On-policy learning | Off-policy learning |
|---|---|---|
| Data source | Current policy only | Any behavior policy, including old data |
| Sample efficiency | Lower, data used briefly | Higher, data reused many times |
| Training stability | Generally more stable | Prone to divergence with function approximation |
| Experience replay | Not used or very limited | Central component |
| Exploration | Built into the learned policy | Can be separated from the target policy |
| Typical algorithms | SARSA, REINFORCE, A3C, PPO | Q-learning, DQN, DDPG, SAC |
The classic illustration is the contrast between SARSA and Q-learning. SARSA updates with the action actually chosen next, Q(s,a) ← Q(s,a) + α[r + γQ(s′,a′) − Q(s,a)], whereas Q-learning substitutes the maximum over next actions. In the well-known cliff-walking task with epsilon-greedy exploration, SARSA learns a safer path away from the cliff, because it accounts for its own occasional random moves, while Q-learning learns the shorter but riskier optimal path.
Main on-policy algorithms
The main on-policy algorithms span both value-based and policy-based approaches, and several of the most widely used deep reinforcement learning methods belong to this group. The table summarizes their core mechanisms.
| Algorithm | Type | Core mechanism | Data reuse |
|---|---|---|---|
| SARSA | Value-based, tabular or approximate | Temporal difference update with the next chosen action | Single step |
| REINFORCE | Policy gradient | Monte Carlo returns weight log-probability gradients | One episode, one update |
| A2C and A3C | Actor-critic | Critic estimates advantages for the actor | Short rollouts, one update |
| TRPO | Policy gradient | Constrains each update by KL divergence | One batch per update |
| PPO | Policy gradient | Clipped probability ratio limits policy change | Several epochs on one batch |
| GRPO | Policy gradient for language models | Group-relative advantages without a critic | Several epochs on one batch |
PPO and TRPO are often described as nearly on-policy. They reuse a batch for a few optimization epochs and correct for the small mismatch with an importance sampling ratio, but clipping or a trust region keeps the updated policy close enough that the data remains approximately on-policy.
When to choose on-policy learning
On-policy learning is the better choice when environment interaction is cheap and stability matters more than the number of samples. Fast simulators, games and massively parallel environments make it easy to generate fresh data, which removes the main drawback.
| Situation | Recommended approach | Reason |
|---|---|---|
| Fast simulator with parallel workers | On-policy, such as PPO | Fresh data is cheap and training is robust |
| Physical robot with costly trials | Off-policy, such as SAC | Every sample must be reused |
| Fine-tuning language models from rewards | On-policy, such as PPO or GRPO | Updates must track current model outputs |
| Fixed historical dataset only | Offline reinforcement learning | No new interaction is possible |
| Safety-sensitive exploration | On-policy with conservative updates | Learned values reflect actual behavior |
Limitations and common misconceptions
The limitations of on-policy learning stem mostly from its strict data requirements, and several misunderstandings appear repeatedly in practice.
- Sample inefficiency can make training require millions or billions of environment steps for complex continuous control tasks.
- High-variance gradient estimates in pure Monte Carlo methods such as REINFORCE demand baselines or critics to train reliably.
- Adding an experience replay buffer to an on-policy algorithm silently breaks its assumptions unless importance weights correct the mismatch.
- Calling PPO fully off-policy is incorrect; its data reuse is deliberately limited to a few epochs per batch.
- On-policy does not mean exploration-free; a deterministic policy without exploration noise stops discovering better actions.
Understanding it simply
On-policy learning means learning only from what you just did. Picture someone learning to cook by trying a recipe, tasting the result and adjusting the recipe. The next attempt uses the adjusted recipe, and notes from older versions are set aside because they describe a cook who no longer exists.
This keeps the lessons honest and up to date, but it is slow, because each meal is tasted once and then forgotten. Off-policy learning is more like reading other people’s cooking notes as well: more information, but some of it no longer fits the way you cook.
Applied in practice
A robotics research group trains a four-legged robot to walk over rough terrain entirely in simulation before deploying it on hardware. The team runs 4,096 simulated robots in parallel on a single GPU, each collecting 24 steps per iteration, which yields about 98,000 fresh transitions every update cycle.
They use proximal policy optimization, reuse each batch for five epochs with a clipping range of 0.2 and then discard it. Because fresh simulated experience is nearly free, the poor sample efficiency matters little, while the stable updates let the policy reach a reliable walking gait within a few hours of training. The trained policy is then transferred to the physical robot, where randomized terrain and sensor noise during training help it handle conditions it never saw exactly in simulation.