Proximal policy optimization (PPO) is an on-policy policy gradient algorithm that improves an agent’s policy in small, controlled steps by clipping the probability ratio between the new and old policy. In reinforcement learning, it is one of the most widely used methods because it combines much of the stability of trust region methods with the simplicity of ordinary first-order gradient descent.
PPO belongs to the actor-critic family: an actor network outputs action probabilities and a critic network estimates the value function used to compute advantages. Introduced in 2017 as a simpler successor to trust region policy optimization, it became a default baseline for continuous control, game playing and the reinforcement learning from human feedback stage used to fine-tune large language models.
How the clipped objective keeps updates small
PPO limits each update by optimizing a clipped surrogate objective instead of the raw policy gradient. For every sampled action, it computes the probability ratio, the action’s probability under the new policy divided by its probability under the policy that collected the data, and multiplies that ratio by the estimated advantage.
The ratio is clipped to the interval from 1 minus epsilon to 1 plus epsilon, with epsilon typically 0.1 to 0.2. The objective takes the minimum of the clipped and unclipped terms, which removes any incentive to push the ratio further once an action has already become sufficiently more or less likely. The result is a pessimistic lower bound on improvement that discourages destructive policy jumps.
A complete PPO iteration follows a fixed cycle:
- Collect rollouts with the current policy, often from many parallel environments.
- Estimate advantages, usually with generalized advantage estimation.
- Run several epochs of minibatch stochastic gradient ascent on the clipped objective, a value loss and an entropy bonus.
- Discard the data and repeat, because the samples are now off-policy.
Key hyperparameters and typical values
PPO is robust compared with earlier policy gradient methods, but its performance still depends heavily on hyperparameters and implementation details. The following table lists the settings that matter most and the ranges commonly used in published baselines.
| Hyperparameter | Typical range | Effect when increased |
|---|---|---|
| Clip range epsilon | 0.1 to 0.3 | Larger, riskier policy steps |
| Learning rate | 1e-5 to 3e-4 | Faster but less stable learning |
| Epochs per batch | 3 to 10 | Better sample reuse, more overfitting to batch |
| Discount factor gamma | 0.95 to 0.999 | Longer planning horizon |
| GAE lambda | 0.9 to 0.98 | Lower bias, higher variance |
| Entropy coefficient | 0 to 0.01 | More exploration |
| Rollout length | 128 to 4096 steps | Lower-variance gradient estimates |
Beyond these parameters, practical implementations rely on advantage normalization, observation normalization, orthogonal weight initialization, gradient norm clipping and learning rate annealing. Studies of PPO implementations have shown that such code-level details can change results as much as the clipping mechanism itself.
PPO compared with other policy optimization methods
PPO occupies a middle ground between simple policy gradients and constrained optimization methods. It trades the formal guarantees of TRPO for easier implementation, and it trades the sample efficiency of off-policy methods for stability and generality across discrete and continuous action spaces.
| Algorithm | Policy type | Update constraint | Main trade-off |
|---|---|---|---|
| REINFORCE | On-policy | None | Simple but high variance |
| TRPO | On-policy | Hard KL-divergence limit | Stable but complex second-order math |
| PPO | On-policy | Clipped probability ratio | Simple and stable, sample-hungry |
| Soft actor-critic | Off-policy | Entropy-regularized objective | Sample-efficient, continuous actions only |
| GRPO | On-policy | Clipping plus group baseline | No critic, needs grouped samples |
A second PPO variant replaces clipping with an adaptive KL-divergence penalty that is increased or decreased depending on how far the new policy drifts. The clipped version became dominant because it performed as well or better with fewer tuning decisions.
Where PPO is used in practice
PPO is the first algorithm many teams try when the environment can be simulated cheaply and in parallel. Its tolerance to hyperparameter choices makes it suitable for problems where extensive tuning is impractical.
| Application | Action space | Why PPO fits |
|---|---|---|
| Robot locomotion in simulation | Continuous | Stable learning with massive parallel rollouts |
| Video game agents | Discrete | Handles large observation spaces with CNNs |
| Language model alignment | Discrete tokens | KL-controlled updates against a reward model |
| Resource scheduling | Discrete | Works with masked invalid actions |
| Multi-agent self-play | Mixed | Scales to distributed training |
In language model alignment, PPO optimizes the model against a learned reward model while a KL penalty toward the reference model prevents the text distribution from drifting into reward hacking. Newer methods such as group relative policy optimization and direct preference optimization have reduced this role because they avoid training a separate critic.
Limitations and common failure modes
PPO’s main weakness is poor sample efficiency: because it is on-policy, every batch is used for a few epochs and then discarded. When environment interactions are expensive, such as with physical robots, off-policy or model-based reinforcement learning is usually a better choice.
- Clipping is not a true trust region: the ratio can still exceed the bounds after several epochs, so many implementations add early stopping on approximate KL.
- Value function errors propagate into advantages and can stall learning even when the actor is sound.
- Sparse rewards remain hard, since PPO explores mainly through policy entropy.
- Entropy collapse happens when the policy becomes deterministic too early and stops exploring.
- Reproducibility issues arise because results vary noticeably across random seeds.
Monitoring the clip fraction, approximate KL divergence, explained variance and policy entropy during training is the most reliable way to detect these problems before they waste compute.
A non-technical view
Imagine teaching someone to drive by letting them try, then telling them which moves went well. If they change their whole driving style after every lesson, they may get worse instead of better. PPO is a rule that says: improve, but only a little at a time.
After each round of practice, the computer program adjusts its behavior toward what earned rewards, yet it refuses changes that are too drastic. This caution makes learning steady and predictable, which is why PPO is popular for training game characters, robots and even chat assistants.
A concrete example
A robotics research group wants a four-legged robot to walk across uneven terrain. Testing on the real machine would take months and risk damage, so the team trains in a physics simulator running 4,096 environments in parallel on a single GPU.
They use PPO with a clip range of 0.2, rollouts of 24 steps per environment, five epochs per batch and a reward that favors forward speed while penalizing energy use and falls. After about two billion simulated steps, completed in under a day, the policy walks steadily over slopes and small obstacles. With randomized friction and mass during training, the learned controller transfers to the physical robot with only minor adjustments.