Proximal policy optimization

Also known as: PPO

On this page 7

Proximal policy optimization (PPO) is an on-policy policy gradient algorithm that improves an agent’s policy in small, controlled steps by clipping the probability ratio between the new and old policy. In reinforcement learning, it is one of the most widely used methods because it combines much of the stability of trust region methods with the simplicity of ordinary first-order gradient descent.

PPO belongs to the actor-critic family: an actor network outputs action probabilities and a critic network estimates the value function used to compute advantages. Introduced in 2017 as a simpler successor to trust region policy optimization, it became a default baseline for continuous control, game playing and the reinforcement learning from human feedback stage used to fine-tune large language models.

How the clipped objective keeps updates small

PPO limits each update by optimizing a clipped surrogate objective instead of the raw policy gradient. For every sampled action, it computes the probability ratio, the action’s probability under the new policy divided by its probability under the policy that collected the data, and multiplies that ratio by the estimated advantage.

The ratio is clipped to the interval from 1 minus epsilon to 1 plus epsilon, with epsilon typically 0.1 to 0.2. The objective takes the minimum of the clipped and unclipped terms, which removes any incentive to push the ratio further once an action has already become sufficiently more or less likely. The result is a pessimistic lower bound on improvement that discourages destructive policy jumps.

A complete PPO iteration follows a fixed cycle:

  1. Collect rollouts with the current policy, often from many parallel environments.
  2. Estimate advantages, usually with generalized advantage estimation.
  3. Run several epochs of minibatch stochastic gradient ascent on the clipped objective, a value loss and an entropy bonus.
  4. Discard the data and repeat, because the samples are now off-policy.

Key hyperparameters and typical values

PPO is robust compared with earlier policy gradient methods, but its performance still depends heavily on hyperparameters and implementation details. The following table lists the settings that matter most and the ranges commonly used in published baselines.

HyperparameterTypical rangeEffect when increased
Clip range epsilon0.1 to 0.3Larger, riskier policy steps
Learning rate1e-5 to 3e-4Faster but less stable learning
Epochs per batch3 to 10Better sample reuse, more overfitting to batch
Discount factor gamma0.95 to 0.999Longer planning horizon
GAE lambda0.9 to 0.98Lower bias, higher variance
Entropy coefficient0 to 0.01More exploration
Rollout length128 to 4096 stepsLower-variance gradient estimates
Core PPO hyperparameters

Beyond these parameters, practical implementations rely on advantage normalization, observation normalization, orthogonal weight initialization, gradient norm clipping and learning rate annealing. Studies of PPO implementations have shown that such code-level details can change results as much as the clipping mechanism itself.

PPO compared with other policy optimization methods

PPO occupies a middle ground between simple policy gradients and constrained optimization methods. It trades the formal guarantees of TRPO for easier implementation, and it trades the sample efficiency of off-policy methods for stability and generality across discrete and continuous action spaces.

AlgorithmPolicy typeUpdate constraintMain trade-off
REINFORCEOn-policyNoneSimple but high variance
TRPOOn-policyHard KL-divergence limitStable but complex second-order math
PPOOn-policyClipped probability ratioSimple and stable, sample-hungry
Soft actor-criticOff-policyEntropy-regularized objectiveSample-efficient, continuous actions only
GRPOOn-policyClipping plus group baselineNo critic, needs grouped samples
How PPO relates to neighboring algorithms

A second PPO variant replaces clipping with an adaptive KL-divergence penalty that is increased or decreased depending on how far the new policy drifts. The clipped version became dominant because it performed as well or better with fewer tuning decisions.

Where PPO is used in practice

PPO is the first algorithm many teams try when the environment can be simulated cheaply and in parallel. Its tolerance to hyperparameter choices makes it suitable for problems where extensive tuning is impractical.

ApplicationAction spaceWhy PPO fits
Robot locomotion in simulationContinuousStable learning with massive parallel rollouts
Video game agentsDiscreteHandles large observation spaces with CNNs
Language model alignmentDiscrete tokensKL-controlled updates against a reward model
Resource schedulingDiscreteWorks with masked invalid actions
Multi-agent self-playMixedScales to distributed training
Application areas and why PPO fits

In language model alignment, PPO optimizes the model against a learned reward model while a KL penalty toward the reference model prevents the text distribution from drifting into reward hacking. Newer methods such as group relative policy optimization and direct preference optimization have reduced this role because they avoid training a separate critic.

Limitations and common failure modes

PPO’s main weakness is poor sample efficiency: because it is on-policy, every batch is used for a few epochs and then discarded. When environment interactions are expensive, such as with physical robots, off-policy or model-based reinforcement learning is usually a better choice.

  • Clipping is not a true trust region: the ratio can still exceed the bounds after several epochs, so many implementations add early stopping on approximate KL.
  • Value function errors propagate into advantages and can stall learning even when the actor is sound.
  • Sparse rewards remain hard, since PPO explores mainly through policy entropy.
  • Entropy collapse happens when the policy becomes deterministic too early and stops exploring.
  • Reproducibility issues arise because results vary noticeably across random seeds.

Monitoring the clip fraction, approximate KL divergence, explained variance and policy entropy during training is the most reliable way to detect these problems before they waste compute.

A non-technical view

Imagine teaching someone to drive by letting them try, then telling them which moves went well. If they change their whole driving style after every lesson, they may get worse instead of better. PPO is a rule that says: improve, but only a little at a time.

After each round of practice, the computer program adjusts its behavior toward what earned rewards, yet it refuses changes that are too drastic. This caution makes learning steady and predictable, which is why PPO is popular for training game characters, robots and even chat assistants.

A concrete example

A robotics research group wants a four-legged robot to walk across uneven terrain. Testing on the real machine would take months and risk damage, so the team trains in a physics simulator running 4,096 environments in parallel on a single GPU.

They use PPO with a clip range of 0.2, rollouts of 24 steps per environment, five epochs per batch and a reward that favors forward speed while penalizing energy use and falls. After about two billion simulated steps, completed in under a day, the policy walks steadily over slopes and small obstacles. With randomized friction and mass during training, the learned controller transfers to the physical robot with only minor adjustments.