Trust region policy optimization

Also known as: TRPO

On this page 7

Trust region policy optimization (TRPO) is a policy gradient algorithm in reinforcement learning that improves a stochastic policy by maximizing a surrogate objective while keeping the new policy within a trust region defined by the KL divergence from the old policy. The constraint prevents a single update from changing behavior so much that performance collapses.

TRPO was introduced in 2015 by Schulman and colleagues as a practical approximation of a theoretical result that guarantees monotonic policy improvement. It became a reference method for continuous control with neural network policies and directly inspired proximal policy optimization, its simpler and now more widely used successor.

The problem TRPO solves in policy gradient learning

TRPO addresses the instability of plain policy gradient updates. Methods such as REINFORCE and basic actor-critic take a gradient step in parameter space, but a small change in parameters can produce a large change in action probabilities, especially in deep networks. One overly large step can push the policy into a region where it collects poor data, and recovery may never happen.

The key insight is that step size should be measured in policy space, not parameter space. TRPO therefore limits how far the action distribution may move, measured by the average Kullback-Leibler divergence between the old and new policies over visited states. This makes the update size meaningful regardless of how the network is parameterized.

How the TRPO update works

Each TRPO iteration collects trajectories with the current policy, estimates advantages, and then solves a constrained optimization problem. The objective is the expected advantage weighted by the probability ratio between new and old policies, which estimates how much the new policy would improve on data gathered by the old one.

Formally, TRPO maximizes the mean of ratio times advantage subject to the mean KL divergence staying below a threshold delta, typically 0.01 to 0.05. Solving this exactly is impractical, so TRPO uses a linear approximation of the objective and a quadratic approximation of the constraint, which turns the problem into a natural gradient step.

  1. Run the current policy to collect a batch of state, action and reward samples.
  2. Estimate advantages, often with generalized advantage estimation and a learned value function.
  3. Compute the policy gradient of the surrogate objective.
  4. Use the conjugate gradient method to approximate the natural gradient direction.
  5. Scale the step to the KL limit and run a backtracking line search.
  6. Accept the step only if the surrogate improves and the KL constraint holds.

The conjugate gradient method avoids building or inverting the full Fisher information matrix, which would be infeasible for networks with many parameters. It needs only Fisher-vector products, which can be computed efficiently with automatic differentiation in about ten iterations.

Key components and hyperparameters

TRPO has fewer sensitive hyperparameters than many policy gradient methods, because the KL limit replaces a learning rate for the policy. The table below lists the settings that matter most and their usual ranges.

ParameterTypical valueEffect
Maximum KL divergence0.01 to 0.05Controls the size of each policy update
Conjugate gradient iterations10 to 20Accuracy of the natural gradient direction
Fisher damping coefficient0.01 to 0.1Stabilizes the curvature estimate
Line search backtracksUp to 10Shrinks steps that violate constraints
Batch size per update1,000 to 50,000 stepsLower gradient variance at higher sample cost
GAE lambda0.95 to 0.98Bias-variance balance in advantage estimates
Discount factor0.99 to 0.995Effective planning horizon
Main TRPO hyperparameters

The value function is trained separately, usually by regression on returns, and only serves to reduce variance in the advantage function. TRPO itself is an on-policy method, so each batch is discarded after one policy update.

TRPO sits between vanilla policy gradients and the clipped objectives that followed it. The comparison below shows how it differs from neighboring methods in the same family.

AlgorithmStep controlData useImplementation effort
REINFORCELearning rate onlyOn-policy, one passLow
Natural policy gradientFisher-scaled fixed stepOn-policyMedium
TRPOHard KL constraint with line searchOn-policy, one update per batchHigh
Proximal policy optimizationClipped probability ratioOn-policy, several epochsLow to medium
Soft actor-criticEntropy-regularized objectiveOff-policy with replayMedium
TRPO and related policy optimization methods

Proximal policy optimization replaced TRPO in most practice because clipping the probability ratio achieves similar stability with first-order optimizers and allows several passes over each batch. TRPO still offers a more explicit guarantee on how far each update moves, which some researchers prefer for analysis and for safety-sensitive experiments.

Strengths, limitations and practical pitfalls

TRPO is valued for reliable, monotonic-looking learning curves across many tasks without heavy tuning. Its main drawbacks are computational cost and sample inefficiency, both consequences of second-order approximations and on-policy data collection.

AspectStrength or limitationPractical consequence
StabilityStrengthRarely collapses after a bad update
TuningStrengthKL limit transfers well between tasks
Sample efficiencyLimitationNeeds many environment interactions
Compute per updateLimitationConjugate gradient and line search add overhead
Architecture flexibilityLimitationShared actor-critic networks and dropout complicate the constraint
Practical trade-offs of TRPO

A common pitfall is assuming the theoretical monotonic improvement guarantee holds in practice. The guarantee applies to an idealized penalty form, while practical TRPO uses an average KL constraint and sampled estimates, so performance can still decrease occasionally. Another pitfall is a poorly fitted value function, which produces noisy advantages that no trust region can fully compensate for.

The short version

TRPO is a way to teach an agent through trial and error without letting it change its habits too abruptly. After each round of practice, the agent adjusts its behavior, but only within a safe distance of what it did before.

Think of a learner driver whose instructor allows small corrections at a time instead of a completely new driving style after every lesson. Progress may be slower per lesson, but it is steady and rarely goes backward.

Where it makes a difference

A robotics research group trains a simulated four-legged walking robot in a physics simulator. With a plain policy gradient method, the robot sometimes learns a promising gait and then, after one large update, falls over at every step and never recovers within the training budget.

The team switches to TRPO with a KL limit of 0.01, batches of 25,000 simulation steps and generalized advantage estimation with lambda 0.97. Training takes longer per iteration, but across five random seeds the learning curves rise steadily and the robot reaches a stable trotting gait in every run. The group later uses this TRPO baseline to check whether a faster PPO configuration matches its reliability.