Trust region policy optimization (TRPO) is a policy gradient algorithm in reinforcement learning that improves a stochastic policy by maximizing a surrogate objective while keeping the new policy within a trust region defined by the KL divergence from the old policy. The constraint prevents a single update from changing behavior so much that performance collapses.
TRPO was introduced in 2015 by Schulman and colleagues as a practical approximation of a theoretical result that guarantees monotonic policy improvement. It became a reference method for continuous control with neural network policies and directly inspired proximal policy optimization, its simpler and now more widely used successor.
The problem TRPO solves in policy gradient learning
TRPO addresses the instability of plain policy gradient updates. Methods such as REINFORCE and basic actor-critic take a gradient step in parameter space, but a small change in parameters can produce a large change in action probabilities, especially in deep networks. One overly large step can push the policy into a region where it collects poor data, and recovery may never happen.
The key insight is that step size should be measured in policy space, not parameter space. TRPO therefore limits how far the action distribution may move, measured by the average Kullback-Leibler divergence between the old and new policies over visited states. This makes the update size meaningful regardless of how the network is parameterized.
How the TRPO update works
Each TRPO iteration collects trajectories with the current policy, estimates advantages, and then solves a constrained optimization problem. The objective is the expected advantage weighted by the probability ratio between new and old policies, which estimates how much the new policy would improve on data gathered by the old one.
Formally, TRPO maximizes the mean of ratio times advantage subject to the mean KL divergence staying below a threshold delta, typically 0.01 to 0.05. Solving this exactly is impractical, so TRPO uses a linear approximation of the objective and a quadratic approximation of the constraint, which turns the problem into a natural gradient step.
- Run the current policy to collect a batch of state, action and reward samples.
- Estimate advantages, often with generalized advantage estimation and a learned value function.
- Compute the policy gradient of the surrogate objective.
- Use the conjugate gradient method to approximate the natural gradient direction.
- Scale the step to the KL limit and run a backtracking line search.
- Accept the step only if the surrogate improves and the KL constraint holds.
The conjugate gradient method avoids building or inverting the full Fisher information matrix, which would be infeasible for networks with many parameters. It needs only Fisher-vector products, which can be computed efficiently with automatic differentiation in about ten iterations.
Key components and hyperparameters
TRPO has fewer sensitive hyperparameters than many policy gradient methods, because the KL limit replaces a learning rate for the policy. The table below lists the settings that matter most and their usual ranges.
| Parameter | Typical value | Effect |
|---|---|---|
| Maximum KL divergence | 0.01 to 0.05 | Controls the size of each policy update |
| Conjugate gradient iterations | 10 to 20 | Accuracy of the natural gradient direction |
| Fisher damping coefficient | 0.01 to 0.1 | Stabilizes the curvature estimate |
| Line search backtracks | Up to 10 | Shrinks steps that violate constraints |
| Batch size per update | 1,000 to 50,000 steps | Lower gradient variance at higher sample cost |
| GAE lambda | 0.95 to 0.98 | Bias-variance balance in advantage estimates |
| Discount factor | 0.99 to 0.995 | Effective planning horizon |
The value function is trained separately, usually by regression on returns, and only serves to reduce variance in the advantage function. TRPO itself is an on-policy method, so each batch is discarded after one policy update.
TRPO compared with related algorithms
TRPO sits between vanilla policy gradients and the clipped objectives that followed it. The comparison below shows how it differs from neighboring methods in the same family.
| Algorithm | Step control | Data use | Implementation effort |
|---|---|---|---|
| REINFORCE | Learning rate only | On-policy, one pass | Low |
| Natural policy gradient | Fisher-scaled fixed step | On-policy | Medium |
| TRPO | Hard KL constraint with line search | On-policy, one update per batch | High |
| Proximal policy optimization | Clipped probability ratio | On-policy, several epochs | Low to medium |
| Soft actor-critic | Entropy-regularized objective | Off-policy with replay | Medium |
Proximal policy optimization replaced TRPO in most practice because clipping the probability ratio achieves similar stability with first-order optimizers and allows several passes over each batch. TRPO still offers a more explicit guarantee on how far each update moves, which some researchers prefer for analysis and for safety-sensitive experiments.
Strengths, limitations and practical pitfalls
TRPO is valued for reliable, monotonic-looking learning curves across many tasks without heavy tuning. Its main drawbacks are computational cost and sample inefficiency, both consequences of second-order approximations and on-policy data collection.
| Aspect | Strength or limitation | Practical consequence |
|---|---|---|
| Stability | Strength | Rarely collapses after a bad update |
| Tuning | Strength | KL limit transfers well between tasks |
| Sample efficiency | Limitation | Needs many environment interactions |
| Compute per update | Limitation | Conjugate gradient and line search add overhead |
| Architecture flexibility | Limitation | Shared actor-critic networks and dropout complicate the constraint |
A common pitfall is assuming the theoretical monotonic improvement guarantee holds in practice. The guarantee applies to an idealized penalty form, while practical TRPO uses an average KL constraint and sampled estimates, so performance can still decrease occasionally. Another pitfall is a poorly fitted value function, which produces noisy advantages that no trust region can fully compensate for.
The short version
TRPO is a way to teach an agent through trial and error without letting it change its habits too abruptly. After each round of practice, the agent adjusts its behavior, but only within a safe distance of what it did before.
Think of a learner driver whose instructor allows small corrections at a time instead of a completely new driving style after every lesson. Progress may be slower per lesson, but it is steady and rarely goes backward.
Where it makes a difference
A robotics research group trains a simulated four-legged walking robot in a physics simulator. With a plain policy gradient method, the robot sometimes learns a promising gait and then, after one large update, falls over at every step and never recovers within the training budget.
The team switches to TRPO with a KL limit of 0.01, batches of 25,000 simulation steps and generalized advantage estimation with lambda 0.97. Training takes longer per iteration, but across five random seeds the learning curves rise steadily and the robot reaches a stable trotting gait in every run. The group later uses this TRPO baseline to check whether a faster PPO configuration matches its reliability.