A policy gradient method is a family of reinforcement learning algorithms that optimize a parameterized policy directly by following the gradient of the expected cumulative reward with respect to the policy parameters. Instead of learning a value for every action and then acting greedily, the agent adjusts the probabilities of its actions so that behavior leading to higher return becomes more likely.
Policy gradient methods are the foundation of modern algorithms such as REINFORCE, actor-critic, trust region policy optimization and proximal policy optimization, which is widely used for robotics control and for fine-tuning large language models with human feedback.
How the policy gradient theorem works
The method starts from a stochastic policy π(a | s; θ), typically a neural network that outputs a probability distribution over actions. The objective J(θ) is the expected return of trajectories generated by that policy, and learning performs gradient ascent on J(θ).
The difficulty is that the return depends on the environment dynamics, which are usually unknown. The policy gradient theorem solves this by showing that the gradient equals the expectation of ∇ log π(a | s; θ) multiplied by a return signal. This form, known as the likelihood ratio or score function estimator, needs only sampled actions and rewards, not a model of the environment.
Intuitively, the log-probability gradient points toward making the chosen action more likely, and the return signal scales that push up or down. Good actions are reinforced and poor actions are suppressed in proportion to their outcomes.
The table lists the main ingredients of the update.
| Element | Role | Typical form |
|---|---|---|
| Policy network | Maps states to action probabilities | Softmax or Gaussian output head |
| Score function | Direction that increases action likelihood | ∇ log π(a | s; θ) |
| Return signal | Weights each update by outcome | Return, advantage or TD error |
| Learning rate | Controls step size | Roughly 1e-5 to 1e-3 with Adam |
| Batch of trajectories | Provides the Monte Carlo estimate | Thousands of environment steps |
Reducing variance in gradient estimates
The raw estimator is unbiased but extremely noisy, because a single trajectory return mixes the effect of many actions and random events. High variance is the main practical weakness of policy gradient learning, and most algorithmic progress has targeted it.
The most important technique is subtracting a baseline, usually a learned state-value estimate, which leaves the gradient unbiased while removing much of the noise. The resulting quantity is the advantage, which measures how much better an action was than the policy average in that state.
| Return signal | Bias | Variance | Used in |
|---|---|---|---|
| Full episode return | None | Very high | Basic REINFORCE |
| Reward-to-go | None | High | Improved REINFORCE |
| Return minus baseline | None | Moderate | REINFORCE with baseline |
| TD error | Some | Low | One-step actor-critic |
| GAE advantage | Tunable via λ | Tunable | PPO, TRPO |
Generalized advantage estimation blends multi-step temporal difference errors with a parameter λ, usually between 0.9 and 0.99, giving practitioners a direct dial between bias and variance.
Main policy gradient algorithms
Policy gradient algorithms differ in how they estimate the return signal and how they limit the size of each update. Large steps can collapse a policy that took hours to train, so modern methods constrain how far the new policy may move from the old one.
| Algorithm | Key idea | Data use | Strength |
|---|---|---|---|
| REINFORCE | Monte Carlo score function estimate | On-policy | Simple and unbiased |
| Actor-critic | Critic supplies a learned baseline | On-policy | Lower variance, online updates |
| TRPO | KL divergence trust region | On-policy | Stable monotonic improvement |
| PPO | Clipped probability ratio | On-policy, several epochs | Robust and easy to tune |
| DDPG and TD3 | Deterministic policy gradient | Off-policy with replay | Sample-efficient continuous control |
Proximal policy optimization has become the default choice because its clipping range of about 0.1 to 0.3 offers most of the stability of trust region methods with a simple first-order optimizer.
Policy gradient versus value-based methods
Policy gradient methods and value-based methods such as Q-learning solve the same problem from opposite directions. Value-based methods learn action values and derive a policy implicitly; policy gradient methods learn the policy itself and may use values only as a helper.
| Aspect | Policy gradient | Value-based (Q-learning, DQN) |
|---|---|---|
| Action spaces | Discrete and continuous | Mainly discrete |
| Stochastic policies | Learned natively | Require added exploration rules |
| Sample efficiency | Lower, mostly on-policy | Higher, uses replay buffers |
| Convergence | To a local optimum, smoothly | Can diverge with function approximation |
| Gradient noise | High without baselines | Lower, bootstrapped targets |
The decisive advantage of policy gradients is natural handling of continuous actions, such as joint torques in a robot arm, where maximizing over a Q-function at every step would be impractical.
Practical pitfalls and failure modes
Policy gradient training is notoriously sensitive to implementation details. Experienced practitioners watch for the following problems:
- Premature entropy collapse: the policy becomes deterministic too early; an entropy bonus keeps exploration alive.
- Unnormalized advantages and rewards: wildly scaled signals destabilize learning; normalize them per batch.
- Oversized updates: a single large step can destroy performance; use clipping or a KL penalty.
- Stale data reuse: on-policy methods lose correctness when old samples are reused for too many epochs.
- Reward hacking: the policy exploits flaws in the reward function instead of solving the task.
- Seed sensitivity: results vary strongly between random seeds, so report several runs.
Because policy gradient methods converge only to local optima, initialization, network architecture and reward design often matter as much as the algorithm choice.
The simple explanation
A policy gradient method teaches an agent by nudging its habits. The agent tries something, sees whether the outcome was good or bad, and then makes the actions that led to good outcomes a little more likely next time.
It resembles learning to throw darts. You do not calculate the physics of each throw. You simply notice which arm movements landed closer to the center and repeat them more often. Over many attempts, small adjustments add up to skill.
How it is used in practice
A robotics research group trains a simulated quadruped robot to walk across uneven terrain. The action space consists of 12 continuous joint targets per control step, which rules out simple value-based methods.
The team uses proximal policy optimization with 4,096 parallel simulated robots, a clipping range of 0.2, GAE with λ set to 0.95 and a small entropy bonus. Rewards favor forward velocity and penalize energy use and falls. Every few seconds of wall-clock time, the policy network receives a fresh batch of trajectories and performs several epochs of clipped updates.
After roughly one billion simulated steps, which take a few hours on a single GPU, the policy produces a stable trotting gait. Randomizing friction and mass during training lets the same policy transfer to the physical robot with only minor tuning.