Policy gradient

Also known as: policy gradient method

On this page 7

A policy gradient method is a family of reinforcement learning algorithms that optimize a parameterized policy directly by following the gradient of the expected cumulative reward with respect to the policy parameters. Instead of learning a value for every action and then acting greedily, the agent adjusts the probabilities of its actions so that behavior leading to higher return becomes more likely.

Policy gradient methods are the foundation of modern algorithms such as REINFORCE, actor-critic, trust region policy optimization and proximal policy optimization, which is widely used for robotics control and for fine-tuning large language models with human feedback.

How the policy gradient theorem works

The method starts from a stochastic policy π(a | s; θ), typically a neural network that outputs a probability distribution over actions. The objective J(θ) is the expected return of trajectories generated by that policy, and learning performs gradient ascent on J(θ).

The difficulty is that the return depends on the environment dynamics, which are usually unknown. The policy gradient theorem solves this by showing that the gradient equals the expectation of ∇ log π(a | s; θ) multiplied by a return signal. This form, known as the likelihood ratio or score function estimator, needs only sampled actions and rewards, not a model of the environment.

Intuitively, the log-probability gradient points toward making the chosen action more likely, and the return signal scales that push up or down. Good actions are reinforced and poor actions are suppressed in proportion to their outcomes.

The table lists the main ingredients of the update.

ElementRoleTypical form
Policy networkMaps states to action probabilitiesSoftmax or Gaussian output head
Score functionDirection that increases action likelihood∇ log π(a | s; θ)
Return signalWeights each update by outcomeReturn, advantage or TD error
Learning rateControls step sizeRoughly 1e-5 to 1e-3 with Adam
Batch of trajectoriesProvides the Monte Carlo estimateThousands of environment steps
Ingredients of a policy gradient update

Reducing variance in gradient estimates

The raw estimator is unbiased but extremely noisy, because a single trajectory return mixes the effect of many actions and random events. High variance is the main practical weakness of policy gradient learning, and most algorithmic progress has targeted it.

The most important technique is subtracting a baseline, usually a learned state-value estimate, which leaves the gradient unbiased while removing much of the noise. The resulting quantity is the advantage, which measures how much better an action was than the policy average in that state.

Return signalBiasVarianceUsed in
Full episode returnNoneVery highBasic REINFORCE
Reward-to-goNoneHighImproved REINFORCE
Return minus baselineNoneModerateREINFORCE with baseline
TD errorSomeLowOne-step actor-critic
GAE advantageTunable via λTunablePPO, TRPO
Return signals and their bias-variance profile

Generalized advantage estimation blends multi-step temporal difference errors with a parameter λ, usually between 0.9 and 0.99, giving practitioners a direct dial between bias and variance.

Main policy gradient algorithms

Policy gradient algorithms differ in how they estimate the return signal and how they limit the size of each update. Large steps can collapse a policy that took hours to train, so modern methods constrain how far the new policy may move from the old one.

AlgorithmKey ideaData useStrength
REINFORCEMonte Carlo score function estimateOn-policySimple and unbiased
Actor-criticCritic supplies a learned baselineOn-policyLower variance, online updates
TRPOKL divergence trust regionOn-policyStable monotonic improvement
PPOClipped probability ratioOn-policy, several epochsRobust and easy to tune
DDPG and TD3Deterministic policy gradientOff-policy with replaySample-efficient continuous control
Representative policy gradient algorithms

Proximal policy optimization has become the default choice because its clipping range of about 0.1 to 0.3 offers most of the stability of trust region methods with a simple first-order optimizer.

Policy gradient versus value-based methods

Policy gradient methods and value-based methods such as Q-learning solve the same problem from opposite directions. Value-based methods learn action values and derive a policy implicitly; policy gradient methods learn the policy itself and may use values only as a helper.

AspectPolicy gradientValue-based (Q-learning, DQN)
Action spacesDiscrete and continuousMainly discrete
Stochastic policiesLearned nativelyRequire added exploration rules
Sample efficiencyLower, mostly on-policyHigher, uses replay buffers
ConvergenceTo a local optimum, smoothlyCan diverge with function approximation
Gradient noiseHigh without baselinesLower, bootstrapped targets
Comparison with value-based reinforcement learning

The decisive advantage of policy gradients is natural handling of continuous actions, such as joint torques in a robot arm, where maximizing over a Q-function at every step would be impractical.

Practical pitfalls and failure modes

Policy gradient training is notoriously sensitive to implementation details. Experienced practitioners watch for the following problems:

  • Premature entropy collapse: the policy becomes deterministic too early; an entropy bonus keeps exploration alive.
  • Unnormalized advantages and rewards: wildly scaled signals destabilize learning; normalize them per batch.
  • Oversized updates: a single large step can destroy performance; use clipping or a KL penalty.
  • Stale data reuse: on-policy methods lose correctness when old samples are reused for too many epochs.
  • Reward hacking: the policy exploits flaws in the reward function instead of solving the task.
  • Seed sensitivity: results vary strongly between random seeds, so report several runs.

Because policy gradient methods converge only to local optima, initialization, network architecture and reward design often matter as much as the algorithm choice.

The simple explanation

A policy gradient method teaches an agent by nudging its habits. The agent tries something, sees whether the outcome was good or bad, and then makes the actions that led to good outcomes a little more likely next time.

It resembles learning to throw darts. You do not calculate the physics of each throw. You simply notice which arm movements landed closer to the center and repeat them more often. Over many attempts, small adjustments add up to skill.

How it is used in practice

A robotics research group trains a simulated quadruped robot to walk across uneven terrain. The action space consists of 12 continuous joint targets per control step, which rules out simple value-based methods.

The team uses proximal policy optimization with 4,096 parallel simulated robots, a clipping range of 0.2, GAE with λ set to 0.95 and a small entropy bonus. Rewards favor forward velocity and penalize energy use and falls. Every few seconds of wall-clock time, the policy network receives a fresh batch of trajectories and performs several epochs of clipped updates.

After roughly one billion simulated steps, which take a few hours on a single GPU, the policy produces a stable trotting gait. Randomizing friction and mass during training lets the same policy transfer to the physical robot with only minor tuning.