Off-policy learning

On this page 8

Off-policy learning is the family of reinforcement learning methods in which an agent learns about one policy, the target policy, from data generated by a different policy, the behavior policy. The agent can explore with a noisy or older strategy while estimating the value of, and improving, a separate greedy or optimal strategy.

This separation makes experience reusable. Transitions collected yesterday, by another agent or by a human operator, can still train today’s policy, which is why off-policy methods underpin experience replay, Q-learning, Deep Q-networks and most sample-efficient continuous-control algorithms.

How off-policy learning separates behavior and target policies

Off-policy learning works by decoupling the policy that acts from the policy that is evaluated. The behavior policy, often written as mu or b, chooses actions in the environment and produces trajectories. The target policy, written as pi, is the one whose value function or performance the algorithm estimates and optimizes.

The classic example is Q-learning. Its update target is r + gamma max Q(s’, a’), which uses the greedy action in the next state regardless of which action the behavior policy actually took. The agent may follow an epsilon-greedy policy for exploration, yet it learns the action values of the greedy policy. This is the defining contrast with SARSA, whose target uses the action actually taken.

The roles of the two policies are summarized below, because confusing them is the most common source of error when reading off-policy algorithms.

AspectBehavior policyTarget policy
PurposeCollects experienceIs evaluated and improved
Typical formEpsilon-greedy, noisy or loggedGreedy or stochastic optimum
Must exploreYes, broadly enoughNo
Appears in updateOnly through the data or weightsThrough the bootstrap target
Can be unknownSometimes, with special estimatorsNo, it must be specified
Roles of the two policies in off-policy learning

Correcting for distribution mismatch

Because the data come from the wrong distribution, off-policy methods must account for the mismatch between behavior and target action probabilities. Methods that learn action values with one-step targets, such as Q-learning, avoid this for the current action, because the action is given and only the next action is replaced by the target choice.

Multi-step and state-value methods need an explicit correction, usually importance sampling. The importance ratio rho = pi(a|s) / mu(a|s) reweights each step, and the product of ratios over a trajectory reweights the return. Products of ratios grow or shrink exponentially with horizon, so ordinary importance sampling is unbiased but can have extreme variance.

Several correction strategies trade bias against variance, and the table compares the ones most often encountered.

MethodBiasVarianceTypical use
Ordinary importance samplingUnbiasedVery high on long horizonsTheory and short episodes
Weighted importance samplingBiased, consistentMuch lowerOff-policy Monte Carlo evaluation
Per-decision importance samplingUnbiasedLower than ordinaryMulti-step returns
Retrace(lambda)LowBounded by truncated ratiosSafe multi-step deep RL
V-traceSome bias at clippingBoundedDistributed actor-learner systems
Doubly robust estimatorsLow if one model is correctReducedOff-policy policy evaluation
Common correction methods for off-policy estimates

Truncating or clipping ratios, as in Retrace and V-trace, keeps updates stable at the cost of a controlled bias toward the behavior policy. In practice, this trade is almost always worth making in deep reinforcement learning.

Off-policy learning sits between on-policy learning, which uses only fresh data from the current policy, and offline reinforcement learning, which uses a fixed dataset with no further interaction. The boundaries matter because each setting fails in a different way.

SettingData sourceEnvironment interactionRepresentative algorithmsMain risk
On-policyCurrent policy onlyContinuousSARSA, REINFORCE, PPOPoor sample efficiency
Off-policyAny policy, including past versionsContinuousQ-learning, DQN, DDPG, TD3, SACInstability and overestimation
OfflineFixed logged datasetNone during trainingCQL, IQL, BCQExtrapolation error on unseen actions
Off-policy, on-policy and offline settings compared

Some algorithms blur the lines. PPO reuses each batch for several epochs and clips the probability ratio, which makes it slightly off-policy in practice. Conversely, a pure off-policy algorithm such as SAC becomes an offline method when the replay buffer is frozen, and it then typically performs poorly without conservative regularization.

The deadly triad and stability

Off-policy learning is one of the three ingredients of the deadly triad, alongside function approximation and bootstrapping. When all three are present, value estimates can diverge even on simple problems, as classic counterexamples with linear features demonstrate.

Deep off-policy algorithms rely on engineering safeguards rather than convergence guarantees. Target networks slow the movement of bootstrap targets, experience replay decorrelates samples, and clipped double Q-learning in TD3 and SAC counters the overestimation bias that the max operator introduces.

  • Gradient temporal difference methods such as GTD2 and TDC provide convergence with linear features under off-policy sampling.
  • Emphatic TD reweights updates to restore stability, at the price of high variance.
  • Double DQN separates action selection from evaluation to reduce optimistic bias.

When to choose an off-policy method

Off-policy methods are the preferred choice when environment interaction is expensive and every sample must be reused many times. They are less attractive when simulation is cheap and stability matters more than data efficiency.

SituationRecommended approachReason
Real robot with limited trialsOff-policy, for example SACReplay buffer maximizes reuse
Cheap massively parallel simulatorOn-policy, for example PPOStability outweighs sample cost
Learning from human demonstrations plus interactionOff-policy with demonstration bufferDemonstrations act as behavior data
Evaluating a new recommender from logsOff-policy evaluation estimatorsNo live deployment needed
No further interaction allowedOffline reinforcement learningStandard off-policy methods extrapolate badly
Decision guide for off-policy versus on-policy methods

Common mistakes with off-policy learning

Most practical failures come from insufficient coverage by the behavior policy. If the behavior policy never takes an action that the target policy prefers, the ratio pi/mu is undefined or enormous, and no correction can recover reliable estimates for that action.

  • Using a very large, stale replay buffer so that old data no longer resemble the current policy.
  • Ignoring overestimation bias from the max operator in noisy environments.
  • Applying multi-step returns without any importance correction and assuming the bias is negligible.
  • Logging data without recording behavior probabilities, which later blocks unbiased evaluation.

The simple explanation

Off-policy learning means learning from someone else’s choices, or from your own past choices, to become better at a different way of acting. Think of a new chess player who studies records of games played by others. The player never made those moves, yet still learns which moves would have been best.

A learning agent does the same thing with its stored experience. It can try cautious or random actions to explore while quietly working out what the best possible strategy would be. The difficulty is that the stored games may not show enough examples of the moves the agent now wants to judge.

How it is used in practice

A robotics team trains a robotic arm to grasp irregular warehouse items, where each real grasp attempt takes about 15 seconds. Running an on-policy method would discard data after each update and would require weeks of robot time. The team instead uses soft actor-critic with a replay buffer of 1 million transitions, adding earlier scripted grasp attempts to the buffer as additional behavior data.

Each new transition is reused in many gradient updates, and the scripted data give the agent early examples of successful grasps. After roughly 40 hours of interaction, the learned policy reaches a grasp success rate above 85 percent on held-out items, compared with a substantially longer projected training time for an on-policy baseline. The team monitors Q-value estimates for overestimation and limits buffer age to keep the data relevant.