Off-policy learning is the family of reinforcement learning methods in which an agent learns about one policy, the target policy, from data generated by a different policy, the behavior policy. The agent can explore with a noisy or older strategy while estimating the value of, and improving, a separate greedy or optimal strategy.
This separation makes experience reusable. Transitions collected yesterday, by another agent or by a human operator, can still train today’s policy, which is why off-policy methods underpin experience replay, Q-learning, Deep Q-networks and most sample-efficient continuous-control algorithms.
How off-policy learning separates behavior and target policies
Off-policy learning works by decoupling the policy that acts from the policy that is evaluated. The behavior policy, often written as mu or b, chooses actions in the environment and produces trajectories. The target policy, written as pi, is the one whose value function or performance the algorithm estimates and optimizes.
The classic example is Q-learning. Its update target is r + gamma max Q(s’, a’), which uses the greedy action in the next state regardless of which action the behavior policy actually took. The agent may follow an epsilon-greedy policy for exploration, yet it learns the action values of the greedy policy. This is the defining contrast with SARSA, whose target uses the action actually taken.
The roles of the two policies are summarized below, because confusing them is the most common source of error when reading off-policy algorithms.
| Aspect | Behavior policy | Target policy |
|---|---|---|
| Purpose | Collects experience | Is evaluated and improved |
| Typical form | Epsilon-greedy, noisy or logged | Greedy or stochastic optimum |
| Must explore | Yes, broadly enough | No |
| Appears in update | Only through the data or weights | Through the bootstrap target |
| Can be unknown | Sometimes, with special estimators | No, it must be specified |
Correcting for distribution mismatch
Because the data come from the wrong distribution, off-policy methods must account for the mismatch between behavior and target action probabilities. Methods that learn action values with one-step targets, such as Q-learning, avoid this for the current action, because the action is given and only the next action is replaced by the target choice.
Multi-step and state-value methods need an explicit correction, usually importance sampling. The importance ratio rho = pi(a|s) / mu(a|s) reweights each step, and the product of ratios over a trajectory reweights the return. Products of ratios grow or shrink exponentially with horizon, so ordinary importance sampling is unbiased but can have extreme variance.
Several correction strategies trade bias against variance, and the table compares the ones most often encountered.
| Method | Bias | Variance | Typical use |
|---|---|---|---|
| Ordinary importance sampling | Unbiased | Very high on long horizons | Theory and short episodes |
| Weighted importance sampling | Biased, consistent | Much lower | Off-policy Monte Carlo evaluation |
| Per-decision importance sampling | Unbiased | Lower than ordinary | Multi-step returns |
| Retrace(lambda) | Low | Bounded by truncated ratios | Safe multi-step deep RL |
| V-trace | Some bias at clipping | Bounded | Distributed actor-learner systems |
| Doubly robust estimators | Low if one model is correct | Reduced | Off-policy policy evaluation |
Truncating or clipping ratios, as in Retrace and V-trace, keeps updates stable at the cost of a controlled bias toward the behavior policy. In practice, this trade is almost always worth making in deep reinforcement learning.
Off-policy learning compared with related approaches
Off-policy learning sits between on-policy learning, which uses only fresh data from the current policy, and offline reinforcement learning, which uses a fixed dataset with no further interaction. The boundaries matter because each setting fails in a different way.
| Setting | Data source | Environment interaction | Representative algorithms | Main risk |
|---|---|---|---|---|
| On-policy | Current policy only | Continuous | SARSA, REINFORCE, PPO | Poor sample efficiency |
| Off-policy | Any policy, including past versions | Continuous | Q-learning, DQN, DDPG, TD3, SAC | Instability and overestimation |
| Offline | Fixed logged dataset | None during training | CQL, IQL, BCQ | Extrapolation error on unseen actions |
Some algorithms blur the lines. PPO reuses each batch for several epochs and clips the probability ratio, which makes it slightly off-policy in practice. Conversely, a pure off-policy algorithm such as SAC becomes an offline method when the replay buffer is frozen, and it then typically performs poorly without conservative regularization.
The deadly triad and stability
Off-policy learning is one of the three ingredients of the deadly triad, alongside function approximation and bootstrapping. When all three are present, value estimates can diverge even on simple problems, as classic counterexamples with linear features demonstrate.
Deep off-policy algorithms rely on engineering safeguards rather than convergence guarantees. Target networks slow the movement of bootstrap targets, experience replay decorrelates samples, and clipped double Q-learning in TD3 and SAC counters the overestimation bias that the max operator introduces.
- Gradient temporal difference methods such as GTD2 and TDC provide convergence with linear features under off-policy sampling.
- Emphatic TD reweights updates to restore stability, at the price of high variance.
- Double DQN separates action selection from evaluation to reduce optimistic bias.
When to choose an off-policy method
Off-policy methods are the preferred choice when environment interaction is expensive and every sample must be reused many times. They are less attractive when simulation is cheap and stability matters more than data efficiency.
| Situation | Recommended approach | Reason |
|---|---|---|
| Real robot with limited trials | Off-policy, for example SAC | Replay buffer maximizes reuse |
| Cheap massively parallel simulator | On-policy, for example PPO | Stability outweighs sample cost |
| Learning from human demonstrations plus interaction | Off-policy with demonstration buffer | Demonstrations act as behavior data |
| Evaluating a new recommender from logs | Off-policy evaluation estimators | No live deployment needed |
| No further interaction allowed | Offline reinforcement learning | Standard off-policy methods extrapolate badly |
Common mistakes with off-policy learning
Most practical failures come from insufficient coverage by the behavior policy. If the behavior policy never takes an action that the target policy prefers, the ratio pi/mu is undefined or enormous, and no correction can recover reliable estimates for that action.
- Using a very large, stale replay buffer so that old data no longer resemble the current policy.
- Ignoring overestimation bias from the max operator in noisy environments.
- Applying multi-step returns without any importance correction and assuming the bias is negligible.
- Logging data without recording behavior probabilities, which later blocks unbiased evaluation.
The simple explanation
Off-policy learning means learning from someone else’s choices, or from your own past choices, to become better at a different way of acting. Think of a new chess player who studies records of games played by others. The player never made those moves, yet still learns which moves would have been best.
A learning agent does the same thing with its stored experience. It can try cautious or random actions to explore while quietly working out what the best possible strategy would be. The difficulty is that the stored games may not show enough examples of the moves the agent now wants to judge.
How it is used in practice
A robotics team trains a robotic arm to grasp irregular warehouse items, where each real grasp attempt takes about 15 seconds. Running an on-policy method would discard data after each update and would require weeks of robot time. The team instead uses soft actor-critic with a replay buffer of 1 million transitions, adding earlier scripted grasp attempts to the buffer as additional behavior data.
Each new transition is reused in many gradient updates, and the scripted data give the agent early examples of successful grasps. After roughly 40 hours of interaction, the learned policy reaches a grasp success rate above 85 percent on held-out items, compared with a substantially longer projected training time for an on-policy baseline. The team monitors Q-value estimates for overestimation and limits buffer age to keep the data relevant.