Offline reinforcement learning

Also known as: batch reinforcement learning · offline RL

On this page 8

Offline reinforcement learning, also called batch reinforcement learning or offline RL, is the setting in which an agent learns a policy exclusively from a fixed dataset of previously logged transitions, with no further interaction with the environment during training. It applies the goals of reinforcement learning, maximizing cumulative reward in a Markov decision process, to situations where trial and error is too expensive, slow or dangerous to run live.

The central difficulty is distributional shift: the learned policy may prefer actions that rarely or never appear in the data, and the value estimates for those actions are unreliable. Modern offline RL methods are designed above all to keep the policy close to what the data can support while still improving on the behavior that generated it.

How offline reinforcement learning differs from online training

In standard online reinforcement learning, the agent alternates between acting and learning: it explores, observes rewards and corrects its estimates. Offline RL removes this loop. The input is a dataset D of tuples (state, action, reward, next state), collected by one or more behavior policies such as human operators, rule-based controllers or earlier models, and the output is a new policy evaluated only after training.

Because the agent cannot test its own hypotheses, it cannot correct an overestimated action by trying it. This turns offline RL into a problem that combines policy optimization with counterfactual reasoning, which is closer to causal inference than to classic trial-and-error learning. The table contrasts the three related regimes.

SettingData sourceNew interaction during trainingMain risk
Online on-policyCurrent policy onlyContinuousSample inefficiency
Online off-policyReplay buffer refreshed by the agentContinuousInstability from stale data
Offline (batch)Fixed logged datasetNoneExtrapolation error on unseen actions
Offline-to-onlineLogged data, then live fine-tuningLimited, after pretrainingPerformance drop at the switch
Online, off-policy and offline settings compared

Why distributional shift breaks naive methods

An off-policy algorithm such as Q-learning or a deep Q-network can, in principle, learn from any data. In practice, running it on a fixed dataset often fails badly. The Bellman update takes a maximum over next actions, and the Q-function generalizes arbitrarily to actions absent from the data. If those estimates are too high, the maximization selects them, the error propagates through bootstrapping and the values diverge or become wildly optimistic.

This failure is known as extrapolation error or out-of-distribution action overestimation. Online agents eventually try the overrated action and learn it is poor; offline agents never get that feedback. The severity depends on dataset coverage: narrow data from a single expert policy is harder to improve on than broad data from many diverse behaviors.

Main families of offline RL algorithms

Offline RL algorithms address distributional shift with a form of conservatism, applied either to the policy, to the value function or to a learned model. The table summarizes the major approaches.

FamilyCore ideaRepresentative methods
Policy constraintKeep the learned policy near the behavior policyBCQ, BEAR, TD3+BC
Value regularizationPush down Q-values of unseen actionsConservative Q-learning (CQL)
In-sample learningAvoid querying actions outside the datasetImplicit Q-learning (IQL), AWR
Uncertainty penaltiesSubtract an ensemble disagreement termEDAC, ensemble Q-functions
Model-based conservatismPenalize rollouts in uncertain model regionsMOPO, MOReL, COMBO
Sequence modelingPredict actions conditioned on desired returnDecision Transformer
Principal algorithm families

Policy constraint methods add a divergence penalty or a behavior-cloning term so the policy cannot drift far from logged actions. Conservative Q-learning instead learns a lower bound on the value, which makes unsupported actions look unattractive. Implicit Q-learning sidesteps the problem by estimating values through expectile regression only on actions that actually occur in the data. Behavior cloning itself remains a strong baseline, especially on high-quality expert data.

Data quality and coverage requirements

The ceiling of an offline RL system is set by its data. A policy can improve on the behavior policy mainly by stitching together good segments of different trajectories, which requires that the dataset contains those segments. The table lists the dataset properties that matter most.

PropertyWhy it mattersPractical check
State-action coverageLimits where values can be trustedInspect action frequencies per state cluster
Behavior policy qualitySets the baseline to improve onCompare logged returns with known benchmarks
Behavior diversityEnables stitching of better trajectoriesCount distinct policies or operators in logs
Reward correctnessWrong rewards produce wrong objectivesAudit reward computation on sample episodes
Logged action propensitiesRequired for importance-weighted evaluationVerify probabilities were recorded
Dataset properties and their influence

Datasets also inherit confounding: if human operators acted on information not recorded in the state, the logged actions correlate with outcomes for hidden reasons, and no algorithm can fully correct for that without additional assumptions.

Evaluating a policy without deploying it

Since deployment is exactly what offline RL tries to avoid, policies are judged through off-policy evaluation (OPE). Common estimators include importance sampling, which reweights logged returns by the ratio of target to behavior action probabilities; the direct method, which uses a learned value or dynamics model; and doubly robust estimators, which combine both to reduce bias and variance.

Each estimator has known weaknesses. Importance sampling suffers exploding variance over long horizons, and model-based estimates inherit model bias. Hyperparameter selection is therefore one of the hardest unsolved practical problems in offline RL, and responsible teams follow OPE with a cautious, monitored rollout such as a shadow mode or small A/B test.

Where offline RL is used and where it struggles

Offline RL is most valuable where large interaction logs already exist and online exploration is costly. Typical domains include recommender systems, ad and pricing decisions, industrial process control, robotics from demonstration data, healthcare treatment policies and dialogue or language-model fine-tuning from logged feedback.

It struggles when coverage is narrow, rewards are sparse or delayed, and the environment changes after data collection. A policy trained on last year’s logs may face nonstationary conditions that its data never showed. Safety-critical applications therefore pair offline RL with constraint handling from safe reinforcement learning and keep a human fallback in place.

The idea in brief

Think of learning to drive by studying thousands of hours of recorded dashboard video instead of sitting behind the wheel. Offline reinforcement learning works the same way: the computer learns only from past recordings and never gets to practice on the real road while it is learning.

The danger is that the learner may invent clever moves nobody in the recordings ever tried, with no way to check whether they work. Good offline methods therefore stay close to what the recordings actually show and improve on it carefully.

A practical scenario

A building-management company wants a smarter controller for heating and cooling in 40 office buildings. Experimenting live would risk uncomfortable staff and wasted energy, but the company already holds three years of five-minute logs of temperatures, occupancy, weather, setpoint decisions and energy use from its existing rule-based controllers and from operators’ manual overrides.

The engineering team defines a reward that combines energy cost with penalties for temperatures outside the comfort band, then trains an implicit Q-learning policy on the logs. Off-policy evaluation with a doubly robust estimator predicts savings of roughly 8 to 12 percent. The team deploys the policy first in shadow mode on four buildings, comparing its recommendations with actual decisions, and then lets it control two buildings with operator override. After one season, measured energy use falls by about 9 percent with no increase in comfort complaints, and the rollout expands gradually.