Offline reinforcement learning, also called batch reinforcement learning or offline RL, is the setting in which an agent learns a policy exclusively from a fixed dataset of previously logged transitions, with no further interaction with the environment during training. It applies the goals of reinforcement learning, maximizing cumulative reward in a Markov decision process, to situations where trial and error is too expensive, slow or dangerous to run live.
The central difficulty is distributional shift: the learned policy may prefer actions that rarely or never appear in the data, and the value estimates for those actions are unreliable. Modern offline RL methods are designed above all to keep the policy close to what the data can support while still improving on the behavior that generated it.
How offline reinforcement learning differs from online training
In standard online reinforcement learning, the agent alternates between acting and learning: it explores, observes rewards and corrects its estimates. Offline RL removes this loop. The input is a dataset D of tuples (state, action, reward, next state), collected by one or more behavior policies such as human operators, rule-based controllers or earlier models, and the output is a new policy evaluated only after training.
Because the agent cannot test its own hypotheses, it cannot correct an overestimated action by trying it. This turns offline RL into a problem that combines policy optimization with counterfactual reasoning, which is closer to causal inference than to classic trial-and-error learning. The table contrasts the three related regimes.
| Setting | Data source | New interaction during training | Main risk |
|---|---|---|---|
| Online on-policy | Current policy only | Continuous | Sample inefficiency |
| Online off-policy | Replay buffer refreshed by the agent | Continuous | Instability from stale data |
| Offline (batch) | Fixed logged dataset | None | Extrapolation error on unseen actions |
| Offline-to-online | Logged data, then live fine-tuning | Limited, after pretraining | Performance drop at the switch |
Why distributional shift breaks naive methods
An off-policy algorithm such as Q-learning or a deep Q-network can, in principle, learn from any data. In practice, running it on a fixed dataset often fails badly. The Bellman update takes a maximum over next actions, and the Q-function generalizes arbitrarily to actions absent from the data. If those estimates are too high, the maximization selects them, the error propagates through bootstrapping and the values diverge or become wildly optimistic.
This failure is known as extrapolation error or out-of-distribution action overestimation. Online agents eventually try the overrated action and learn it is poor; offline agents never get that feedback. The severity depends on dataset coverage: narrow data from a single expert policy is harder to improve on than broad data from many diverse behaviors.
Main families of offline RL algorithms
Offline RL algorithms address distributional shift with a form of conservatism, applied either to the policy, to the value function or to a learned model. The table summarizes the major approaches.
| Family | Core idea | Representative methods |
|---|---|---|
| Policy constraint | Keep the learned policy near the behavior policy | BCQ, BEAR, TD3+BC |
| Value regularization | Push down Q-values of unseen actions | Conservative Q-learning (CQL) |
| In-sample learning | Avoid querying actions outside the dataset | Implicit Q-learning (IQL), AWR |
| Uncertainty penalties | Subtract an ensemble disagreement term | EDAC, ensemble Q-functions |
| Model-based conservatism | Penalize rollouts in uncertain model regions | MOPO, MOReL, COMBO |
| Sequence modeling | Predict actions conditioned on desired return | Decision Transformer |
Policy constraint methods add a divergence penalty or a behavior-cloning term so the policy cannot drift far from logged actions. Conservative Q-learning instead learns a lower bound on the value, which makes unsupported actions look unattractive. Implicit Q-learning sidesteps the problem by estimating values through expectile regression only on actions that actually occur in the data. Behavior cloning itself remains a strong baseline, especially on high-quality expert data.
Data quality and coverage requirements
The ceiling of an offline RL system is set by its data. A policy can improve on the behavior policy mainly by stitching together good segments of different trajectories, which requires that the dataset contains those segments. The table lists the dataset properties that matter most.
| Property | Why it matters | Practical check |
|---|---|---|
| State-action coverage | Limits where values can be trusted | Inspect action frequencies per state cluster |
| Behavior policy quality | Sets the baseline to improve on | Compare logged returns with known benchmarks |
| Behavior diversity | Enables stitching of better trajectories | Count distinct policies or operators in logs |
| Reward correctness | Wrong rewards produce wrong objectives | Audit reward computation on sample episodes |
| Logged action propensities | Required for importance-weighted evaluation | Verify probabilities were recorded |
Datasets also inherit confounding: if human operators acted on information not recorded in the state, the logged actions correlate with outcomes for hidden reasons, and no algorithm can fully correct for that without additional assumptions.
Evaluating a policy without deploying it
Since deployment is exactly what offline RL tries to avoid, policies are judged through off-policy evaluation (OPE). Common estimators include importance sampling, which reweights logged returns by the ratio of target to behavior action probabilities; the direct method, which uses a learned value or dynamics model; and doubly robust estimators, which combine both to reduce bias and variance.
Each estimator has known weaknesses. Importance sampling suffers exploding variance over long horizons, and model-based estimates inherit model bias. Hyperparameter selection is therefore one of the hardest unsolved practical problems in offline RL, and responsible teams follow OPE with a cautious, monitored rollout such as a shadow mode or small A/B test.
Where offline RL is used and where it struggles
Offline RL is most valuable where large interaction logs already exist and online exploration is costly. Typical domains include recommender systems, ad and pricing decisions, industrial process control, robotics from demonstration data, healthcare treatment policies and dialogue or language-model fine-tuning from logged feedback.
It struggles when coverage is narrow, rewards are sparse or delayed, and the environment changes after data collection. A policy trained on last year’s logs may face nonstationary conditions that its data never showed. Safety-critical applications therefore pair offline RL with constraint handling from safe reinforcement learning and keep a human fallback in place.
The idea in brief
Think of learning to drive by studying thousands of hours of recorded dashboard video instead of sitting behind the wheel. Offline reinforcement learning works the same way: the computer learns only from past recordings and never gets to practice on the real road while it is learning.
The danger is that the learner may invent clever moves nobody in the recordings ever tried, with no way to check whether they work. Good offline methods therefore stay close to what the recordings actually show and improve on it carefully.
A practical scenario
A building-management company wants a smarter controller for heating and cooling in 40 office buildings. Experimenting live would risk uncomfortable staff and wasted energy, but the company already holds three years of five-minute logs of temperatures, occupancy, weather, setpoint decisions and energy use from its existing rule-based controllers and from operators’ manual overrides.
The engineering team defines a reward that combines energy cost with penalties for temperatures outside the comfort band, then trains an implicit Q-learning policy on the logs. Off-policy evaluation with a doubly robust estimator predicts savings of roughly 8 to 12 percent. The team deploys the policy first in shadow mode on four buildings, comparing its recommendations with actual decisions, and then lets it control two buildings with operator override. After one season, measured energy use falls by about 9 percent with no increase in comfort complaints, and the rollout expands gradually.