Model-based reinforcement learning

Also known as: model-based RL

On this page 7

Model-based reinforcement learning (model-based RL) is the family of reinforcement learning methods in which an agent learns or is given a model of the environment and uses that model to plan, simulate experience or improve its policy. The model predicts how the state changes after an action and, usually, which reward follows. In artificial intelligence, this approach is valued mainly for its sample efficiency: an agent that can imagine outcomes needs far fewer costly real interactions.

The central trade-off is between data efficiency and model bias. A good model lets the agent learn from cheap simulated rollouts, while an inaccurate model leads the agent to exploit errors that do not exist in the real environment.

How model-based reinforcement learning works

Model-based RL alternates between collecting real experience, fitting a dynamics model and using the model for decision making. Within a Markov decision process, the model approximates the transition function P(s’ | s, a) and the reward function R(s, a), either as a deterministic predictor or as a probability distribution.

  1. Interact with the real environment using the current policy and store transitions.
  2. Train the dynamics model on stored transitions, usually by supervised learning.
  3. Plan or generate synthetic data with the model, for example by short rollouts or tree search.
  4. Update the policy or value function from planned or imagined experience.
  5. Repeat, so that the model improves in the regions the policy actually visits.

The loop resembles classic Dyna architecture, in which real and simulated transitions both feed the same value function. Modern systems replace tabular models with neural network ensembles or learned latent dynamics that operate on compressed representations of images or sensor data.

Main families of model-based methods

Model-based methods differ in how the model is used, not only in how it is built. The table below groups the main approaches by the role the model plays.

ApproachRole of the modelRepresentative methods
Background planningGenerates synthetic transitions for a model-free learnerDyna-Q, MBPO
Decision-time planningSearches action sequences before each stepModel predictive control, PETS
Tree search with a learned modelExpands a search tree in latent spaceMuZero
Learning in imaginationTrains actor-critic entirely on imagined trajectoriesDreamer family
Analytic gradientsBackpropagates returns through a differentiable modelPILCO, SVG
Known-model planningUses exact rules instead of a learned modelAlphaZero, value iteration
Ways of using an environment model

Methods with a known model, such as board games with fixed rules, avoid model error entirely. Most practical applications, however, require a learned model, and that is where most research effort and most failure modes lie.

Model-based versus model-free reinforcement learning

The distinction with model-free reinforcement learning concerns whether the agent builds an explicit predictive model. Model-free methods such as Q-learning or proximal policy optimization learn values or policies directly from experience and never predict the next state.

CriterionModel-based RLModel-free RL
Sample efficiencyHigh, often 10 to 100 times fewer interactionsLow, often millions of steps
Asymptotic performanceCan be limited by model errorOften higher with ample data
Compute per real stepHigh, due to model training and planningLow to moderate
Adaptation to new goalsFast, the model can be reusedSlow, usually requires retraining
Implementation complexityHigher, more components to tuneLower, fewer moving parts
Key differences between the two paradigms

The efficiency gap varies widely by task and is not a fixed ratio; reported gains depend on environment complexity, model architecture and how much synthetic data is used. Many current systems are hybrids that use a model for planning or data augmentation and a model-free learner for the final policy.

Model error and how it is controlled

The defining risk of model-based RL is compounding model error: small prediction errors accumulate over a multi-step rollout, and the policy learns to exploit inaccuracies that promise unrealistic rewards. Controlling this effect is the main engineering task.

ProblemEffect on learningCommon mitigation
Compounding errorLong rollouts drift from realityShort rollout horizons, often 1 to 5 steps
Model exploitationPolicy targets fictional high rewardsEnsembles and uncertainty penalties
Distribution shiftModel is wrong in unvisited statesFrequent retraining on fresh data
Stochastic dynamicsDeterministic model averages outcomesProbabilistic or latent-variable models
Partial observabilityNext state not predictable from one observationRecurrent latent state models
Problems and standard mitigations

Probabilistic ensembles separate aleatoric uncertainty, which is inherent randomness, from epistemic uncertainty, which reflects lack of data. Planning conservatively where epistemic uncertainty is high is also a key idea in offline reinforcement learning and safe reinforcement learning.

When to choose a model-based approach

Model-based RL is most attractive when real interaction is expensive, slow or risky, and when the dynamics are smooth enough to learn. It is less attractive when a fast simulator already provides unlimited data.

  • Robotics and physical control, where each trial costs time and causes wear.
  • Industrial process control, where model predictive control is already established practice.
  • Games and puzzles with known rules, where exact planning is possible.
  • Multi-task settings, where one dynamics model serves several reward functions.
  • Avoid it when dynamics are highly chaotic or when cheap simulation removes the need for sample efficiency.

Explained without jargon

Think of two ways to learn a new route through a city. One person simply drives around until they remember which turns work. Another person first studies how the streets connect and then thinks through routes in their head before driving. The second person usually needs far fewer trips.

Model-based reinforcement learning is the second approach. The computer builds its own mental map of how the world reacts to its actions and practices inside that map. The catch is that a wrong map can send it confidently in the wrong direction, so it keeps checking the map against reality.

A typical use case

A robotics team wants a robotic arm to learn to insert pegs into holes with tight tolerances. Each real attempt takes about 20 seconds and occasionally damages parts, so millions of trials are not an option.

The team trains an ensemble of five neural dynamics models that predict the next joint positions and contact forces from the current state and motor command. Every few minutes of real practice, the models generate thousands of short imagined rollouts of three steps each, and a soft actor-critic learner updates the policy on both real and imagined data. Rollouts where the ensemble members disagree strongly are discarded.

After roughly two hours of real interaction, the arm reaches a reliable insertion success rate that a purely model-free setup in earlier tests had not reached after several days. The learned model is later reused to teach a second task, placing pegs at new angles, with only a short additional training period.