Model-based reinforcement learning (model-based RL) is the family of reinforcement learning methods in which an agent learns or is given a model of the environment and uses that model to plan, simulate experience or improve its policy. The model predicts how the state changes after an action and, usually, which reward follows. In artificial intelligence, this approach is valued mainly for its sample efficiency: an agent that can imagine outcomes needs far fewer costly real interactions.
The central trade-off is between data efficiency and model bias. A good model lets the agent learn from cheap simulated rollouts, while an inaccurate model leads the agent to exploit errors that do not exist in the real environment.
How model-based reinforcement learning works
Model-based RL alternates between collecting real experience, fitting a dynamics model and using the model for decision making. Within a Markov decision process, the model approximates the transition function P(s’ | s, a) and the reward function R(s, a), either as a deterministic predictor or as a probability distribution.
- Interact with the real environment using the current policy and store transitions.
- Train the dynamics model on stored transitions, usually by supervised learning.
- Plan or generate synthetic data with the model, for example by short rollouts or tree search.
- Update the policy or value function from planned or imagined experience.
- Repeat, so that the model improves in the regions the policy actually visits.
The loop resembles classic Dyna architecture, in which real and simulated transitions both feed the same value function. Modern systems replace tabular models with neural network ensembles or learned latent dynamics that operate on compressed representations of images or sensor data.
Main families of model-based methods
Model-based methods differ in how the model is used, not only in how it is built. The table below groups the main approaches by the role the model plays.
| Approach | Role of the model | Representative methods |
|---|---|---|
| Background planning | Generates synthetic transitions for a model-free learner | Dyna-Q, MBPO |
| Decision-time planning | Searches action sequences before each step | Model predictive control, PETS |
| Tree search with a learned model | Expands a search tree in latent space | MuZero |
| Learning in imagination | Trains actor-critic entirely on imagined trajectories | Dreamer family |
| Analytic gradients | Backpropagates returns through a differentiable model | PILCO, SVG |
| Known-model planning | Uses exact rules instead of a learned model | AlphaZero, value iteration |
Methods with a known model, such as board games with fixed rules, avoid model error entirely. Most practical applications, however, require a learned model, and that is where most research effort and most failure modes lie.
Model-based versus model-free reinforcement learning
The distinction with model-free reinforcement learning concerns whether the agent builds an explicit predictive model. Model-free methods such as Q-learning or proximal policy optimization learn values or policies directly from experience and never predict the next state.
| Criterion | Model-based RL | Model-free RL |
|---|---|---|
| Sample efficiency | High, often 10 to 100 times fewer interactions | Low, often millions of steps |
| Asymptotic performance | Can be limited by model error | Often higher with ample data |
| Compute per real step | High, due to model training and planning | Low to moderate |
| Adaptation to new goals | Fast, the model can be reused | Slow, usually requires retraining |
| Implementation complexity | Higher, more components to tune | Lower, fewer moving parts |
The efficiency gap varies widely by task and is not a fixed ratio; reported gains depend on environment complexity, model architecture and how much synthetic data is used. Many current systems are hybrids that use a model for planning or data augmentation and a model-free learner for the final policy.
Model error and how it is controlled
The defining risk of model-based RL is compounding model error: small prediction errors accumulate over a multi-step rollout, and the policy learns to exploit inaccuracies that promise unrealistic rewards. Controlling this effect is the main engineering task.
| Problem | Effect on learning | Common mitigation |
|---|---|---|
| Compounding error | Long rollouts drift from reality | Short rollout horizons, often 1 to 5 steps |
| Model exploitation | Policy targets fictional high rewards | Ensembles and uncertainty penalties |
| Distribution shift | Model is wrong in unvisited states | Frequent retraining on fresh data |
| Stochastic dynamics | Deterministic model averages outcomes | Probabilistic or latent-variable models |
| Partial observability | Next state not predictable from one observation | Recurrent latent state models |
Probabilistic ensembles separate aleatoric uncertainty, which is inherent randomness, from epistemic uncertainty, which reflects lack of data. Planning conservatively where epistemic uncertainty is high is also a key idea in offline reinforcement learning and safe reinforcement learning.
When to choose a model-based approach
Model-based RL is most attractive when real interaction is expensive, slow or risky, and when the dynamics are smooth enough to learn. It is less attractive when a fast simulator already provides unlimited data.
- Robotics and physical control, where each trial costs time and causes wear.
- Industrial process control, where model predictive control is already established practice.
- Games and puzzles with known rules, where exact planning is possible.
- Multi-task settings, where one dynamics model serves several reward functions.
- Avoid it when dynamics are highly chaotic or when cheap simulation removes the need for sample efficiency.
Explained without jargon
Think of two ways to learn a new route through a city. One person simply drives around until they remember which turns work. Another person first studies how the streets connect and then thinks through routes in their head before driving. The second person usually needs far fewer trips.
Model-based reinforcement learning is the second approach. The computer builds its own mental map of how the world reacts to its actions and practices inside that map. The catch is that a wrong map can send it confidently in the wrong direction, so it keeps checking the map against reality.
A typical use case
A robotics team wants a robotic arm to learn to insert pegs into holes with tight tolerances. Each real attempt takes about 20 seconds and occasionally damages parts, so millions of trials are not an option.
The team trains an ensemble of five neural dynamics models that predict the next joint positions and contact forces from the current state and motor command. Every few minutes of real practice, the models generate thousands of short imagined rollouts of three steps each, and a soft actor-critic learner updates the policy on both real and imagined data. Rollouts where the ensemble members disagree strongly are discarded.
After roughly two hours of real interaction, the arm reaches a reliable insertion success rate that a purely model-free setup in earlier tests had not reached after several days. The learned model is later reused to teach a second task, placing pegs at new angles, with only a short additional training period.