A world model is a learned internal model of an environment that predicts how the state of the world will change in response to an agent’s actions. In embodied AI and robotics, it lets a robot imagine the consequences of actions before executing them, so it can plan, evaluate options or train a policy largely inside its own predictions instead of through costly physical trial and error.
Formally, a world model approximates the transition dynamics of the environment: given the current state, or a compressed representation of recent observations, and a candidate action, it outputs the predicted next state, often together with predicted reward, task progress or uncertainty. The concept comes from model-based reinforcement learning and control, and it now spans compact latent dynamics models as well as large video-prediction models trained on extensive robot and video data.
How a world model works
Most modern world models share three components. Together they turn raw sensor streams into a compact state that can be rolled forward in time under hypothetical actions.
| Component | Function | Typical implementation |
|---|---|---|
| Encoder | Compresses images and proprioception into a latent state | Convolutional network or vision transformer |
| Dynamics model | Predicts the next latent state from state and action | Recurrent state-space model or transformer |
| Decoder or heads | Predicts observations, reward or task signals | Image decoder, reward and value heads |
Training uses logged interaction data: sequences of observations and actions. The model learns to minimize prediction error on future observations or on their latent representations. Once trained, the robot can feed a candidate action sequence into the dynamics model and obtain an imagined trajectory, called a rollout, without moving a single joint.
Types of world models
World models differ mainly in what they predict and at what level of abstraction. Predicting every pixel captures appearance but wastes capacity on irrelevant detail; predicting abstract features is efficient but harder to inspect.
| Family | Prediction target | Strength | Weakness |
|---|---|---|---|
| Analytical physics model | Explicit states via equations | Precise and interpretable | Hard to build for contact-rich, deformable scenes |
| Latent dynamics model | Compact learned state | Fast rollouts for planning and policy learning | Latent state is hard to interpret |
| Joint-embedding predictive model | Future representations, not pixels | Ignores unpredictable visual detail | Needs careful training to avoid collapse |
| Video world model | Future camera frames | Rich, general visual knowledge | Expensive and prone to physically implausible output |
| Object-centric model | States of individual objects | Structured reasoning about interactions | Depends on reliable object segmentation |
In practice, the families blend. A robot may use a latent model for fast planning and a video model to generate training scenarios or to evaluate policies offline.
How robots use world models
A world model is useful only through the decisions it supports. Robotics applications fall into a few patterns, each with different requirements for accuracy, speed and horizon.
- Planning by imagination: sample many action sequences, roll them out in the model and execute the best first action, as in model predictive control.
- Policy learning in imagination: train a policy on imagined rollouts, which can cut real-world interaction substantially.
- Policy evaluation: score a candidate policy on predicted outcomes before deploying it on hardware.
- Data generation: synthesize additional training scenarios for perception or control.
- Anomaly detection: flag situations where reality diverges strongly from the prediction.
The usable planning horizon is limited by compounding error: each predicted step feeds the next, so small inaccuracies grow. Typical robot planners therefore roll out only short horizons, from a fraction of a second to a few seconds, and replan frequently with fresh observations.
World models compared with related embodied AI concepts
World models are often discussed together with other embodied AI approaches, but they answer a different question: what will happen if the robot acts, rather than what the robot should do.
| Concept | Core question | Relationship to a world model |
|---|---|---|
| World model | What happens if this action is taken | Predicts consequences |
| Vision-language-action model | Which action fits this instruction and scene | Can be trained or evaluated with a world model |
| Imitation learning | How would a demonstrator act here | Learns a policy directly, without predicting outcomes |
| Physics simulator | What do equations predict | Hand-built model, not learned from data |
| Sim-to-real transfer | How to carry simulated skills to hardware | Learned world models can narrow the reality gap |
A useful distinction is between a policy, which maps states to actions, and a world model, which maps states and actions to future states. Systems that combine both can plan, while pure policy models react.
Limitations and open problems
World models remain an active research area, and their predictions should be treated as uncertain estimates rather than ground truth. Several limitations recur across applications.
- Compounding error: long rollouts drift away from physically possible states.
- Model exploitation: a planner may favor actions that look good only because the model is wrong in that region.
- Contact and deformation: friction, soft objects and liquids remain difficult to predict.
- Distribution shift: predictions degrade in scenes unlike the training data.
- Compute cost: large video models are too slow for closed-loop control on many robots.
Common mitigations include ensembles that estimate uncertainty, short-horizon replanning, conservative planners that avoid poorly modeled regions, and continued training on newly collected real data.
The short version
A world model is a robot’s ability to imagine what will happen next. Before you push a glass toward the edge of a table, you can picture it falling. You do not need to try it to know the result.
A robot with a world model does something similar. It has learned from experience how things move and react. It can test ideas in its head before acting, which saves time and avoids mistakes. Its imagination is still imperfect, so it checks reality often and corrects itself.
Where it makes a difference
A research team trains a robot arm to fold towels, a task in which cloth behaves in complex ways that a hand-built simulator models poorly. The team records several hours of the robot manipulating towels and trains a latent world model that predicts future camera features from current images and gripper actions.
At run time, the robot samples hundreds of short action sequences, rolls each out one to two seconds ahead in the model and executes the first action of the sequence predicted to bring the towel closest to a folded shape. It then observes the result and plans again. The arm learns to fold with far fewer physical trials than a policy trained by trial and error alone, although unusual fabrics still require additional data before predictions become reliable.