Sim-to-real is the practice of developing or training robot behavior in simulation and transferring it to a physical robot so that it still works under real-world physics, sensing and timing. In embodied AI it is the main route to learning skills that would be too slow, expensive or dangerous to learn on hardware, such as legged locomotion, dexterous in-hand manipulation and agile flight.
The central obstacle is the reality gap, the set of differences between the simulated and the real system. A policy that exploits a simulator’s imperfections can score perfectly in simulation and fail within seconds on a robot, so sim-to-real work is largely the discipline of making learned behavior robust to everything the simulator gets wrong.
Where the reality gap comes from
The reality gap has several independent sources, and each calls for a different remedy. Experienced teams diagnose which source dominates before choosing a transfer technique, because no single method closes every kind of gap.
| Source | Examples | Typical symptom on hardware |
|---|---|---|
| Dynamics | Mass, friction, damping, motor torque curves, backlash | Oscillation, slipping, falls |
| Contact modeling | Rigid contact approximations, soft materials | Unstable grasps, objects behaving differently |
| Perception | Rendering, lighting, sensor noise, depth artifacts | Wrong detections, poor state estimates |
| Actuation and latency | Control delay, communication lag, saturation | Jerky or delayed responses |
| Unmodeled environment | Cables, uneven floors, wear, temperature | Rare but severe failures |
Contact is often the hardest part. Physics engines approximate collisions with simplified friction cones and penetration handling, which is adequate for walking on flat ground but much less faithful for sliding, rolling or deformable objects in manipulation.
Techniques for closing the gap
Sim-to-real methods fall into two broad strategies: make the simulator more like reality, or make the policy indifferent to the differences. Most successful projects combine both.
| Technique | Core idea | Main trade-off |
|---|---|---|
| System identification | Measure real parameters and set them in the simulator | Accurate for known effects, blind to unmodeled ones |
| Domain randomization | Train across wide random variations of the simulation | Robust policies, but possibly conservative behavior |
| Actuator networks | Learn a model of real motor behavior from data | Improves dynamics fidelity, needs hardware logs |
| Domain adaptation | Align simulated and real sensor distributions | Helps perception, adds training complexity |
| Real-world fine-tuning | Continue training briefly on the robot | Closes residual gap, costs hardware time |
| Teacher-student distillation | Train a privileged teacher, distill to sensor-based student | Simplifies learning, needs careful observation design |
Teacher-student training deserves special mention. A teacher policy is trained with privileged information available only in simulation, such as exact terrain heights or friction coefficients. A student policy then learns to imitate the teacher using only real sensor inputs, often inferring the hidden parameters from its recent history of observations.
Sim-to-real compared with other embodied AI approaches
Sim-to-real is one of several ways to give a robot skills, and it is increasingly combined with data-driven methods rather than used alone.
| Approach | Main data source | Strength | Limitation |
|---|---|---|---|
| Sim-to-real reinforcement learning | Massive simulated experience | Learns agile control beyond human demonstration | Reality gap, reward design effort |
| Imitation learning | Human teleoperation demonstrations | Natural for complex manipulation tasks | Costly data collection, limited coverage |
| Vision-language-action model | Large mixed robot and web datasets | Generalizes across tasks and instructions | Heavy compute, imprecise low-level control |
| World model | Learned predictive model of the environment | Planning and imagination without a hand-built simulator | Model errors compound over long horizons |
In practice, simulation also supplies synthetic demonstrations for imitation learning and evaluation scenarios for foundation models, so the boundary between these approaches is becoming fluid.
Practical workflow and design choices
A robust sim-to-real workflow starts with modeling the robot carefully: accurate kinematics, inertias, joint limits, motor models and sensor rates. Engineers then define observation and action spaces that exist identically on hardware, add realistic latency and noise, and train with randomized parameters on thousands of parallel simulated environments, often on GPUs.
Policies typically output joint position targets for low-level PD controllers rather than raw torques, because the PD loop on the robot absorbs part of the modeling error. Before deployment, teams test policies in a second, independent simulator or with held-out randomization ranges to estimate robustness, then validate on hardware with safety limits, tethers or reduced speed.
Measuring transfer quality requires the same metrics in both worlds. Teams compare success rate, tracking error, energy use and failure modes between simulation and hardware; the difference between the two numbers is the practical measure of the reality gap. A policy that loses little performance on hardware is considered to transfer well, even if its absolute simulated score is lower than that of a more aggressive policy.
Sim-to-real has proven most reliable for locomotion and flight, where rigid-body dynamics dominate and simulators are mature. It remains harder for contact-rich manipulation of deformable, transparent or articulated objects, where many teams combine simulation pretraining with real demonstrations.
Limitations and common mistakes
Sim-to-real successes are impressive but often narrower than they look. The following mistakes recur across projects.
- Using simulator-only observations such as exact object poses, which do not exist on the real robot.
- Randomizing too little, producing brittle policies, or randomizing too much, producing timid ones.
- Ignoring control latency, which can turn a stable simulated gait into an oscillating one.
- Evaluating only in the training simulator, which hides overfitting to its specific physics engine.
- Underestimating the effort of safety validation before a learned policy runs near people.
- Neglecting hardware wear and drift, since motors heat up, joints loosen and batteries sag over the life of a real robot, while the simulated robot never ages.
The short version
Sim-to-real is like learning to fly in a flight simulator before flying a real plane. A robot practices millions of times in a virtual world, where mistakes cost nothing, and then uses what it learned on a real machine.
The catch is that no simulator is perfectly realistic. Engineers therefore make the practice world deliberately varied and imperfect, so the robot learns skills that survive the surprises of real floors, real objects and real motors.
Where it makes a difference
A research team wants a quadruped robot to walk over gravel, stairs and grass. Training on hardware would take months and damage the robot, so they train a locomotion policy in a GPU-based simulator with thousands of parallel robots, randomized terrain, friction, payload mass and motor strength, and an actuator model learned from real motor logs.
A teacher policy sees the terrain directly; a student policy learns to walk using only joint encoders and an IMU. On the first hardware test, the robot walks stably but stumbles on wet grass. The team widens the friction range and adds sensor latency to training, after which the robot crosses all test surfaces without any training on the real robot.