Sensor fusion is the process of combining measurements from several sensors into a single estimate of a robot’s state or environment that is more accurate, more complete or more reliable than any of the sensors could provide alone. In robot perception, it merges data from cameras, LiDAR, radar, inertial measurement units, wheel encoders and satellite positioning into one consistent picture of where the robot is and what surrounds it.
Fusion works because sensors fail in different ways. An IMU reacts within milliseconds but drifts, satellite positioning is absolute but slow and easily blocked, and a camera sees texture but not reliable depth. Weighting each source by its known uncertainty lets the strengths of one cover the weaknesses of another.
Why combining sensors improves the estimate
Sensor fusion improves an estimate by weighting each measurement according to its uncertainty, so precise sources count more and noisy ones count less. For two independent measurements of the same quantity, the optimal fused variance is the product of the two variances divided by their sum. Two range readings with standard deviations of 3 cm and 4 cm therefore fuse to about 2.4 cm, better than either sensor on its own.
That gain holds only if the errors are truly independent and correctly modeled. The classic taxonomy describes three ways sensors can relate to each other, and real systems usually mix them.
| Fusion type | Relationship between sensors | Robotic example |
|---|---|---|
| Complementary | Each covers what the other misses | IMU for fast motion, GNSS for absolute position |
| Competitive (redundant) | Both measure the same quantity | Two LiDARs viewing the same zone |
| Cooperative | Together they derive a new quantity | Stereo cameras computing depth |
Levels of sensor fusion
Sensor fusion can take place at three levels of abstraction: on raw data, on extracted features or on finished decisions. The level fixes how much information survives and how tightly the system depends on precise calibration.
| Level | What is combined | Advantage | Drawback |
|---|---|---|---|
| Low level (early) | Raw measurements, such as LiDAR points projected onto image pixels | Keeps the most information | Needs exact calibration and timing |
| Feature level | Edges, keypoints, learned feature maps | Balances detail and bandwidth | Feature design affects results |
| Decision level (late) | Object lists or classifications from each sensor | Modular, tolerant of sensor loss | Discards evidence below thresholds |
The same distinction appears in state estimation as loosely coupled versus tightly coupled fusion. A loosely coupled system fuses finished position fixes, while a tightly coupled one fuses raw observations, such as individual satellite pseudoranges or camera feature tracks, and remains useful even when each source alone could not produce a fix.
Estimation algorithms behind sensor fusion
Most robotic sensor fusion is probabilistic state estimation: a motion model predicts the next state, and each new measurement corrects the prediction in proportion to its relative certainty. The Kalman filter formalizes this predict-and-update cycle, and its Kalman gain sets how strongly each measurement pulls the estimate.
| Algorithm | Key assumption | Computing cost | Typical use |
|---|---|---|---|
| Complementary filter | Sensors differ mainly in frequency content | Very low | Attitude from gyroscope and accelerometer |
| Kalman filter | Linear models, Gaussian noise | Low | Tracking with linear dynamics |
| Extended Kalman filter (EKF) | Nonlinear models linearized locally | Moderate | Robot localization, inertial navigation |
| Unscented Kalman filter (UKF) | Nonlinearity captured by sigma points | Moderate | Strongly nonlinear motion or sensors |
| Particle filter | Arbitrary, multimodal distributions | High | Global localization on a known map |
| Factor graph optimization | Batch or sliding-window least squares | High, but sparse | Visual-inertial odometry and SLAM |
Filters process data once and move on, which keeps latency low. Factor graph methods revisit a window of past states and relinearize them, which is why modern visual odometry and SLAM systems increasingly prefer them when computing power allows.
Calibration and synchronization requirements
Sensor fusion is only as good as the spatial calibration and time alignment of its inputs, and small errors in either grow with speed and distance. The table shows how modest mistakes turn into large position errors.
| Requirement | Error example | Resulting effect | Usual mitigation |
|---|---|---|---|
| Time synchronization | 10 ms timestamp offset at 30 m/s | About 30 cm position mismatch | Hardware triggering, PTP clock sync |
| Extrinsic calibration | 1 degree rotation error between sensors | About 35 cm offset at 20 m | Target-based or online calibration |
| Noise model | Covariance set too small | Filter overconfident, then diverges | Tune on logged data, inflate conservatively |
| Outlier rejection | Multipath GNSS fix accepted | Sudden jump in estimate | Mahalanobis gating, robust cost functions |
A frequent and subtle mistake is double-counting correlated information, for example fusing a position that was itself derived from the same IMU data. The filter then believes it is far more certain than it is, rejects valid measurements and can drift without warning.
Common misconceptions about sensor fusion
- More sensors do not automatically mean better results. An uncalibrated or poorly modeled sensor can degrade the fused estimate.
- Fusion cannot remove shared blind spots. If all sensors are blinded by the same fog or glare, no algorithm recovers the missing information.
- A small reported covariance is not proof of accuracy. It reflects the model’s assumptions and must be validated against ground truth.
- Deep learning does not eliminate the need for calibration. Learned fusion networks still depend on consistent geometry and timing between sensors.
Understanding it simply
Sensor fusion is how a robot combines several imperfect senses into one trustworthy picture. Each sensor is good at something and bad at something else, so the robot listens to all of them and trusts each one according to how reliable it usually is.
People do the same thing. When you walk in a dark room, you combine what you faintly see, what your feet feel and your sense of balance. No single sense is enough, but together they keep you upright and moving in the right direction.
Applied in practice
A sidewalk delivery robot runs an extended Kalman filter at 100 Hz. Each cycle, IMU and wheel encoder readings predict how far the robot has moved and turned. Satellite positioning with real-time kinematic corrections arrives at 10 Hz and pulls the estimate toward an absolute position with an uncertainty of a few centimeters in open sky.
Between tall buildings, the satellite fixes become unreliable, and the filter’s gating step rejects them. The robot then matches LiDAR scans against a prebuilt map to keep correcting its position. Because every source is weighted by its current uncertainty, the robot stays within its lane on the sidewalk through an urban canyon where satellite positioning alone would have placed it several meters off, inside a building.