Stereo vision is the technique of recovering three-dimensional depth from two images taken by cameras a known distance apart. By finding the same point in both images and measuring how far it shifts between them, a system computes how far away that point is. In robot perception, stereo vision is one of the main ways to build dense depth maps and point clouds without active illumination.
The principle mirrors human binocular vision: the two eyes see slightly different views, and the brain fuses them into depth. A stereo camera rig does the same with geometry and computation, producing depth for every matched pixel at camera frame rates.
How disparity becomes depth
The core relation of stereo vision is simple. For a rectified camera pair, depth Z equals focal length times baseline divided by disparity, written Z = f·B/d. The baseline B is the distance between the camera centers, the focal length f is expressed in pixels, and the disparity d is the horizontal pixel shift of a point between the left and right images.
Because disparity sits in the denominator, depth error grows with the square of distance. A one-pixel matching error barely matters at one meter but becomes large at twenty meters. This quadratic error growth is the defining limitation of stereo vision and drives most design choices.
| Step | Purpose | Typical method |
|---|---|---|
| Calibration | Find intrinsics, distortion and relative pose | Checkerboard or ChArUco target |
| Rectification | Align epipolar lines with image rows | Warp both images using calibration |
| Matching | Find corresponding pixels | Block matching, SGM or neural networks |
| Disparity refinement | Remove outliers and fill gaps | Left-right check, subpixel interpolation |
| Triangulation | Convert disparity to 3D points | Z = f·B/d per pixel |
Rectification is what makes matching practical. It reprojects the images so that corresponding points lie on the same image row, reducing a two-dimensional search to a one-dimensional search along that row.
Design parameters of a stereo rig
Stereo performance is set mainly by baseline, focal length, resolution and matching quality. Each parameter trades range against other properties, so a rig designed for a tabletop arm differs from one designed for an outdoor vehicle.
| Parameter | Typical range | Effect of increasing it |
|---|---|---|
| Baseline | About 5-30 cm on robots | Better far accuracy, larger minimum range |
| Focal length | Wide to normal lenses | Better accuracy, narrower field of view |
| Resolution | About 0.3-5 megapixels | Finer disparity, more computation |
| Disparity search range | About 64-256 pixels | Closer minimum range, slower matching |
| Subpixel accuracy | About 0.1-0.25 pixel | Lower depth noise at long range |
A wider baseline improves accuracy at distance but enlarges the blind zone near the cameras and increases occlusion, where a point is visible to only one camera. Many robots therefore use a baseline of roughly 6 to 12 cm for manipulation and wider rigs for driving.
Stereo matching algorithms
The hardest part of stereo vision is the correspondence problem: deciding which pixel in the right image matches a pixel in the left image. Algorithms differ in how much context they use to resolve ambiguity.
- Block matching compares small windows with a cost such as the sum of absolute differences; it is fast but noisy on weak texture.
- Semi-global matching (SGM) adds smoothness costs along several image directions and offers a strong balance of speed and quality, often on FPGAs or GPUs.
- Deep stereo networks learn matching costs and regularization from data and usually give the best accuracy on difficult surfaces, at higher compute cost.
- Active stereo projects an infrared dot pattern to add texture, which helps on blank walls and tabletops indoors.
Textureless and repetitive surfaces defeat passive matching because many candidate pixels look alike. White walls, glossy floors and fences are classic failure cases, which is why active stereo and learned methods have spread.
Stereo vision compared with other depth sensing
Stereo vision competes with time-of-flight cameras, structured light and LiDAR. Each method fits different ranges, lighting conditions and budgets.
| Method | Strengths | Weaknesses |
|---|---|---|
| Passive stereo | Works in sunlight, cheap, dense output | Fails on textureless surfaces |
| Active stereo | Handles blank surfaces indoors | Projector washed out by strong sun |
| Time-of-flight camera | Direct depth, little computation | Multipath errors, limited sunlight range |
| Structured light | High accuracy at short range | Sensitive to ambient light |
| LiDAR | Accurate long range, independent of texture | Sparse points, higher cost |
Stereo also differs from monocular depth estimation, which predicts depth from a single image using learned priors. Monocular methods give plausible relative depth but lack the metric geometry that a calibrated baseline provides, so stereo remains preferable when absolute distances matter.
Common errors and how to avoid them
Most poor stereo results come from calibration, synchronization and scene conditions, not from the matching algorithm itself. The following table lists frequent symptoms.
| Symptom | Likely cause | Fix |
|---|---|---|
| Depth warps or tilts | Calibration drift after shocks or heat | Rigid mount and periodic recalibration |
| Ghost edges on moving objects | Unsynchronized camera exposures | Hardware-triggered global shutter cameras |
| Holes on walls and floors | Missing texture | Active pattern or learned matching |
| Noisy far depth | Short baseline or low resolution | Wider baseline or longer lens |
| Errors near object borders | Occlusion between views | Left-right consistency check |
A plain-language summary
Stereo vision lets a machine judge distance with two cameras, the way people use two eyes. Hold a finger in front of your face and close one eye, then the other: the finger seems to jump sideways. Close things jump far, while distant things barely move.
A stereo camera measures that jump for every point it can match in both pictures. A big jump means close, a small jump means far. With a little geometry, the robot turns those jumps into a full three-dimensional picture of its surroundings.
An example from practice
An agricultural robot picks strawberries in a polytunnel. It needs millimeter-level depth within about one meter to guide its gripper, works partly in sunlight that washes out infrared projectors, and must keep hardware costs low.
The team mounts a global shutter stereo camera with an 8 cm baseline on the robot arm and runs a learned stereo network on an embedded GPU. At 40 cm, a disparity error of a quarter pixel corresponds to only about a millimeter of depth error, which is enough to position the cutter at the stem.
The system detects berries in the left image, looks up their 3D positions in the depth map and plans approach paths around leaves. Because the camera is passive, performance stays stable in direct sun, which is where the earlier time-of-flight prototype struggled.