Depth estimation is the task in computer vision of predicting, for every pixel or region of an image, the distance between the camera and the visible surface. The result is a depth map: an image-sized grid in which each value encodes distance in meters or a relative unit instead of color.
Within artificial intelligence, depth estimation recovers the third dimension that a camera discards when it projects a 3D scene onto a 2D sensor. It can work from two or more views, as in stereo and multi-view systems, or from a single image, which is called monocular depth estimation and relies entirely on learned visual cues.
How depth estimation recovers distance
Every depth estimation method exploits some cue that varies with distance. Geometric methods measure that cue directly, while learned methods infer it from patterns seen during training. The choice of cue determines whether the output is metric depth in real units or relative depth that is only correct up to an unknown scale.
In stereo depth estimation, two calibrated cameras a known distance apart, the baseline, observe the same point. The horizontal shift of that point between the two images is the disparity, and depth follows from the relation depth = focal length x baseline / disparity. Small disparities mean distant objects, so accuracy falls roughly with the square of distance.
Monocular depth estimation has no second view, so a neural network must use pictorial cues: perspective convergence, relative object size, texture gradients, occlusion order, shading and atmospheric haze. Because a single image is consistent with infinitely many scene scales, a network can only estimate absolute depth when it has learned the typical size of familiar objects or receives camera intrinsics.
The following table summarizes the main families of methods and the cue each one depends on.
| Approach | Input | Cue used | Output scale |
|---|---|---|---|
| Stereo matching | Two calibrated images | Horizontal disparity | Metric |
| Multi-view stereo | Many posed images | Triangulation across views | Metric if poses are metric |
| Monocular network | One image | Learned pictorial cues | Relative or metric |
| Video-based self-supervision | Image sequence | Motion between frames | Relative, up to scale |
| Depth completion | Image plus sparse LiDAR | Measured points plus image | Metric |
Network architectures and training signals
Modern monocular models use an encoder-decoder design: an encoder, often a convolutional backbone or a vision transformer, extracts features at several resolutions, and a decoder upsamples them into a dense map. Transformer encoders capture global scene layout, which helps the network reason about room geometry or road perspective across the whole frame.
Training signals vary. Supervised training uses ground truth from LiDAR, structured light or synthetic renderings. Self-supervised training uses stereo pairs or video: the network predicts depth and camera motion, warps one frame into another and minimizes the photometric reconstruction error. Large foundation depth models mix many datasets and use scale-and-shift-invariant losses so that sources with different units can be combined.
Researchers compare models with standard error metrics. Knowing them helps when reading benchmark claims.
| Metric | What it measures | Better value |
|---|---|---|
| AbsRel | Mean absolute error divided by true depth | Lower, often 0.05-0.15 |
| RMSE | Root mean squared error in meters | Lower |
| Delta < 1.25 | Share of pixels within 25 percent of truth | Higher, close to 1.0 |
| SILog | Scale-invariant logarithmic error | Lower |
How depth estimation compares with related tasks
Depth estimation is a dense, per-pixel regression problem, which sets it apart from neighboring computer vision tasks that output labels, boxes or full 3D models. It often serves as an intermediate representation that those tasks consume.
| Task | Output | Relation to depth |
|---|---|---|
| Depth estimation | Distance per pixel | Core task |
| Image segmentation | Class or instance per pixel | Shares the dense decoder design |
| 3D reconstruction | Mesh, point cloud or volume | Fuses many depth maps into one model |
| 3D object detection | Oriented 3D boxes | Often lifts image features using depth |
| Pose estimation | Joint positions | Depth resolves front-back ambiguity |
Hardware sensors such as time-of-flight cameras, structured light and LiDAR measure depth physically rather than estimating it. They are more reliable at short and medium range but cost more, consume power and struggle with sunlight, black surfaces and reflections, which is why learned estimation remains attractive for low-cost cameras.
Choosing a depth estimation setup
The right setup depends on whether absolute distances are required, the working range and the compute budget. A navigation stack needs metric depth it can trust, while photo effects only need a plausible ordering of near and far.
| Situation | Recommended choice | Reason |
|---|---|---|
| Portrait blur on a phone | Monocular relative depth | Ordering matters more than meters |
| Warehouse robot at 0.3-5 m | Active stereo or time-of-flight | Reliable metric depth indoors |
| Driving at highway range | LiDAR fused with cameras | Stereo error grows with distance |
| Archived single photos | Monocular foundation model | No second view exists |
| Sparse LiDAR available | Depth completion network | Densifies accurate points |
For stereo rigs, a wider baseline improves far-range precision but increases occlusions and the minimum working distance. Calibration quality matters as much as the algorithm: a small rotation error between cameras shifts every disparity and corrupts the whole map.
Limitations and common mistakes
Depth estimation fails in predictable ways. Practitioners who know these patterns can validate outputs before trusting them in control loops.
- Scale ambiguity: treating relative monocular output as meters produces errors that look plausible but break measurement and planning.
- Textureless and reflective surfaces: white walls, glass and mirrors give stereo no features to match and confuse learned cues.
- Blurred object edges: decoders smooth depth discontinuities, creating flying pixels between foreground and background.
- Domain shift: a model trained on outdoor driving scenes often misjudges indoor rooms, and the reverse.
- Camera mismatch: a different focal length from the training data changes metric predictions unless intrinsics are supplied.
The short version
A normal photo records colors. It forgets how far away things were. Depth estimation lets software guess those distances back. Every point in the picture gets a number for how near or far it is.
People do this without thinking. With two eyes, nearby objects shift more between the views. Even with one eye closed, we know a car looks small because it is far away. Computers copy both tricks. They use two cameras or a trained network that learned these clues from many examples. A single-camera guess is usually good at which object is nearer. It is less reliable about exact meters.
Where it makes a difference
A logistics company equips small indoor delivery robots with a single wide-angle camera to keep hardware costs low. A monocular depth network running on an embedded GPU produces a 256 x 192 depth map at about 20 frames per second, and the planner uses it to detect obstacles within 0.5 to 4 meters.
During testing, the team finds that the robots slow down near glass doors and polished floors, where predicted depth becomes unstable. They add a low-cost time-of-flight sensor for the nearest meter and fine-tune the network on images from their own buildings. The result is fewer false stops and no collisions with glass in the trial period, while the camera-based estimate still covers the longer range the cheap sensor cannot reach.