A depth camera is an imaging sensor that measures the distance from the camera to the scene at every pixel, producing a depth image, often together with a matching color image (RGB-D). In robotics, it gives a robot dense 3D geometry at video frame rates from a compact, inexpensive device, which makes it a standard sensor for grasping, obstacle detection and mapping at short to medium range.
Unlike a regular camera, which records only brightness and color, a depth camera encodes each pixel as a distance value, typically in millimeters. With the camera’s intrinsic parameters, every depth pixel can be converted into a 3D point, so one frame becomes a point cloud that perception algorithms can process directly.
How a depth camera measures distance
Depth cameras use one of three main principles: stereo vision, structured light or time of flight. Each recovers distance differently, and that choice determines the range, precision, robustness to sunlight and failure modes of the sensor.
- Passive or active stereo: two cameras a known baseline apart observe the scene, and depth follows from the disparity of matching points. An infrared pattern projector adds texture to plain surfaces in active stereo.
- Structured light: a projector casts a known pattern, and a camera measures how the pattern deforms on surfaces to triangulate depth.
- Time of flight (ToF): the camera emits modulated infrared light and measures the phase shift or travel time of the reflection at each pixel.
Triangulation-based sensors share a key property: depth error grows roughly with the square of distance. Doubling the distance roughly quadruples the depth uncertainty, so a camera that is accurate to a few millimeters at 0.5 m may show errors of several centimeters at 4 m.
Depth sensing technologies compared
The three technologies trade range, accuracy and environmental robustness against each other. The comparison below uses typical values for compact robotics cameras; exact figures depend on the specific model and settings.
| Technology | Typical range | Sunlight robustness | Main weakness |
|---|---|---|---|
| Active stereo | 0.2 to 10 m | Good, works as passive stereo outdoors | Noisy on low-texture or distant surfaces |
| Structured light | 0.1 to 4 m | Poor, pattern washes out | Interference between several units |
| Indirect time of flight | 0.2 to 5 m | Moderate | Multipath errors in corners |
| Direct time of flight | Up to tens of meters | Better, with narrowband filters | Lower resolution, higher cost |
High-precision structured light scanners used in industrial bin picking reach sub-millimeter accuracy at close range, but they often need a static scene and longer capture times. Consumer-grade RGB-D cameras favor 30 to 90 frames per second over metrology-level accuracy.
Key parameters of a depth camera
Choosing a depth camera starts with its working range, depth accuracy, field of view and frame rate. These parameters interact: a wide field of view spreads pixels over a larger area, which lowers spatial resolution at the same distance.
| Parameter | Typical values | What it affects |
|---|---|---|
| Minimum depth | 0.1 to 0.5 m | Whether close objects in a gripper can be seen |
| Depth resolution | 320 × 240 to 1280 × 720 | Detail on small parts and thin edges |
| Depth accuracy | Under 1 % to about 2 % of distance | Reliability of grasp and collision estimates |
| Field of view | About 55 to 90 degrees horizontal | Coverage versus pixel density |
| Frame rate | 15 to 90 fps | Usability on moving robots and arms |
| Baseline (stereo) | About 50 to 120 mm | Longer baseline improves far-range precision |
Calibration and synchronization matter as much as the datasheet. The depth and color images must be registered to each other, and the camera pose must be known relative to the robot base, a process called hand-eye calibration when the camera is mounted on or observes the arm.
Depth camera compared with other robot sensors
A depth camera is one of several sensors a robot combines, and it fills a specific niche: dense geometry at short range for a low price. It does not replace longer-range or contact sensors.
| Sensor | What it measures | Range | Best role |
|---|---|---|---|
| Depth camera | Dense depth per pixel | Short to medium | Grasping, near-field obstacles, 3D object pose |
| LiDAR | Sparse to dense ranges by laser | Medium to long | Mapping and navigation outdoors and in large halls |
| IMU | Acceleration and angular rate | Not applicable | Motion estimation between frames |
| Force sensor | Contact forces and torques | Contact only | Controlling insertion and grip force |
| Tactile sensor | Pressure distribution on a surface | Contact only | Slip detection and fine manipulation |
In practice, a mobile robot often pairs a LiDAR for navigation with one or more depth cameras that see low or overhanging obstacles the planar scanner misses, while an arm uses the depth camera to find objects and a force sensor to finish the task.
Limitations and typical artifacts
Depth cameras fail in predictable ways, and robust perception code must detect and handle invalid pixels rather than trust every value. Most sensors mark missing measurements as zero depth.
- Reflective and transparent surfaces: glass, polished metal and clear plastic return wrong or missing depth.
- Dark, absorbing materials: black foam or rubber reflects too little infrared light.
- Flying pixels at edges: depth values blend foreground and background along object borders.
- Multipath interference: ToF cameras overestimate distance in concave corners.
- Sensor crosstalk: several active cameras facing the same area can disturb each other.
Common mitigations include temporal and spatial filtering, masking unreliable regions, staggering exposure between cameras, and fusing depth with color-based object recognition when geometry alone is ambiguous.
In plain words
A normal camera tells a robot what things look like. A depth camera also tells it how far away each point is, much like a pair of human eyes judges distance.
With that information, the robot can see the shape of a box, the edge of a table or the gap between two objects. It turns a flat picture into a 3D view of the nearby world, which the robot needs to pick things up and avoid bumping into them. It works best indoors and at arm’s length, and it struggles with shiny or see-through objects.
A real-world example
A warehouse deploys a robot arm to pick mixed cardboard boxes from a pallet. An active stereo depth camera mounted 1.5 m above the pallet captures depth and color at 30 frames per second, and hand-eye calibration links the camera frame to the robot base.
Software converts each depth frame into a point cloud, finds the top faces of the boxes as planes, and computes a suction grasp at the center of each face. At this distance, the depth error is a few millimeters, which is well within the compliance of the suction cup. The arm picks boxes of varying size without any prior model of the load. Shrink-wrapped pallets initially caused gaps in the depth image because of reflections, so the team tilted the camera slightly and added filtering, which restored reliable picks.