Visual servoing is a robot control technique that uses camera images as direct feedback in a closed control loop, moving the robot until features observed in the image match a desired configuration. In robotics perception, it is the point where seeing and acting merge: instead of measuring a target once and then moving blindly, the robot keeps looking while it moves and corrects continuously.
The approach is used when the target position is uncertain, moves or cannot be modeled precisely, for example when aligning a tool with a part, grasping objects on a moving conveyor or docking a mobile robot. It reduces dependence on perfect calibration and kinematic accuracy, because errors are measured in the image itself.
How the visual servoing loop works
The controller defines an error between current and desired visual features. Features can be image coordinates of points, lines, the area and centroid of a blob, or a full estimated pose. At each cycle the camera captures an image, the features are extracted, the error is computed and a velocity command is sent to the robot to reduce it.
The link between camera motion and feature motion is the interaction matrix, also called the image Jacobian. A classic control law commands a camera velocity proportional to the error, multiplied by the pseudo-inverse of the interaction matrix and a gain. With a well-chosen gain the error decreases roughly exponentially toward zero.
The interaction matrix depends on the depth of each feature, which is often unknown. Practical systems use the depth at the desired position, an estimate from stereo vision or a depth camera, or an online estimate updated while the robot moves.
The computed camera velocity must finally be converted into joint velocities through the robot Jacobian and the hand-eye transformation. Most industrial controllers accept such commands only through a dedicated real-time streaming interface, so the available interface and its cycle time strongly influence how well visual servoing performs on a given robot.
Image-based, position-based and hybrid approaches
Visual servoing schemes differ in where the error is defined. Image-based visual servoing (IBVS) computes it directly in image coordinates. Position-based visual servoing (PBVS) first estimates the 3D pose of the target and computes the error in Cartesian space. Hybrid schemes, often called 2.5D visual servoing, combine image features with partial pose information.
| Scheme | Error defined in | Strength | Weakness |
|---|---|---|---|
| IBVS | Image feature coordinates | Robust to calibration errors | Indirect, sometimes poor 3D paths |
| PBVS | Estimated 3D pose | Straight, predictable 3D motion | Sensitive to model and calibration |
| 2.5D hybrid | Image features and rotation | Balances both behaviors | More complex to implement |
| Direct or photometric | Pixel intensities | No feature extraction needed | Small convergence region |
| Learning-based | Learned features or policies | Handles unstructured scenes | Hard to guarantee stability |
A known weakness of IBVS is behavior under large rotations about the optical axis: the controller may move the camera backward before rotating, sometimes reaching joint limits. PBVS avoids this but can let features leave the field of view, after which the loop loses feedback.
Camera configurations and system parameters
Visual servoing systems are classified by where the camera is mounted. In the eye-in-hand configuration the camera rides on the robot end effector, so it sees the target close up and its view changes with every move. In the eye-to-hand configuration a fixed camera watches both the robot and the target. Both depend on hand-eye calibration to relate camera and robot coordinates, although IBVS tolerates moderate errors.
| Parameter | Typical values | Effect |
|---|---|---|
| Camera frame rate | About 30 to several hundred fps | Loop bandwidth and tracking speed |
| Total latency | Ideally tens of milliseconds | Stability and overshoot |
| Control gain | Tuned per task | Convergence speed versus oscillation |
| Feature count | At least four points for full 6-DoF | Uniqueness and robustness |
| Image resolution | Depends on required precision | Final positioning accuracy |
Latency is usually the limiting factor. Image capture, processing and communication delays reduce the usable gain, so systems that must follow fast targets combine visual servoing with motion prediction or use high-speed cameras and dedicated processing.
Visual servoing compared with related perception methods
Visual servoing is a control method, not just a perception method, which distinguishes it from neighboring Terms in robot perception. The table contrasts it with approaches that also use cameras.
| Method | Main output | Closes the control loop |
|---|---|---|
| Visual servoing | Robot velocity commands | Yes, continuously |
| Look-then-move | One target pose, then motion | No, single measurement |
| Visual odometry | Estimated camera motion | No, estimation only |
| SLAM | Map and robot pose | No, used by planners |
| Object recognition | Object identity and location | No, informs decisions |
The look-then-move approach is simpler and common in industry: a camera measures a part once, and the robot moves to the computed pose. It works well when the robot is accurate and the target is static, while visual servoing is preferred when targets move or accuracy depends on final alignment.
Practical challenges and failure modes
- Feature loss: occlusion by the gripper or features leaving the image breaks the loop; planners keep features visible.
- Local minima: with redundant point features, IBVS can converge to a wrong configuration.
- Lighting changes: reflections and shadows disturb feature extraction, especially on metal parts.
- Singularities: some feature configurations make the interaction matrix ill-conditioned.
- Safety near people: velocity commands from vision must respect speed and force limits.
Robust implementations use distinctive fiducial markers where possible, filter feature measurements and switch to a fallback strategy when confidence drops.
In plain words
Think of threading a needle. You do not measure the hole once and then close your eyes; you watch the thread and the needle the whole time and make small corrections until they line up. Visual servoing lets a robot work the same way.
A camera keeps checking how far the current view is from the desired view, and the robot moves a little to shrink that difference, again and again, many times per second. The result is precise alignment even when the robot or the object is not exactly where everyone expected.
A real-world example
An electronics manufacturer needs a robot to insert connectors into circuit boards that arrive on a conveyor with position errors of a few millimeters. A single camera snapshot followed by a blind move often misses, because the board shifts slightly and the robot’s absolute accuracy is limited.
Engineers mount a small camera on the robot wrist and use image-based visual servoing on four fiducial marks near the connector socket. During the final approach the loop runs at about 100 cycles per second, aligning the connector to within a fraction of a millimeter before insertion. Failed insertions become rare, and the cell no longer needs precise fixtures for every board variant.