Hand tracking is the ability of an XR system to estimate the position, orientation and finger joint angles of the user’s bare hands in real time, usually with cameras on the headset, so that hands can serve as an input device without controllers. As a core element of tracking & interaction in extended reality, it turns natural movements such as pointing, pinching and grabbing into commands and lets a virtual hand mirror the real one.
Hand tracking is now standard on most standalone and mixed reality headsets. On some devices it is an optional alternative to controllers; on others, such as gaze-and-pinch interfaces, it is the primary input method together with eye tracking.
How hand tracking reconstructs the hand
Hand tracking works by detecting the hands in camera images and fitting a skeletal model to them. Headset cameras, often monochrome and sensitive to infrared, capture the hands from several angles. A machine learning model first locates each hand, then estimates keypoints for the wrist and finger joints, typically around 20 to 26 per hand depending on the platform.
Those keypoints are fitted to a kinematic hand model with realistic joint limits, which produces a full 3D pose: the position and rotation of the wrist plus the bend of every finger joint. Temporal filtering smooths the result between frames, and prediction compensates for part of the processing delay. Applications then receive the skeleton through the platform API or the OpenXR hand tracking extension.
The table below lists the main stages of a typical camera-based hand tracking pipeline.
| Stage | What happens | Main difficulty |
|---|---|---|
| Image capture | Cameras record the hands | Lighting and motion blur |
| Hand detection | Finds hand regions in each image | Hands partly out of view |
| Keypoint estimation | Predicts joint positions | Fingers hidden by other fingers |
| Model fitting | Fits a skeleton with joint limits | Varying hand sizes |
| Filtering and prediction | Smooths jitter and reduces lag | Balance between stability and delay |
| Gesture interpretation | Detects pinches, grabs and pokes | Avoiding accidental triggers |
Interaction techniques built on hand tracking
Hand tracking enables several interaction techniques, each suited to different distances and tasks. Designers combine them so that users can reach close objects directly and control distant ones through rays or gaze.
| Technique | How it works | Best suited to |
|---|---|---|
| Direct touch | Fingertip presses virtual buttons | Panels within arm’s reach |
| Direct grab | Closing the hand around an object | Picking up and moving objects |
| Ray and pinch | A ray from the hand, pinch to select | Distant interface elements |
| Gaze and pinch | Eyes target, fingers confirm | Fast selection with little arm movement |
| Discrete gestures | Recognized poses or motions | System shortcuts and menus |
The pinch between thumb and index finger has become the most reliable selection gesture because the touching fingertips are easy to detect and the user feels the contact physically. Gestures without such contact, such as an air tap in empty space, provide no physical confirmation, which makes visual and audio feedback essential.
Hand tracking compared with controllers and related tracking
Hand tracking differs from motion controllers in its trade-off between naturalness and precision. Controllers offer buttons, triggers, thumbsticks, haptic feedback and tracking that works even when the hands are out of camera view; bare hands need nothing to hold but lose tactile feedback and are less accurate during fast movement.
| Method | Precision | Haptic feedback | Typical use |
|---|---|---|---|
| Hand tracking | Medium, drops in fast motion | None | Menus, casual apps, mixed reality |
| Motion controller | High and consistent | Vibration | Games and precise manipulation |
| Haptic glove | High, with finger sensing | Force or vibration | Training and research |
| Eye tracking | High for targeting | None | Selection and foveated rendering |
| Gesture recognition | Recognizes poses, not full skeleton | None | Simple commands |
Gesture recognition is related but narrower: it classifies predefined poses, while full hand tracking delivers a continuous skeleton that applications can use freely.
Designing applications for hand tracking
Good hand tracking design starts from the limits of the sensors and the human body. Hands must stay within the cameras’ field of view, arms tire quickly when held up, and users need clear feedback because they cannot feel virtual objects.
| Problem | Recommended solution | Reason |
|---|---|---|
| Arm fatigue | Place panels low and close to the body | Reduces raised-arm strain |
| No tactile feedback | Highlight, sound and visual press depth | Confirms each action |
| Accidental selections | Require deliberate pinch with hysteresis | Prevents false triggers |
| Lost tracking | Fade virtual hands and pause actions | Avoids erratic behavior |
| Small targets | Use large buttons, about 2 cm or more | Compensates for jitter |
Developers usually access hand data through OpenXR, Unity’s XR Hands package or platform SDKs, and many projects support both hands and controllers so that users can switch freely.
Limitations and common failure modes
Camera-based hand tracking fails mainly because of occlusion and image quality. When one hand covers the other, when fingers point straight at the cameras or when the light is poor, the system must guess, and the virtual hand can jitter, freeze or snap into a wrong pose.
- Self-occlusion: overlapping hands and curled fingers reduce accuracy.
- Fast motion causes blur and noticeable latency.
- Out-of-view hands are lost when lowered or moved behind the head.
- Gorilla arm fatigue limits long sessions with raised arms.
- Fine precision tasks such as drawing or aiming remain better with controllers.
Stripped to the essentials
Hand tracking means the headset watches your hands with its cameras and copies their movements, finger by finger, into the virtual world. You do not need to hold anything.
To press a button, you touch it in the air; to pick something up, you pinch or grab it as you would in real life. It feels natural, but because there is nothing physical to touch, the system uses light, sound and animation to tell you that your action worked.
Where it is applied
A hospital uses a mixed reality headset with hand tracking to train nurses in preparing an infusion pump. The trainee sees the real training room through pass-through, while a virtual pump stands on a real cart.
Trainees press the pump’s virtual buttons with a fingertip, pinch to open menus and grab the virtual tubing to route it correctly. Because no controllers are needed, the motions resemble the real procedure more closely, and trainees can wear gloves as they would in practice, provided the gloves contrast well with the background.
The training team keeps all controls large and near waist height after noticing arm fatigue in early sessions. The result is a controller-free simulation that new staff learn within minutes.