Keypoint detection is the computer vision task of predicting the pixel coordinates of specific, semantically defined points in an image, such as the joints of a human body, the corners of the eyes or the corners of a package. Also called landmark detection, it turns an image into a compact set of locations that describe the geometry and pose of an object rather than only its presence.
The term covers two related traditions. Semantic keypoints follow a fixed schema, for example the 17 body joints of the COCO format, while generic interest points found by detectors such as SIFT or corner detectors have no fixed meaning and serve matching and reconstruction. This page focuses mainly on the semantic, learned form used in modern deep learning.
How deep keypoint detectors locate points
Modern keypoint detectors use a convolutional or transformer backbone to extract features and then predict each point’s location with one of two output representations. The dominant approach outputs one heatmap per keypoint, a low-resolution map in which a two-dimensional Gaussian peak marks the most likely location; the predicted coordinate is the peak or a refined sub-pixel estimate around it.
The alternative is direct coordinate regression, where the network outputs numeric x and y values. Regression is lighter and avoids quantization, but historically it was less accurate; techniques such as integral regression (soft-argmax over a heatmap) and SimCC-style classification of x and y bins narrow that gap. Each keypoint is usually accompanied by a visibility or confidence score, which tells downstream systems whether the point is occluded or uncertain.
The table below compares the main output representations used by keypoint models.
| Representation | Strength | Weakness |
|---|---|---|
| Gaussian heatmaps | Accurate, spatially robust, easy to supervise | Memory cost grows with resolution and keypoint count |
| Direct regression | Compact, fast, no quantization error | Harder to train, weaker on ambiguous points |
| Integral regression | Differentiable sub-pixel output from heatmaps | Sensitive to multimodal heatmaps |
| Coordinate classification | Separate x and y bins, low memory | Bin size limits precision |
Top-down and bottom-up pipelines
When an image contains several objects of the same kind, a detector must decide which keypoints belong to which instance. Two pipeline designs solve this problem in different ways, and a third, single-stage family has become common in real-time systems.
| Pipeline | How it works | Best suited for |
|---|---|---|
| Top-down | Detect each object with a bounding box, then find keypoints inside each crop | High accuracy with few instances |
| Bottom-up | Find all keypoints in the image, then group them into instances | Crowded scenes with many people |
| Single-stage | Predict boxes and keypoints together in one network pass | Real-time video on edge hardware |
Top-down accuracy depends on the object detector: a missed or badly cropped person cannot be recovered later, and runtime grows linearly with the number of people. Bottom-up methods keep a roughly constant runtime but need a grouping mechanism, such as part affinity fields or associative embeddings, which can fail when limbs overlap. Single-stage designs, including keypoint heads added to YOLO-family detectors, trade some precision for speed.
Keypoint detection and related computer vision tasks
Keypoint detection is a building block rather than an end goal in most systems, and it overlaps with several neighboring tasks. The comparison clarifies where it fits.
| Task | Output | Relation to keypoint detection |
|---|---|---|
| Keypoint detection | Coordinates of defined points | The task described here |
| Pose estimation | Body or object configuration | Usually built on keypoints plus a skeleton |
| Object detection | Bounding boxes and classes | Often the first stage of top-down pipelines |
| Corner detection | Generic high-gradient points | Classical, schema-free form of interest points |
| Feature matching | Correspondences between images | Uses generic keypoints with descriptors |
| Instance segmentation | Pixel masks per object | Describes shape, not articulated structure |
In structure from motion and image registration, the keypoints are generic and paired with descriptors that allow matching across views. In pose and landmark work, the keypoints carry fixed semantic labels, so the left wrist in one image corresponds to the left wrist in any other.
Datasets, schemas and evaluation metrics
A keypoint task is defined by its schema: the list of points, their order, their left-right pairs and the skeleton that connects them. Public schemas include 17 body keypoints in COCO, 68 facial landmarks in the iBUG 300-W convention and 21 hand points in common hand datasets. Custom schemas for industrial or animal applications are frequent, and the left-right flip pairs must be declared so that horizontal augmentation does not swap labels.
Accuracy is measured with metrics that account for object scale and annotation uncertainty, because a 5-pixel error is severe on a small face but negligible on a large person.
| Metric | What it measures | Typical use |
|---|---|---|
| Object keypoint similarity (OKS) | Scale-normalized distance with per-keypoint tolerance | COCO keypoint benchmark, AP and AR at OKS thresholds |
| PCK | Share of points within a fraction of a reference length | Body and hand pose, often PCKh at 0.5 |
| Normalized mean error (NME) | Average error divided by interocular or box size | Facial landmark detection |
| Mean per-joint position error | Average 3D distance in millimeters | 3D pose estimation |
Practical choices when building a keypoint model
The right configuration depends on instance count, latency budget and required precision. Heatmap models at input sizes of 256 by 192 pixels or larger remain a strong default for accuracy, while lightweight regression or single-stage models suit mobile and embedded devices.
- Define the schema and visibility rules before annotation; changing them later invalidates labels.
- Annotate occluded points consistently, either as estimated positions or as not visible, and document the choice.
- Use flip, rotation and scale augmentation with correct left-right pair swapping.
- Apply temporal smoothing such as a one-euro filter in video to reduce frame-to-frame jitter.
- Report metrics on held-out scenes and subjects, not only on random frames from the training videos.
Limitations and failure modes
Keypoint detectors fail in predictable ways, and most errors trace back to ambiguity in the image or in the annotation. Knowing the symptoms speeds up debugging.
- Left-right swaps: symmetric points such as knees are confused, often from incorrect flip pairs or back-facing views.
- Occlusion errors: hidden joints are placed on visible but wrong body parts.
- Instance mixing: in crowds, points from two people merge into one implausible skeleton.
- Domain shift: models trained on everyday photos degrade on thermal, fisheye or underwater images.
- Annotation noise: inconsistent labeling of points such as hips sets an upper bound on achievable accuracy.
Put simply
Keypoint detection teaches a computer to put pins on the important spots in a picture. For a person, the pins go on the shoulders, elbows, knees and other joints. For a face, they go on the corners of the eyes, the tip of the nose and the edges of the mouth.
Once those pins are in place, software can connect them like a stick figure and understand how something is positioned or moving, without having to analyze every pixel again.
In practice
A physiotherapy clinic uses a tablet app to check how patients perform squats during knee rehabilitation. The app runs a lightweight single-stage keypoint model that locates 17 body points at about 30 frames per second, then computes the knee and hip angles from the hip, knee and ankle coordinates.
Early tests show jitter in the knee angle of several degrees between frames and occasional left-right knee swaps when patients turn sideways. The developers add temporal smoothing, restrict the camera to a side view and fine-tune the model on 3,000 annotated clinic frames.
After these changes, the measured knee angle stays within a few degrees of goniometer readings, and therapists use the app to track range-of-motion progress between sessions instead of relying on visual estimates.