Keypoint detection

Also known as: landmark detection

On this page 8

Keypoint detection is the computer vision task of predicting the pixel coordinates of specific, semantically defined points in an image, such as the joints of a human body, the corners of the eyes or the corners of a package. Also called landmark detection, it turns an image into a compact set of locations that describe the geometry and pose of an object rather than only its presence.

The term covers two related traditions. Semantic keypoints follow a fixed schema, for example the 17 body joints of the COCO format, while generic interest points found by detectors such as SIFT or corner detectors have no fixed meaning and serve matching and reconstruction. This page focuses mainly on the semantic, learned form used in modern deep learning.

How deep keypoint detectors locate points

Modern keypoint detectors use a convolutional or transformer backbone to extract features and then predict each point’s location with one of two output representations. The dominant approach outputs one heatmap per keypoint, a low-resolution map in which a two-dimensional Gaussian peak marks the most likely location; the predicted coordinate is the peak or a refined sub-pixel estimate around it.

The alternative is direct coordinate regression, where the network outputs numeric x and y values. Regression is lighter and avoids quantization, but historically it was less accurate; techniques such as integral regression (soft-argmax over a heatmap) and SimCC-style classification of x and y bins narrow that gap. Each keypoint is usually accompanied by a visibility or confidence score, which tells downstream systems whether the point is occluded or uncertain.

The table below compares the main output representations used by keypoint models.

RepresentationStrengthWeakness
Gaussian heatmapsAccurate, spatially robust, easy to superviseMemory cost grows with resolution and keypoint count
Direct regressionCompact, fast, no quantization errorHarder to train, weaker on ambiguous points
Integral regressionDifferentiable sub-pixel output from heatmapsSensitive to multimodal heatmaps
Coordinate classificationSeparate x and y bins, low memoryBin size limits precision
Output representations for keypoint prediction

Top-down and bottom-up pipelines

When an image contains several objects of the same kind, a detector must decide which keypoints belong to which instance. Two pipeline designs solve this problem in different ways, and a third, single-stage family has become common in real-time systems.

PipelineHow it worksBest suited for
Top-downDetect each object with a bounding box, then find keypoints inside each cropHigh accuracy with few instances
Bottom-upFind all keypoints in the image, then group them into instancesCrowded scenes with many people
Single-stagePredict boxes and keypoints together in one network passReal-time video on edge hardware
Multi-instance keypoint pipelines

Top-down accuracy depends on the object detector: a missed or badly cropped person cannot be recovered later, and runtime grows linearly with the number of people. Bottom-up methods keep a roughly constant runtime but need a grouping mechanism, such as part affinity fields or associative embeddings, which can fail when limbs overlap. Single-stage designs, including keypoint heads added to YOLO-family detectors, trade some precision for speed.

Keypoint detection is a building block rather than an end goal in most systems, and it overlaps with several neighboring tasks. The comparison clarifies where it fits.

TaskOutputRelation to keypoint detection
Keypoint detectionCoordinates of defined pointsThe task described here
Pose estimationBody or object configurationUsually built on keypoints plus a skeleton
Object detectionBounding boxes and classesOften the first stage of top-down pipelines
Corner detectionGeneric high-gradient pointsClassical, schema-free form of interest points
Feature matchingCorrespondences between imagesUses generic keypoints with descriptors
Instance segmentationPixel masks per objectDescribes shape, not articulated structure
How keypoint detection relates to neighboring tasks

In structure from motion and image registration, the keypoints are generic and paired with descriptors that allow matching across views. In pose and landmark work, the keypoints carry fixed semantic labels, so the left wrist in one image corresponds to the left wrist in any other.

Datasets, schemas and evaluation metrics

A keypoint task is defined by its schema: the list of points, their order, their left-right pairs and the skeleton that connects them. Public schemas include 17 body keypoints in COCO, 68 facial landmarks in the iBUG 300-W convention and 21 hand points in common hand datasets. Custom schemas for industrial or animal applications are frequent, and the left-right flip pairs must be declared so that horizontal augmentation does not swap labels.

Accuracy is measured with metrics that account for object scale and annotation uncertainty, because a 5-pixel error is severe on a small face but negligible on a large person.

MetricWhat it measuresTypical use
Object keypoint similarity (OKS)Scale-normalized distance with per-keypoint toleranceCOCO keypoint benchmark, AP and AR at OKS thresholds
PCKShare of points within a fraction of a reference lengthBody and hand pose, often PCKh at 0.5
Normalized mean error (NME)Average error divided by interocular or box sizeFacial landmark detection
Mean per-joint position errorAverage 3D distance in millimeters3D pose estimation
Common keypoint evaluation metrics

Practical choices when building a keypoint model

The right configuration depends on instance count, latency budget and required precision. Heatmap models at input sizes of 256 by 192 pixels or larger remain a strong default for accuracy, while lightweight regression or single-stage models suit mobile and embedded devices.

  • Define the schema and visibility rules before annotation; changing them later invalidates labels.
  • Annotate occluded points consistently, either as estimated positions or as not visible, and document the choice.
  • Use flip, rotation and scale augmentation with correct left-right pair swapping.
  • Apply temporal smoothing such as a one-euro filter in video to reduce frame-to-frame jitter.
  • Report metrics on held-out scenes and subjects, not only on random frames from the training videos.

Limitations and failure modes

Keypoint detectors fail in predictable ways, and most errors trace back to ambiguity in the image or in the annotation. Knowing the symptoms speeds up debugging.

  • Left-right swaps: symmetric points such as knees are confused, often from incorrect flip pairs or back-facing views.
  • Occlusion errors: hidden joints are placed on visible but wrong body parts.
  • Instance mixing: in crowds, points from two people merge into one implausible skeleton.
  • Domain shift: models trained on everyday photos degrade on thermal, fisheye or underwater images.
  • Annotation noise: inconsistent labeling of points such as hips sets an upper bound on achievable accuracy.

Put simply

Keypoint detection teaches a computer to put pins on the important spots in a picture. For a person, the pins go on the shoulders, elbows, knees and other joints. For a face, they go on the corners of the eyes, the tip of the nose and the edges of the mouth.

Once those pins are in place, software can connect them like a stick figure and understand how something is positioned or moving, without having to analyze every pixel again.

In practice

A physiotherapy clinic uses a tablet app to check how patients perform squats during knee rehabilitation. The app runs a lightweight single-stage keypoint model that locates 17 body points at about 30 frames per second, then computes the knee and hip angles from the hip, knee and ankle coordinates.

Early tests show jitter in the knee angle of several degrees between frames and occasional left-right knee swaps when patients turn sideways. The developers add temporal smoothing, restrict the camera to a side view and fine-tune the model on 3,000 annotated clinic frames.

After these changes, the measured knee angle stays within a few degrees of goniometer readings, and therapists use the app to track range-of-motion progress between sessions instead of relying on visual estimates.