Anchor box

On this page 7

An anchor box is a predefined reference rectangle of fixed size and aspect ratio, placed at regular positions over an image, which an object detector refines into a final bounding box prediction. In computer vision, anchor boxes turn the open-ended problem of locating objects into a classification and regression task over a fixed set of candidates.

Anchor boxes became central to detectors such as Faster R-CNN, SSD, RetinaNet and YOLO versions 2 to 5. Instead of predicting absolute coordinates from scratch, the network predicts small offsets relative to each anchor, which makes training more stable and allows a single model to handle objects of very different scales.

How anchor boxes guide object detection

A detector produces one or more feature maps through a convolutional backbone. At every cell of each feature map, it places a set of anchors that share the same center but differ in scale and aspect ratio. A feature map of 50 by 50 cells with 9 anchors per cell already yields 22,500 candidate boxes, and multi-scale detectors often generate tens of thousands more.

For every anchor, the detection head predicts two things: a class score (or objectness score) and four regression offsets. The offsets are usually encoded as shifts of the center, scaled by the anchor size, and as logarithmic scale factors for width and height. Decoding these offsets transforms the anchor into a predicted box, and non-maximum suppression then removes overlapping duplicates.

The standard parameterization is summarized below, where x, y, w and h describe the anchor and the predicted values are marked with t.

Predicted valueFormula for decodingWhat it controls
t_xx_pred = x_anchor + t_x × w_anchorHorizontal shift of the center
t_yy_pred = y_anchor + t_y × h_anchorVertical shift of the center
t_ww_pred = w_anchor × exp(t_w)Width scaling
t_hh_pred = h_anchor × exp(t_h)Height scaling
Box offset encoding relative to an anchor

Scales, aspect ratios and label assignment

The anchor set is defined by scales and aspect ratios. Faster R-CNN used three scales (areas of 128², 256² and 512² pixels) and three ratios (1:2, 1:1 and 2:1), giving nine anchors per location. Feature pyramid networks instead assign each scale to a different pyramid level, so small anchors sit on high-resolution maps and large anchors on coarse maps.

During training, each anchor is labeled by comparing it with the ground-truth boxes using intersection over union (IoU). The thresholds strongly influence how many positive examples the model sees and how it balances precision against recall.

Anchor IoU with ground truthAssigned labelEffect on training
At or above 0.7 (RPN) or 0.5 (RetinaNet)Positive, matched to that objectContributes to class and box loss
Below 0.3 (RPN) or 0.4 (RetinaNet)Negative, backgroundContributes to classification loss only
Between the thresholdsIgnoredExcluded to avoid ambiguous signals
Highest IoU for an objectForced positiveEnsures every object has at least one anchor
Typical IoU-based assignment rules

Because most anchors cover background, detectors face a severe class imbalance, often thousands of negatives per positive. Two-stage detectors handle this with sampling, while RetinaNet introduced focal loss to down-weight easy negatives.

Anchor-based and anchor-free detectors compared

Anchor boxes are one design choice among several for generating detection candidates. Since about 2019, anchor-free detectors such as FCOS, CenterNet and YOLOv8 predict boxes directly from points or centers, and transformer models such as DETR use learned object queries instead.

ApproachHow candidates are formedMain trade-off
Anchor-basedFixed boxes at every feature map locationStable training, but many hyperparameters
Anchor-free, point-basedDistances from a point to the four box edgesFewer settings, needs careful center sampling
Anchor-free, center-basedHeatmap peaks plus width and heightSimple decoding, weaker on overlapping objects
Query-based (DETR)Learned queries matched one-to-one to objectsNo anchors or NMS, slower convergence
Main candidate-generation approaches

Anchor boxes are closely tied to the region proposal network, which uses them to propose candidate regions for a second classification stage, and to the broader idea of a bounding box as the output format. They should not be confused with the final predictions: an anchor is a prior, while a bounding box is the refined result.

Designing anchor sets for a dataset

Default anchors are tuned for datasets such as COCO and can perform poorly on domain-specific data like aerial imagery, documents or medical scans. A practical rule is that every ground-truth box should overlap at least one anchor with an IoU above the positive threshold; otherwise the object can never be learned properly.

YOLOv2 popularized k-means clustering of training boxes with 1 minus IoU as the distance metric, which produces anchors that match the actual shape distribution. Later YOLO versions added automatic anchor evolution that checks recall before training and adjusts anchors when coverage is low.

SituationRecommended choiceReason
Many tiny objectsSmaller anchors on high-resolution levelsCoarse levels miss small targets
Elongated objects such as text linesExtreme ratios like 1:5 or 1:10Square anchors yield low IoU
Custom dataset with unusual shapesK-means clustered anchorsMatches real box distribution
Rotated objects in aerial imagesRotated anchors or anchor-free headsAxis-aligned boxes fit poorly
Limited tuning timeAnchor-free detectorRemoves anchor hyperparameters
Anchor design decisions by situation

Limitations of anchor boxes

Anchors introduce sensitive hyperparameters: the number of scales, ratios and IoU thresholds all affect accuracy, and a poor choice can reduce mean average precision by several points. They also create heavy computational load, since tens of thousands of boxes must be scored, decoded and filtered.

  • Scale mismatch: objects far outside the anchor range are rarely matched as positives.
  • Imbalance: the overwhelming number of background anchors slows training without special losses.
  • Redundant predictions: neighboring anchors detect the same object and depend on non-maximum suppression to clean up.
  • Poor transfer: anchors tuned for one dataset may generalize badly to another camera, resolution or domain.

The simple explanation

An anchor box is a ready-made guess about where an object might be and what shape it might have. Imagine laying a grid of transparent stencils of different sizes over a photo, some tall, some wide, some square. The computer checks each stencil. Is there something here? How should the stencil be stretched or moved to fit it?

Starting from these guesses is easier than drawing every box from nothing. That is why anchor boxes helped detectors learn faster. They also help one model handle both small and large objects.

How it is used in practice

A logistics company trains an object detector to find parcels and shipping labels on a conveyor belt filmed by overhead cameras at 1280 by 720 pixels. The first model, using default COCO anchors, misses many labels because they are long, thin rectangles with aspect ratios around 1:6, which square and moderately wide anchors match poorly.

The engineers run k-means clustering on 8,000 annotated boxes and replace the defaults with nine dataset-specific anchors, including several very wide shapes. A coverage check shows that the share of labels matched by at least one anchor above 0.5 IoU rises from roughly 70 percent to above 95 percent.

After retraining, label recall improves noticeably and the model no longer needs a lower confidence threshold to catch thin labels. The outcome is fewer missed scans on the sorting line without changing the network architecture, only the anchor priors it starts from.