An anchor box is a predefined reference rectangle of fixed size and aspect ratio, placed at regular positions over an image, which an object detector refines into a final bounding box prediction. In computer vision, anchor boxes turn the open-ended problem of locating objects into a classification and regression task over a fixed set of candidates.
Anchor boxes became central to detectors such as Faster R-CNN, SSD, RetinaNet and YOLO versions 2 to 5. Instead of predicting absolute coordinates from scratch, the network predicts small offsets relative to each anchor, which makes training more stable and allows a single model to handle objects of very different scales.
How anchor boxes guide object detection
A detector produces one or more feature maps through a convolutional backbone. At every cell of each feature map, it places a set of anchors that share the same center but differ in scale and aspect ratio. A feature map of 50 by 50 cells with 9 anchors per cell already yields 22,500 candidate boxes, and multi-scale detectors often generate tens of thousands more.
For every anchor, the detection head predicts two things: a class score (or objectness score) and four regression offsets. The offsets are usually encoded as shifts of the center, scaled by the anchor size, and as logarithmic scale factors for width and height. Decoding these offsets transforms the anchor into a predicted box, and non-maximum suppression then removes overlapping duplicates.
The standard parameterization is summarized below, where x, y, w and h describe the anchor and the predicted values are marked with t.
| Predicted value | Formula for decoding | What it controls |
|---|---|---|
| t_x | x_pred = x_anchor + t_x × w_anchor | Horizontal shift of the center |
| t_y | y_pred = y_anchor + t_y × h_anchor | Vertical shift of the center |
| t_w | w_pred = w_anchor × exp(t_w) | Width scaling |
| t_h | h_pred = h_anchor × exp(t_h) | Height scaling |
Scales, aspect ratios and label assignment
The anchor set is defined by scales and aspect ratios. Faster R-CNN used three scales (areas of 128², 256² and 512² pixels) and three ratios (1:2, 1:1 and 2:1), giving nine anchors per location. Feature pyramid networks instead assign each scale to a different pyramid level, so small anchors sit on high-resolution maps and large anchors on coarse maps.
During training, each anchor is labeled by comparing it with the ground-truth boxes using intersection over union (IoU). The thresholds strongly influence how many positive examples the model sees and how it balances precision against recall.
| Anchor IoU with ground truth | Assigned label | Effect on training |
|---|---|---|
| At or above 0.7 (RPN) or 0.5 (RetinaNet) | Positive, matched to that object | Contributes to class and box loss |
| Below 0.3 (RPN) or 0.4 (RetinaNet) | Negative, background | Contributes to classification loss only |
| Between the thresholds | Ignored | Excluded to avoid ambiguous signals |
| Highest IoU for an object | Forced positive | Ensures every object has at least one anchor |
Because most anchors cover background, detectors face a severe class imbalance, often thousands of negatives per positive. Two-stage detectors handle this with sampling, while RetinaNet introduced focal loss to down-weight easy negatives.
Anchor-based and anchor-free detectors compared
Anchor boxes are one design choice among several for generating detection candidates. Since about 2019, anchor-free detectors such as FCOS, CenterNet and YOLOv8 predict boxes directly from points or centers, and transformer models such as DETR use learned object queries instead.
| Approach | How candidates are formed | Main trade-off |
|---|---|---|
| Anchor-based | Fixed boxes at every feature map location | Stable training, but many hyperparameters |
| Anchor-free, point-based | Distances from a point to the four box edges | Fewer settings, needs careful center sampling |
| Anchor-free, center-based | Heatmap peaks plus width and height | Simple decoding, weaker on overlapping objects |
| Query-based (DETR) | Learned queries matched one-to-one to objects | No anchors or NMS, slower convergence |
Anchor boxes are closely tied to the region proposal network, which uses them to propose candidate regions for a second classification stage, and to the broader idea of a bounding box as the output format. They should not be confused with the final predictions: an anchor is a prior, while a bounding box is the refined result.
Designing anchor sets for a dataset
Default anchors are tuned for datasets such as COCO and can perform poorly on domain-specific data like aerial imagery, documents or medical scans. A practical rule is that every ground-truth box should overlap at least one anchor with an IoU above the positive threshold; otherwise the object can never be learned properly.
YOLOv2 popularized k-means clustering of training boxes with 1 minus IoU as the distance metric, which produces anchors that match the actual shape distribution. Later YOLO versions added automatic anchor evolution that checks recall before training and adjusts anchors when coverage is low.
| Situation | Recommended choice | Reason |
|---|---|---|
| Many tiny objects | Smaller anchors on high-resolution levels | Coarse levels miss small targets |
| Elongated objects such as text lines | Extreme ratios like 1:5 or 1:10 | Square anchors yield low IoU |
| Custom dataset with unusual shapes | K-means clustered anchors | Matches real box distribution |
| Rotated objects in aerial images | Rotated anchors or anchor-free heads | Axis-aligned boxes fit poorly |
| Limited tuning time | Anchor-free detector | Removes anchor hyperparameters |
Limitations of anchor boxes
Anchors introduce sensitive hyperparameters: the number of scales, ratios and IoU thresholds all affect accuracy, and a poor choice can reduce mean average precision by several points. They also create heavy computational load, since tens of thousands of boxes must be scored, decoded and filtered.
- Scale mismatch: objects far outside the anchor range are rarely matched as positives.
- Imbalance: the overwhelming number of background anchors slows training without special losses.
- Redundant predictions: neighboring anchors detect the same object and depend on non-maximum suppression to clean up.
- Poor transfer: anchors tuned for one dataset may generalize badly to another camera, resolution or domain.
The simple explanation
An anchor box is a ready-made guess about where an object might be and what shape it might have. Imagine laying a grid of transparent stencils of different sizes over a photo, some tall, some wide, some square. The computer checks each stencil. Is there something here? How should the stencil be stretched or moved to fit it?
Starting from these guesses is easier than drawing every box from nothing. That is why anchor boxes helped detectors learn faster. They also help one model handle both small and large objects.
How it is used in practice
A logistics company trains an object detector to find parcels and shipping labels on a conveyor belt filmed by overhead cameras at 1280 by 720 pixels. The first model, using default COCO anchors, misses many labels because they are long, thin rectangles with aspect ratios around 1:6, which square and moderately wide anchors match poorly.
The engineers run k-means clustering on 8,000 annotated boxes and replace the defaults with nine dataset-specific anchors, including several very wide shapes. A coverage check shows that the share of labels matched by at least one anchor above 0.5 IoU rises from roughly 70 percent to above 95 percent.
After retraining, label recall improves noticeably and the model no longer needs a lower confidence threshold to catch thin labels. The outcome is fewer missed scans on the sorting line without changing the network architecture, only the anchor priors it starts from.