Object detection is the computer vision task of locating every object of interest in an image and assigning each one a class label, a bounding box and a confidence score. It answers two questions at once, “what is in the image?” and “where exactly is it?”, which distinguishes it from image classification, which outputs a single label for the whole frame.
In artificial intelligence, object detection is a foundation for autonomous driving, robotics, retail analytics, medical imaging and video surveillance. Modern detectors are deep neural networks that predict many candidate boxes in parallel and then filter them, and their quality is judged by how well localization and classification succeed together.
How object detection models work
An object detector works by extracting visual features with a backbone network, combining features from several resolutions, and predicting box coordinates and class scores from those features. A typical architecture has three parts: a backbone such as a ResNet or a Vision Transformer, a neck such as a feature pyramid network that merges coarse and fine feature maps, and a detection head that outputs the predictions.
Most detectors produce far more candidate boxes than there are objects, often thousands per image. Non-maximum suppression (NMS) then removes duplicates: it keeps the highest-scoring box and discards any other box of the same class whose overlap with it exceeds a threshold, commonly an IoU of 0.5 to 0.7.
Overlap is measured by intersection over union (IoU), the area shared by two boxes divided by the area they cover together. Two 100 by 100 pixel boxes offset horizontally by 50 pixels share 5,000 square pixels and cover 15,000 together, giving an IoU of 0.33, which most benchmarks would not accept as a correct detection.
Detector families and their trade-offs
Object detectors fall into families that differ mainly in how they generate candidate boxes, which in turn sets the balance between accuracy and speed. The table summarizes the main families and representative models.
| Family | Representative models | How boxes are proposed | Typical trade-off |
|---|---|---|---|
| Two-stage | Faster R-CNN, Cascade R-CNN | Region proposal network, then refinement | High accuracy, slower inference |
| One-stage anchor-based | SSD, RetinaNet, earlier YOLO versions | Dense predictions over predefined anchors | Fast, sensitive to anchor design |
| One-stage anchor-free | FCOS, CenterNet, recent YOLO versions | Points or centers predict box extents | Fewer hyperparameters, strong speed |
| Transformer-based | DETR, Deformable DETR, RT-DETR | Learned object queries with set matching | No NMS needed, longer training |
| Open-vocabulary | Grounding DINO, OWL-ViT | Text prompts matched to region features | Detects unseen classes, heavier models |
One-stage detectors face a strong imbalance between background and object locations. RetinaNet addressed it with focal loss, which down-weights easy background examples. DETR removed NMS entirely by using Hungarian matching to assign each ground-truth object to exactly one prediction during training.
How detection quality is measured
Detection quality is measured mainly by mean average precision (mAP), which summarizes the precision-recall curve of each class and averages the result across classes. A prediction counts as a true positive only if its class is correct and its IoU with an unmatched ground-truth box meets the threshold.
| Metric | Definition | What it reveals |
|---|---|---|
| IoU | Overlap area divided by union area | Localization accuracy of a single box |
| AP50 | Average precision at IoU 0.5 | Whether objects are found at all |
| AP75 | Average precision at IoU 0.75 | Whether boxes are tight |
| COCO mAP | Mean AP over IoU 0.50 to 0.95 in steps of 0.05 | Combined detection and localization quality |
| APs, APm, APl | AP for objects under 32 squared, up to 96 squared, and above 96 squared pixels | Performance by object size |
| Latency and FPS | Time per image, frames per second | Suitability for real-time use |
The COCO metric averages over ten IoU thresholds and is therefore much stricter than the AP50 metric used by the older PASCAL VOC benchmark, so scores from the two protocols must not be compared directly. COCO covers 80 object categories, while PASCAL VOC covers 20.
Object detection among related vision tasks
Object detection sits between whole-image classification and pixel-level segmentation in the amount of spatial detail it outputs. Choosing the right task depends on whether the application needs counts and positions, exact shapes or structural keypoints.
| Task | Output | Typical application |
|---|---|---|
| Image classification | One label per image | Sorting photos by content |
| Object detection | Class, box and score per object | Counting vehicles, locating defects |
| Instance segmentation | Pixel mask per object | Measuring exact object shape or area |
| Pose estimation | Keypoints per person or object | Analyzing body posture or movement |
| OCR | Text regions and their characters | Reading labels, plates and documents |
Many systems chain these tasks. A license plate reader, for example, first runs object detection to find the plate, then applies OCR to the cropped region, and a tracking algorithm can link detections across video frames to follow each object over time.
Practical pitfalls in building detectors
Most failures in deployed detectors come from data and deployment conditions rather than architecture choice. Experienced teams check the following problems before changing the model.
- Small objects are lost when images are downscaled; higher input resolution or tiled inference often helps more than a larger backbone.
- Inconsistent annotations, such as loose boxes or missing labels, cap achievable mAP and teach the model to ignore real objects.
- Domain shift between training and deployment, for example daylight images versus night footage, causes sharp drops in recall.
- Confidence thresholds must be tuned per application, because the default trades precision against recall in a way that may not fit the cost of errors.
- Crowded scenes suffer when NMS suppresses genuinely overlapping objects, such as people standing close together.
For edge deployment, quantization to 8-bit integers and export to an optimized runtime commonly cut latency substantially, but the accuracy loss must be verified on the target data rather than assumed.
A plain-language summary
Object detection teaches a computer to look at a picture, find each thing it knows and draw a box around it with a name and a confidence level. It works much like a person scanning a street scene and pointing out “car here, cyclist there, traffic light up there”.
The key takeaway is that detection gives both identity and position. That combination lets machines count, track and react to objects, which a simple yes-or-no label for the whole image cannot do.
An example from practice
A logistics company wants to reduce collisions between forklifts and pedestrians in a warehouse. It installs ceiling cameras that stream 1280 by 720 video and trains a one-stage, anchor-free detector on about 12,000 labeled frames covering three classes: person, forklift and pallet. Frames from night shifts and from areas with strong backlight are deliberately included to limit domain shift.
Exported with 8-bit quantization, the model runs at about 30 frames per second on an edge GPU beside each camera. When a person box and a forklift box come within a calibrated floor distance, the system triggers a warning light. The team sets a low confidence threshold for the person class, accepting more false alarms in exchange for higher recall on pedestrians, because a missed person costs far more than an unnecessary warning. In the pilot area, recorded near-miss incidents fall noticeably over the first quarter of operation.