Non-maximum suppression

Also known as: NMS

On this page 6

Non-maximum suppression (NMS) is a post-processing algorithm that removes redundant, overlapping predictions so that each detected object is represented by a single result. In computer vision, it is best known as the final step of object detection, where a detector typically proposes many bounding boxes around the same object and NMS keeps the highest-scoring box while discarding neighbors that overlap it too much.

The same idea appears earlier in image processing, for example in edge and corner detection, where NMS thins responses so that only local maxima survive. This page focuses on box-level NMS for detectors, the most common use today.

How non-maximum suppression selects boxes

Greedy NMS works on a list of candidate boxes, each with a confidence score and a class label. The algorithm first drops boxes below a score threshold, then sorts the remainder by score in descending order and processes them one by one.

  1. Select the box with the highest remaining score and add it to the output.
  2. Compute its intersection over union (IoU) with every remaining box of the same class.
  3. Remove each box whose IoU exceeds the IoU threshold, commonly between 0.45 and 0.7.
  4. Repeat until no candidates remain, then optionally cap the output at a maximum number of detections.

IoU is the area of overlap divided by the area of the union of two boxes, so it ranges from 0 for disjoint boxes to 1 for identical ones. Because each box is compared against the kept boxes, the naive implementation has quadratic complexity in the number of candidates, which is why detectors usually keep only the top few hundred or thousand boxes before running NMS.

The parameters below control most of the behavior and are worth tuning for each dataset.

ParameterTypical rangeEffect of raising it
IoU threshold0.45 to 0.7Keeps more overlapping boxes, more duplicates
Score threshold0.001 to 0.5Fewer weak detections, lower recall
Maximum detections100 to 300 per imageMore results, slower downstream steps
Pre-NMS top-k1,000 to 5,000 candidatesBetter recall, higher compute cost
Main NMS parameters and their effects

Variants of non-maximum suppression

Greedy NMS makes a hard keep-or-delete decision, which is fast but loses information. Several variants change the suppression rule, the scope or the overlap measure to address specific weaknesses.

VariantKey ideaBest suited to
Greedy NMSDelete boxes above the IoU thresholdGeneral detection, fast inference
Soft-NMSDecay scores of overlapping boxes instead of deletingCrowded scenes with occlusion
Class-agnostic NMSSuppress across all classes at onceClasses that are easily confused
DIoU-NMSAdd center distance to the overlap measureAdjacent objects with similar boxes
Weighted box fusionAverage overlapping boxes by confidenceEnsembles and test-time augmentation
Matrix NMSParallel score decay in one matrix operationInstance segmentation on GPUs
Common NMS variants and when to use them

Soft-NMS multiplies the scores of overlapping boxes by a linear or Gaussian penalty rather than removing them, which often improves mean average precision on crowded benchmarks at almost no extra cost. Batched NMS is an implementation trick rather than a new rule: it offsets box coordinates by class so that per-class suppression runs in a single GPU call.

The choice of variant should follow the data. Scenes with heavy occlusion, such as pedestrians or shelved products, benefit most from score decay, while ensemble pipelines gain more from fusing boxes than from deleting them. For real-time systems on edge hardware, plain greedy NMS with a tight candidate limit often remains the most practical option.

Non-maximum suppression in the detection pipeline

NMS sits between the raw detector output and the final predictions. Two-stage detectors such as R-CNN variants apply it twice: once to proposals from the region proposal network and once to the classified boxes. One-stage detectors such as YOLO apply it once, after decoding predictions from anchor boxes or anchor-free grids.

Some modern architectures are designed to avoid NMS entirely. The comparison below shows how the main detector families handle duplicate predictions.

Detector familyRole of NMSConsequence
Two-stage (R-CNN family)Applied to proposals and final boxesAccurate but more latency
One-stage (YOLO, SSD)Single pass after predictionFast, sensitive to thresholds
DETR-style transformersNot requiredOne-to-one matching learned in training
NMS-free one-stage designsRemoved by consistent label assignmentLower and more predictable latency
How detector families handle duplicates

Transformer detectors achieve this through bipartite matching during training, which teaches each query to predict a distinct object. The trade-off is usually slower training convergence in exchange for a simpler and more deterministic inference step.

Limitations and tuning mistakes

The central weakness of NMS is that a single fixed threshold cannot fit every scene. A low threshold deletes genuine neighbors in a crowd, while a high threshold leaves duplicates on isolated objects. NMS is also not differentiable in its standard form, so the detector cannot learn to cooperate with it during training.

The table below lists typical symptoms seen in deployed detectors, with their usual cause and fix.

SymptomLikely causeTypical fix
Missed people in crowdsIoU threshold too lowRaise threshold or use Soft-NMS
Duplicate boxes on one objectIoU threshold too highLower threshold to about 0.5
Two labels on one objectPer-class NMS onlyAdd class-agnostic NMS step
Variable latency spikesToo many candidates in busy framesLimit pre-NMS top-k
Poor accuracy on rotated objectsAxis-aligned IoU usedUse rotated or polygon IoU
Common NMS problems and remedies

A frequent evaluation mistake is to tune thresholds on the test set instead of a validation split, which inflates reported mean average precision. Another is to compare models that were evaluated with different NMS settings, since the threshold alone can shift scores noticeably.

Put simply

A detection model rarely draws just one box around a car. It often draws ten slightly different boxes around the same car. Non-maximum suppression is the cleanup step. It keeps the box the model is most sure about. Then it erases the other boxes that cover almost the same area.

Think of several people pointing at the same bird. NMS listens to the most confident pointer and ignores the rest. The main challenge is deciding how much overlap counts as the same object.

In practice

A retail analytics team runs a people-counting camera at a busy store entrance. Their YOLO-based detector produces about 40 candidate boxes per person before post-processing. With greedy NMS at an IoU threshold of 0.45, the system counted single shoppers correctly but undercounted groups walking shoulder to shoulder, because boxes of neighboring people overlapped by more than 45 percent and were deleted.

The engineers evaluated alternatives on a week of labeled footage. Raising the threshold to 0.6 created duplicates on lone shoppers, so they switched to Gaussian Soft-NMS and kept the score threshold at 0.3. The result was a count error below 3 percent during peak hours, with inference time per frame almost unchanged.