Instance segmentation

On this page 7

Instance segmentation is the computer vision task of detecting every object of interest in an image and assigning each one its own pixel-accurate mask, so that two touching cars or three overlapping people are kept apart as separate instances. It combines the localization of object detection with the per-pixel precision of segmentation.

The output for each detected object is a class label, a confidence score and a binary mask, often with a bounding box as well. This makes instance segmentation the method of choice whenever a system must count, measure or manipulate individual objects rather than only know which regions belong to which category.

How instance segmentation separates individual objects

An instance segmentation model must solve two problems at once: find each object and delineate its exact shape. Most architectures handle the first with a detection stage and the second with a mask prediction stage that operates on each candidate object.

The dominant two-stage design, popularized by Mask R-CNN, uses a backbone network and a feature pyramid to extract features, a region proposal network to suggest candidate boxes, and a small fully convolutional head that predicts a 28 by 28 mask for each region. The RoIAlign operation samples features with bilinear interpolation, which preserves spatial alignment and noticeably improves mask accuracy over earlier pooling methods.

One-stage and query-based models remove the proposal step. YOLACT combines shared prototype masks with per-instance coefficients, SOLO assigns instances to grid cells by location, and transformer models such as Mask2Former use learned object queries with masked attention, predicting a fixed set of masks that are matched to ground truth during training.

FamilyRepresentative modelsStrengthsWeaknesses
Two-stage, box-firstMask R-CNN, Cascade Mask R-CNNMature, accurate, well understoodCoarse mask resolution, slower
One-stage, prototypeYOLACT, YOLO segmentation variantsReal-time speedWeaker on small or thin objects
Location-basedSOLO, SOLOv2No boxes or proposals neededStruggles with crowded centers
Query-based transformerMask2Former, Mask DINOHigh accuracy, unified designHeavy compute and memory
Main architecture families for instance segmentation

Instance segmentation compared with other segmentation tasks

Instance segmentation differs from its neighbors in what it distinguishes and what it ignores. Semantic segmentation labels every pixel with a class but merges all objects of the same class, while panoptic segmentation combines both views into one complete scene description.

TaskOutputSeparates instancesCovers background
Object detectionBoxes with labelsYesNo
Semantic segmentationOne class per pixelNoYes
Instance segmentationOne mask per objectYesNo
Panoptic segmentationClass and instance ID per pixelYesYes
How instance segmentation compares with related tasks

The usual terminology divides scenes into countable objects, such as people or bottles, and amorphous background regions, such as sky, road or grass. Instance segmentation focuses on countable objects; it normally does not produce masks for background regions, and its masks may overlap because each instance is predicted independently.

Evaluation metrics and datasets

Instance segmentation is evaluated mainly with mask average precision, which is computed like detection mean average precision but uses mask intersection over union instead of box overlap. A predicted mask counts as correct only when its pixel overlap with a ground-truth mask exceeds a threshold.

MetricWhat it measuresTypical use
Mask AP (0.50:0.95)Average precision over IoU 0.50 to 0.95Primary COCO benchmark score
AP50 and AP75Precision at loose and strict IoUSeparating detection from mask quality
APs, APm, APlAP for small, medium and large objectsDiagnosing scale weaknesses
Boundary APOverlap measured near object contoursJudging edge sharpness
Common metrics for instance segmentation

Standard benchmarks include COCO with 80 thing categories, LVIS with more than 1,000 long-tail categories, and Cityscapes for street scenes. Strong models reach roughly 45 to 55 mask AP on COCO, and the gap between AP50 and AP75 shows how much accuracy is lost on precise boundaries.

Choosing an approach for a real application

The right instance segmentation approach depends on latency budget, object size and labeling resources more than on leaderboard rank. A model that wins on COCO may be a poor fit for a 30 frames per second video stream on an embedded device.

SituationRecommended choiceReason
Real-time video on edge hardwareOne-stage prototype modelLow latency with acceptable masks
Medical or industrial inspectionTwo-stage or transformer with refinementBoundary accuracy matters most
Few labeled imagesPromptable foundation model plus fine-tuningReduces annotation effort
Crowded scenes with occlusionQuery-based transformerHandles overlapping instances better
Need for full scene coveragePanoptic segmentation modelAdds background region classes
Decision guide for common scenarios

Promptable foundation models such as Segment Anything have changed annotation workflows. They produce class-agnostic masks from clicks or boxes, so teams often pair them with a detector or use them to pre-label datasets that human annotators then correct.

Limitations and common failure modes

Instance segmentation remains expensive to label and sensitive to occlusion. Polygon or brush annotation of a single image with dozens of objects can take an annotator many minutes, and inconsistent labeling of boundaries directly limits achievable accuracy.

  • Occlusion and crowding: overlapping objects are merged into one mask or split into fragments.
  • Small and thin objects: low-resolution mask heads blur cables, poles and distant people.
  • Duplicate predictions: weak non-maximum suppression leaves several masks on one object.
  • Domain shift: models trained on everyday photos degrade on microscopy, aerial or thermal imagery.
  • Class imbalance: rare categories in long-tail datasets receive far lower precision.

In plain words

Instance segmentation teaches a computer to outline each object separately. Imagine giving someone a photo of a fruit bowl and a set of colored pencils. Simply shading all the apples red would be one kind of task. Instance segmentation asks for more: trace the exact edge of every single apple and give each one its own color, even when they touch.

Because each object gets its own outline, the computer can count them, measure their size or pick one up with a robot arm. The key idea is separating individuals, not just recognizing types.

A real-world example

A vegetable packing plant installs a camera above a conveyor belt that carries about 12 potatoes per second. The quality team needs each potato’s area and visible defects to sort them into three grades, but potatoes often touch or overlap, so bounding boxes alone would misjudge size.

Engineers fine-tune a one-stage instance segmentation model on 2,500 annotated frames, many pre-labeled with a promptable segmentation model and corrected by staff. The deployed system runs at about 40 frames per second on an industrial GPU and reaches 0.71 mask AP on held-out frames. Because each potato has its own mask, the plant computes size per tuber, and misgraded potatoes fall by more than half compared with the previous box-based system.