Panoptic segmentation

On this page 7

Panoptic segmentation is the computer vision task of assigning every pixel in an image both a semantic class and an instance identity where applicable, so that the output describes the whole scene without gaps or overlaps. It unifies semantic segmentation and instance segmentation into one coherent scene parsing problem.

The task, formalized in 2019 by Kirillov and colleagues, distinguishes things, which are countable objects such as people and cars, from amorphous background regions such as road, sky and grass. Within computer vision and artificial intelligence more broadly, it has become the reference formulation for complete scene understanding in driving, robotics and mapping.

How panoptic segmentation labels a scene

A panoptic model outputs, for each pixel, a pair consisting of a class label and an instance ID. Pixels of a background class share one segment per class, while each thing instance receives its own ID, and the rules require that no pixel belongs to two segments.

This non-overlap constraint is what separates panoptic output from a simple combination of detectors. Instance segmentation models may predict overlapping masks with confidence scores, but a panoptic result must resolve every conflict into a single, consistent partition of the image. Pixels the model cannot classify may be marked as void, and they are ignored in evaluation.

The table below shows how the three segmentation tasks differ in what they produce.

TaskLabels every pixelSeparates instancesHandles background classesOverlaps allowed
Semantic segmentationYesNoYesNo
Instance segmentationNoYesNoYes
Panoptic segmentationYesYesYesNo
Panoptic segmentation compared with related segmentation tasks

In practice, the output is stored as a single segment ID map plus a list of segment descriptions, which is the format used by the COCO panoptic annotations.

Model architectures for panoptic segmentation

Panoptic architectures evolved from two-branch designs with heuristic merging toward unified mask classification models. Early systems ran a semantic head and an instance head on a shared backbone and then fused the outputs with hand-written rules.

Modern transformer-based models instead predict a fixed set of mask and class pairs through learned queries, which removes most post-processing and handles objects and background regions in the same way.

FamilyRepresentative modelsCore ideaTrade-off
Two-branch mergingPanoptic FPN, UPSNetSeparate semantic and instance heads fused by rulesSimple but heuristic conflict resolution
Bottom-up groupingPanoptic-DeepLabPredicts centers and offsets to group pixelsFast, weaker on small or crowded objects
Query-based mask classificationDETR, MaskFormer, Mask2FormerLearned queries each predict one mask and classStrong accuracy, heavier training
Universal and open-vocabularyOneFormer, ODISE-style modelsOne model for several tasks or unseen classesHigher compute, evolving benchmarks
Main families of panoptic segmentation models

Query-based models use a Hungarian matching loss that pairs predictions with ground-truth segments during training, similar to the set prediction used in object detection with transformers. At inference, low-confidence queries are discarded and the remaining masks are assigned to pixels by highest score per pixel.

Measuring quality with panoptic quality

Panoptic segmentation is evaluated with panoptic quality (PQ), a metric that combines recognition and mask accuracy in one number. A predicted segment matches a ground-truth segment only when their intersection over union exceeds 0.5, which guarantees that each matching is unique.

PQ equals the sum of IoU over matched pairs divided by the number of true positives plus half of the false positives and false negatives. It factorizes into segmentation quality (SQ), the average IoU of matched segments, and recognition quality (RQ), an F1-style detection score.

MetricWhat it measuresTypical use
PQOverall quality across all classesHeadline benchmark number
SQMask precision of correctly found segmentsDiagnosing boundary quality
RQDetection accuracy of segmentsDiagnosing missed or spurious segments
PQ thingsQuality on countable objectsComparing instance handling
PQ backgroundQuality on amorphous regionsComparing background labeling
Panoptic metrics and what each one reveals

On the COCO panoptic benchmark, strong current models report PQ values in the mid-50s to around 60, while results on Cityscapes are typically higher because the scenes and class sets are more constrained. Scores are only comparable when the backbone, input resolution and training data match.

Datasets, annotation and practical deployment

Training a panoptic model requires dense, non-overlapping annotations, which are among the most expensive labels in computer vision. Annotators must outline every object and every background region, and a single urban street image can take more than an hour to label completely.

DatasetDomainScaleNotable feature
COCO panopticEveryday scenesAbout 133 classesStandard general benchmark
CityscapesUrban driving19 evaluated classesHigh-resolution street images
Mapillary VistasStreet-level imagery worldwide65 classes in the common versionDiverse cameras and countries
ADE20KIndoor and outdoor scenes150 classes in the panoptic setupMany background categories
Widely used panoptic segmentation datasets

For deployment, teams choose between accuracy and latency. Query-based models with large backbones suit offline mapping, while real-time robots and vehicles often use lighter bottom-up or distilled models running at 20 to 30 frames per second on embedded accelerators.

Limitations and common pitfalls

Panoptic segmentation is powerful but sensitive to class definitions and annotation policy. Whether an object counts as a thing or as a background region is a dataset decision, and the same model can look strong or weak depending on that split.

  • Crowded scenes such as dense crowds or parked car rows cause merged instances and lower RQ.
  • Small objects are often absorbed into neighboring background regions at low input resolutions.
  • Domain shift between training cities, weather or sensors can reduce PQ sharply without obvious warning signs.
  • Closed vocabularies mean unseen objects are forced into a known class or into void.
  • Temporal flicker appears when frame-by-frame models assign different IDs to the same object; video panoptic segmentation addresses this.

The idea in brief

Imagine coloring a photograph so that every single spot is filled in. The sky gets one color, the road another and the grass a third. Each person and each car, however, gets its own individual color, so that two people standing side by side remain clearly separate.

Panoptic segmentation is a computer doing exactly that. It tells you both what each part of the picture is and, for things you can count, which one it is. The result is a complete map of the scene with nothing left out and nothing counted twice.

A practical scenario

A municipal transport department surveys sidewalk accessibility using a vehicle-mounted camera that captures street imagery every few meters. The team trains a query-based panoptic model on a street-level dataset, then fine-tunes it on 2,000 locally annotated images that include curb ramps, bollards and outdoor café furniture.

The model labels sidewalks, roads and vegetation as background regions while counting each parked scooter, pole and obstruction as a separate instance. By measuring how much sidewalk width remains free of instances, the system flags locations where the clear path drops below 1.2 meters. Across 300 kilometers of streets, it identifies about 1,400 candidate obstructions, and field inspectors confirm roughly 85 percent of the flagged sites, which lets the department prioritize repairs within weeks instead of months.