Segmentation mask

On this page 7

A segmentation mask is an image-sized array in which every pixel carries a label that says which class, object or region it belongs to. In computer vision, it is the output format of image segmentation models and the ground-truth format used to train them, giving a pixel-accurate outline of shapes that a bounding box can only approximate.

Masks appear in medical imaging, autonomous driving, photo editing, satellite analysis and robotics. Their exact encoding depends on the task: semantic segmentation labels classes, instance segmentation separates individual objects, and panoptic segmentation combines both in a single map.

How a segmentation mask encodes pixel labels

A segmentation mask encodes labels as integer or binary values aligned to the image grid, so the pixel at row r and column c in the mask describes the pixel at the same position in the image. The mask has the same height and width as the image, or a known scale factor relative to it.

Three encodings dominate practice. A binary mask stores 0 for background and 1 for the object and is common for single-object tasks. A class-index mask stores one integer per pixel, such as 0 for road and 7 for pedestrian. A one-hot or probability tensor stores one channel per class, which is the form a neural network outputs before an argmax turns it into a class-index mask.

The following table summarizes the common mask encodings and where each is used.

EncodingShape and valuesTypical use
Binary maskH x W, values 0 or 1One object, foreground extraction
Class-index maskH x W, integers 0 to K-1Semantic segmentation labels
Probability tensorK x H x W, floats 0 to 1Raw model output, soft masks
Instance ID mapH x W, unique ID per objectInstance and panoptic tasks
Run-length encodingCompressed pixel runsStorage in annotation files
Common mask encodings

Datasets rarely store masks as raw arrays. They use indexed PNG images, polygon coordinates that are rasterized on load, or run-length encoding (RLE), which the widely used COCO annotation format adopts for compact storage of irregular shapes.

Mask types across segmentation tasks

The task defines what a mask label means, so the same pixel array can carry very different information. A semantic mask cannot tell two adjacent cars apart, while an instance mask can, and that difference decides which downstream analysis is possible.

TaskWhat each label meansLimitation
Semantic segmentationClass of the pixelTouching objects of one class merge
Instance segmentationSpecific object, plus classBackground regions often unlabeled
Panoptic segmentationClass plus instance ID for every pixelMore complex annotation and metrics
Video object segmentationObject identity across framesErrors accumulate over time
Interactive segmentationObject chosen by user clicksDepends on prompt quality
Masks by segmentation task

Panoptic masks distinguish countable objects such as people and vehicles from amorphous regions such as sky, grass or road. Countable objects receive instance IDs, while amorphous regions receive only a class label, which gives a complete, non-overlapping description of the scene.

Segmentation masks compared with other annotations

A segmentation mask is the most spatially precise annotation among common computer vision labels, and also the most expensive to create. Choosing it over lighter formats is justified when exact shape, area or boundary matters to the application.

The comparison below places masks next to the neighboring annotation types.

AnnotationSpatial detailRelative labeling effort
Image-level labelNone, whole imageSeconds per image
Bounding boxRectangle around objectA few seconds per object
KeypointsSelected landmark pointsModerate, depends on point count
Polygon outlineVertices along the boundaryTens of seconds per object
Segmentation maskEvery pixel labeledHighest, minutes for complex scenes
Annotation formats compared

Because full masks are costly, teams often use weak supervision: boxes or scribbles are converted into approximate masks, or a promptable segmentation model proposes masks that annotators only correct. This can cut annotation time substantially, although the final quality still depends on human review of boundaries.

Measuring mask quality

Mask quality is measured by comparing predicted pixels with ground-truth pixels, most often through intersection over union (IoU), the area of overlap divided by the area of union. For a whole dataset, the per-class IoU values are averaged into mean IoU (mIoU), the standard semantic segmentation metric.

Other metrics capture aspects that IoU underweights. The Dice coefficient, equal to twice the overlap divided by the sum of both areas, is preferred in medical imaging. Boundary IoU focuses on pixels near edges, where most errors occur, and mask average precision extends mean average precision from boxes to masks for instance tasks. Panoptic tasks use panoptic quality (PQ), which combines recognition and segmentation accuracy.

A known pitfall is that large objects dominate pixel-level scores. A model can reach a high mIoU while missing thin structures such as poles, wires or blood vessels, so per-class and boundary metrics should be reported alongside the headline number.

Practical issues when working with masks

Most errors with segmentation masks come from data handling rather than model design. The table below lists frequent problems, their causes and the usual fixes.

SymptomLikely causeFix
Blurred or invalid labelsMask resized with bilinear interpolationUse nearest-neighbor resizing
Shifted masksAugmentation applied to image onlyApply identical transforms to both
Wrong class colorsPNG palette read as RGBLoad as indexed or single-channel
Ignored edges in trainingBoundary pixels marked as voidCheck the ignore index, often 255
Poor small-object resultsClass imbalance in pixel countsUse weighted or Dice loss
Common mask problems and fixes

Resolution is another trade-off. Many models predict masks at one quarter or one eighth of input resolution and upsample them, which saves memory but softens fine boundaries. Applications that need crisp edges, such as background removal, often add a refinement step or a separate matting model.

Explained without jargon

A segmentation mask is like a coloring-book page laid exactly over a photo. Every tiny dot of the picture gets colored according to what it shows: one color for the road, another for people, another for the sky.

A computer uses this colored overlay to know precisely where each thing is, down to its exact outline, instead of just drawing a rough box around it. That precision is what lets a phone blur the background behind a person or a car tell the sidewalk from the street.

A typical use case

A radiology team trains a model to measure liver tumors on CT scans. Specialists annotate about 300 scans by painting binary masks over each tumor slice by slice, and the model learns to output a mask of the same size for every new scan.

Because each mask pixel corresponds to a known physical size, for example 0.7 by 0.7 millimeters with a 2.5-millimeter slice thickness, the software counts mask pixels to compute tumor volume in milliliters. Validated with a Dice score of around 0.85 against expert annotations, the tool lets radiologists track whether a tumor shrinks between treatment cycles in seconds instead of manual measurement that takes several minutes per scan.