Semantic segmentation

On this page 7

Semantic segmentation is the computer vision task of assigning a class label to every pixel of an image, so that the output is a dense map in which each location is marked as road, person, sky, tumor tissue or any other category the model was trained to recognize. Unlike image classification, which gives one label per image, it answers what is present and exactly where, down to object boundaries.

Semantic segmentation treats all pixels of a class as one region: two adjacent people receive the same label and are not separated into individuals. That separation is the job of instance segmentation, and the combination of both is panoptic segmentation. Within artificial intelligence, the task is a foundation for autonomous driving, medical imaging, satellite analysis and robotics.

How semantic segmentation models produce pixel labels

A semantic segmentation model maps an image of size H by W to a tensor of per-pixel class scores with one channel per class, then takes the highest score at each location. Training typically minimizes pixel-wise cross-entropy loss, often combined with Dice or focal loss when some classes cover only a small share of the image.

Most architectures follow an encoder-decoder design. The encoder, usually a pretrained backbone, extracts increasingly abstract features while reducing spatial resolution, commonly by a factor of 8 to 32. The decoder then recovers spatial detail through upsampling, skip connections from early layers or dilated convolutions that enlarge the receptive field without losing resolution.

The central tension is between global context and fine localization. Deciding that a gray region is road rather than wall requires a wide view of the scene, while drawing a sharp border around a cyclist requires high-resolution detail. Each architecture family resolves this trade-off differently.

Main architecture families

The following table summarizes the architecture families that dominate practice and research, with the idea each introduced.

FamilyKey ideaTypical strength
Fully convolutional network (FCN)Replaces dense layers with convolutionsSimple end-to-end baseline
U-NetSymmetric decoder with skip connectionsSmall datasets, medical images
DeepLab familyDilated convolutions and pyramid poolingMulti-scale context in street scenes
Transformer segmentersSelf-attention over image patchesLong-range context, strong accuracy
Mask classification modelsPredict masks plus class per maskOne design for semantic and panoptic tasks
Lightweight real-time networksTwo-branch or reduced-width designsEmbedded and vehicle hardware
Architecture families for semantic segmentation

Recent foundation segmentation models can produce masks for almost any object from a prompt, but they output class-agnostic masks. Turning them into a semantic segmentation system still requires a labeling step that maps each mask to a defined category.

Semantic segmentation sits in a family of dense and localized prediction tasks, and choosing the wrong one is a frequent project mistake. The table contrasts it with its nearest neighbors.

TaskOutputSeparates individual objects
Image classificationOne label per imageNo
Object detectionBounding boxes with labelsYes, at box level
Semantic segmentationClass label per pixelNo
Instance segmentationPixel mask per counted objectYes, for countable objects
Panoptic segmentationClass and instance ID per pixelYes, plus amorphous regions
How semantic segmentation relates to other localization tasks

A useful rule follows from this comparison: if the application needs area, shape or drivable surface, semantic segmentation fits; if it needs to count or track individual objects, instance or panoptic segmentation is the better choice.

Metrics and datasets used for evaluation

The standard metric is mean intersection over union (mIoU): for each class, the overlap between predicted and true pixels is divided by their union, and the results are averaged across classes. Because every class counts equally, mIoU penalizes failures on rare classes that pixel accuracy would hide.

MetricWhat it measuresMain caveat
Pixel accuracyShare of correctly labeled pixelsDominated by large classes
Mean IoUAverage per-class overlapInsensitive to thin boundary errors
Dice coefficientOverlap weighted toward agreementCommon in medicine, close to IoU
Boundary F-scoreAccuracy of predicted contoursDepends on a distance tolerance
Common evaluation metrics

Widely used benchmarks include Cityscapes for urban driving with 19 evaluation classes, ADE20K with 150 scene categories and PASCAL VOC with 20 object classes plus background. Reported mIoU values differ strongly between datasets, so scores are comparable only within one benchmark and one evaluation protocol.

Evaluation details also shift results noticeably. Multi-scale and flipped test-time inference can add one to two mIoU points, and evaluating at reduced resolution flatters models that struggle with thin structures such as poles, fences and traffic lights. Reports should therefore state input resolution, test-time augmentation and class set together with the score.

Practical challenges and common mistakes

The largest practical cost is annotation: labeling a single dense street image by hand can take well over an hour, which drives interest in weak supervision, synthetic data and model-assisted labeling. Label quality at object borders strongly limits achievable accuracy.

SymptomLikely causeRemedy
Small classes vanishClass imbalance in pixelsWeighted or focal loss, oversampling
Blurry object bordersLow decoder resolutionSkip connections, boundary-aware loss
Good validation, poor field resultsDomain shift in lighting or sensorsDiverse data, domain adaptation
Too slow on deviceHeavy backbone at full resolutionLightweight model, lower input size
Inconsistent labels between framesPer-frame inference without memoryTemporal smoothing or video models
Typical problems, their causes and remedies

Another frequent mistake is resizing masks with smooth interpolation, which creates invalid in-between class values. Label maps must be resized with nearest-neighbor interpolation, and ignore indices for unlabeled pixels must be handled consistently in both loss and metric.

Explained without jargon

Semantic segmentation is like giving a computer a coloring book and asking it to color every part of a picture by category: all road gray, all people red, all trees green, all sky blue.

The computer does not just say that a photo contains a person. It shows exactly which pixels belong to that person. It does not tell two people apart, however; both simply become red. That precise outline of every region is what lets a machine judge where it can drive, where a tumor ends or how much of a field is flooded.

A typical use case

An agricultural cooperative uses drone flights to monitor 2,000 hectares of wheat. Images taken at a ground resolution of about 3 centimeters per pixel are processed by a U-Net style model trained on roughly 1,500 annotated tiles with four classes: healthy crop, weed patches, bare soil and waterlogged areas.

The resulting maps show where each problem lies and how large it is, so the sprayer applies herbicide only to weed patches instead of the whole field. After one season, the cooperative reports a substantial reduction in herbicide use on mapped fields, while agronomists review only flagged zones rather than walking every hectare.