Video object segmentation

Also known as: VOS

On this page 7

Video object segmentation (VOS) is the computer vision task of assigning every pixel in every frame of a video to one or more target objects or to the background, so that each object is outlined by a pixel-accurate mask that follows it through time. It extends image segmentation from single pictures to sequences and adds the problem of temporal consistency: the same object must keep the same identity as it moves, deforms, becomes occluded and reappears.

In computer vision, VOS sits between object tracking, which follows objects with boxes, and per-frame instance segmentation, which outlines objects without linking them across frames. It powers video editing, robotics, autonomous driving perception and medical video analysis.

How video object segmentation works

A VOS system works by propagating object information from frame to frame while correcting it with the appearance of each new frame. Modern systems encode each frame with a convolutional or transformer backbone, compare its features with features of earlier frames and decode a mask for every target object.

The dominant design since about 2019 is the memory-based matching network. It stores feature keys and mask values from past frames in a memory bank, and for each new frame it computes attention between current pixels and memory pixels. Each pixel then inherits the object label of the memory pixels it most resembles. This approach handles fast motion and occlusion better than earlier methods that relied on optical flow alone or on fine-tuning a network on the first frame at test time.

The main architectural families differ in how they carry information across time, as the following table shows.

ApproachCore ideaTypical speedMain weakness
Online fine-tuningRetrains a network on the annotated first frameSeconds per frameToo slow for interactive use
Flow-based propagationWarps the previous mask with optical flowModerateErrors accumulate through occlusion
Memory-based matchingAttends from current pixels to stored past framesReal time on a modern GPUMemory grows with video length
Query-based transformersLearns object queries that persist across framesModerate to fastNeeds large training sets
Promptable foundation modelsSegments any object from clicks, boxes or masksNear real timeHeavy compute and model size
Architectural approaches to VOS

Problem settings and variants

VOS is not one task but a family of settings that differ in how the target is specified. The choice of setting determines which models, datasets and metrics apply.

SettingHow targets are givenTypical use
Semi-supervised VOSMask of each object in the first frameVideo editing and rotoscoping
Unsupervised VOSNo input; the model finds salient moving objectsAutomatic video summarization
Interactive VOSUser clicks or scribbles, refined over roundsAnnotation tools and post-production
Referring VOSA natural-language description of the objectText-driven video search and editing
Video instance segmentationNo input; all objects of known classesDriving and surveillance perception
VOS problem settings

The semi-supervised setting is the most studied and is sometimes simply called VOS. The name is historical: the model is not trained in a semi-supervised way, but it receives one supervised mask at inference time and must propagate it through the rest of the video.

VOS compared with neighboring tasks

Video object segmentation differs from related computer vision tasks in output granularity and temporal linkage. The comparison below clarifies where VOS fits and why it is harder than its neighbors.

TaskOutput per frameIdentity across frames
Video object segmentationPixel mask per target objectYes, consistent object identity
Object trackingBounding box per objectYes, consistent track identity
Instance segmentationPixel mask per detected objectNo, each frame is independent
Semantic segmentationClass label per pixelNo, and no separate instances
Background subtractionForeground versus background pixelsNo object identities
VOS and related computer vision tasks

In practice the boundaries blur. Many trackers now output masks, and VOS models are often combined with an object detector that proposes targets in the first frame, which turns a semi-supervised method into a fully automatic pipeline.

Benchmarks and evaluation metrics

VOS is evaluated with region accuracy and boundary accuracy, usually averaged into a single score. The standard benchmarks include DAVIS, which offers short, densely annotated clips, and YouTube-VOS, which covers many more object categories and tests generalization to unseen classes. Newer sets such as MOSE stress heavy occlusion and crowded scenes, and LVOS tests long videos lasting minutes.

  • Region similarity (J): the intersection over union between predicted and ground-truth masks.
  • Contour accuracy (F): an F-measure on boundary pixels, which rewards precise object outlines.
  • J&F mean: the average of both scores and the headline number on most leaderboards.
  • Frames per second: throughput on a stated GPU, essential for interactive and real-time use.

Leading methods score far higher on short, clean clips than on long or crowded videos, so a single benchmark number overstates real-world robustness. Results also depend on input resolution and on whether the model was pretrained on extra image data.

Practical challenges and design choices

Deployed VOS systems fail in predictable situations, and engineers design around them with memory management, re-detection and human correction.

  • Occlusion and reappearance: an object hidden for many frames may be relabeled or lost; long-term memory and re-identification cues reduce this.
  • Similar distractors: identical objects such as players in the same uniform can swap identities; adding spatial priors helps.
  • Drift: small mask errors propagate and grow, so many tools let users correct a frame and re-propagate.
  • Memory cost: storing every frame is infeasible for long videos, so systems keep selected key frames or compress memory.
  • Thin structures and motion blur: hair, wires and fast limbs remain hard even for state-of-the-art models.

The usual trade-off is accuracy versus latency. Editing and annotation tools favor heavy models with user correction, while robotics and driving favor lightweight models running in real time on embedded hardware.

Understanding it simply

Video object segmentation means cutting an object out of a video, frame by frame. The cut is a precise outline, not a rough box. Imagine tracing a dancer with scissors in every frame of a film. The cut must stay exact as she spins, jumps and slips behind another dancer.

A VOS system does that tracing automatically. You usually show it the object once. It then follows the outline through the rest of the video. It also remembers how the object looked, so it can find it again after it is hidden.

Applied in practice

A small post-production studio must replace the background behind an actor in a 90-second handheld shot filmed without a green screen. Manual rotoscoping of the actor’s outline would take an artist several days, because the shot contains about 2,700 frames at 30 frames per second.

The artist instead draws one mask on the first frame and lets a memory-based VOS model propagate it. Where the model loses detail, such as wind-blown hair and a hand passing in front of the face, the artist corrects about 15 key frames and re-propagates. The finished matte is ready in a few hours, and the remaining work is edge refinement rather than full manual tracing.