Video object segmentation (VOS) is the computer vision task of assigning every pixel in every frame of a video to one or more target objects or to the background, so that each object is outlined by a pixel-accurate mask that follows it through time. It extends image segmentation from single pictures to sequences and adds the problem of temporal consistency: the same object must keep the same identity as it moves, deforms, becomes occluded and reappears.
In computer vision, VOS sits between object tracking, which follows objects with boxes, and per-frame instance segmentation, which outlines objects without linking them across frames. It powers video editing, robotics, autonomous driving perception and medical video analysis.
How video object segmentation works
A VOS system works by propagating object information from frame to frame while correcting it with the appearance of each new frame. Modern systems encode each frame with a convolutional or transformer backbone, compare its features with features of earlier frames and decode a mask for every target object.
The dominant design since about 2019 is the memory-based matching network. It stores feature keys and mask values from past frames in a memory bank, and for each new frame it computes attention between current pixels and memory pixels. Each pixel then inherits the object label of the memory pixels it most resembles. This approach handles fast motion and occlusion better than earlier methods that relied on optical flow alone or on fine-tuning a network on the first frame at test time.
The main architectural families differ in how they carry information across time, as the following table shows.
| Approach | Core idea | Typical speed | Main weakness |
|---|---|---|---|
| Online fine-tuning | Retrains a network on the annotated first frame | Seconds per frame | Too slow for interactive use |
| Flow-based propagation | Warps the previous mask with optical flow | Moderate | Errors accumulate through occlusion |
| Memory-based matching | Attends from current pixels to stored past frames | Real time on a modern GPU | Memory grows with video length |
| Query-based transformers | Learns object queries that persist across frames | Moderate to fast | Needs large training sets |
| Promptable foundation models | Segments any object from clicks, boxes or masks | Near real time | Heavy compute and model size |
Problem settings and variants
VOS is not one task but a family of settings that differ in how the target is specified. The choice of setting determines which models, datasets and metrics apply.
| Setting | How targets are given | Typical use |
|---|---|---|
| Semi-supervised VOS | Mask of each object in the first frame | Video editing and rotoscoping |
| Unsupervised VOS | No input; the model finds salient moving objects | Automatic video summarization |
| Interactive VOS | User clicks or scribbles, refined over rounds | Annotation tools and post-production |
| Referring VOS | A natural-language description of the object | Text-driven video search and editing |
| Video instance segmentation | No input; all objects of known classes | Driving and surveillance perception |
The semi-supervised setting is the most studied and is sometimes simply called VOS. The name is historical: the model is not trained in a semi-supervised way, but it receives one supervised mask at inference time and must propagate it through the rest of the video.
VOS compared with neighboring tasks
Video object segmentation differs from related computer vision tasks in output granularity and temporal linkage. The comparison below clarifies where VOS fits and why it is harder than its neighbors.
| Task | Output per frame | Identity across frames |
|---|---|---|
| Video object segmentation | Pixel mask per target object | Yes, consistent object identity |
| Object tracking | Bounding box per object | Yes, consistent track identity |
| Instance segmentation | Pixel mask per detected object | No, each frame is independent |
| Semantic segmentation | Class label per pixel | No, and no separate instances |
| Background subtraction | Foreground versus background pixels | No object identities |
In practice the boundaries blur. Many trackers now output masks, and VOS models are often combined with an object detector that proposes targets in the first frame, which turns a semi-supervised method into a fully automatic pipeline.
Benchmarks and evaluation metrics
VOS is evaluated with region accuracy and boundary accuracy, usually averaged into a single score. The standard benchmarks include DAVIS, which offers short, densely annotated clips, and YouTube-VOS, which covers many more object categories and tests generalization to unseen classes. Newer sets such as MOSE stress heavy occlusion and crowded scenes, and LVOS tests long videos lasting minutes.
- Region similarity (J): the intersection over union between predicted and ground-truth masks.
- Contour accuracy (F): an F-measure on boundary pixels, which rewards precise object outlines.
- J&F mean: the average of both scores and the headline number on most leaderboards.
- Frames per second: throughput on a stated GPU, essential for interactive and real-time use.
Leading methods score far higher on short, clean clips than on long or crowded videos, so a single benchmark number overstates real-world robustness. Results also depend on input resolution and on whether the model was pretrained on extra image data.
Practical challenges and design choices
Deployed VOS systems fail in predictable situations, and engineers design around them with memory management, re-detection and human correction.
- Occlusion and reappearance: an object hidden for many frames may be relabeled or lost; long-term memory and re-identification cues reduce this.
- Similar distractors: identical objects such as players in the same uniform can swap identities; adding spatial priors helps.
- Drift: small mask errors propagate and grow, so many tools let users correct a frame and re-propagate.
- Memory cost: storing every frame is infeasible for long videos, so systems keep selected key frames or compress memory.
- Thin structures and motion blur: hair, wires and fast limbs remain hard even for state-of-the-art models.
The usual trade-off is accuracy versus latency. Editing and annotation tools favor heavy models with user correction, while robotics and driving favor lightweight models running in real time on embedded hardware.
Understanding it simply
Video object segmentation means cutting an object out of a video, frame by frame. The cut is a precise outline, not a rough box. Imagine tracing a dancer with scissors in every frame of a film. The cut must stay exact as she spins, jumps and slips behind another dancer.
A VOS system does that tracing automatically. You usually show it the object once. It then follows the outline through the rest of the video. It also remembers how the object looked, so it can find it again after it is hidden.
Applied in practice
A small post-production studio must replace the background behind an actor in a 90-second handheld shot filmed without a green screen. Manual rotoscoping of the actor’s outline would take an artist several days, because the shot contains about 2,700 frames at 30 frames per second.
The artist instead draws one mask on the first frame and lets a memory-based VOS model propagate it. Where the model loses detail, such as wind-blown hair and a hand passing in front of the face, the artist corrects about 15 key frames and re-propagates. The finished matte is ready in a few hours, and the remaining work is edge refinement rather than full manual tracing.