Background subtraction

On this page 7

Background subtraction is a computer vision technique that separates moving foreground objects from a static or slowly changing background by comparing each video frame with a statistical model of the scene. Pixels that deviate significantly from the model are labeled as foreground, producing a binary foreground mask that marks people, vehicles or other objects in motion.

The method is a classic building block of video surveillance, traffic monitoring and motion-triggered recording. It is fast, needs no labeled training data and works with a fixed camera, which is why it remains common in embedded systems even though deep learning detectors now dominate many other tasks.

How background subtraction separates motion from the scene

Background subtraction works in a repeating cycle of modeling, comparison and update. The system first builds a background model from initial frames, then classifies each pixel of every new frame by measuring how far it lies from that model, and finally updates the model so that it follows gradual scene changes.

The comparison usually relies on a distance threshold. In the simplest form, a pixel is foreground when the absolute difference between its intensity and the background value exceeds a fixed value, often between 15 and 40 on an 8-bit scale. Statistical methods replace the fixed threshold with a test against the variance of each pixel, so noisy regions tolerate larger deviations.

The raw mask then passes through post-processing. Morphological opening removes isolated noise pixels, closing fills small holes, and connected-component labeling groups the remaining pixels into blobs that later stages can track or count. The stages are summarized below.

StageOperationOutput
InitializationEstimate background from the first framesInitial background model
Foreground detectionCompare each pixel with the modelRaw binary mask
Model updateBlend new observations at a learning rateAdapted background model
Post-processingMorphology, shadow removal, blob groupingClean object regions
Stages of a typical background subtraction pipeline

Main background modeling methods

Background modeling methods differ in how they represent the expected appearance of each pixel, and that choice determines speed, memory use and tolerance to dynamic scenes. The simplest methods store one value per pixel, while advanced ones store distributions or sample sets.

The Gaussian mixture model, often called MOG or its adaptive variant MOG2, represents each pixel with three to five weighted Gaussian distributions. This allows a pixel to have several legitimate background states, such as a swaying branch that alternates between leaf and sky. Sample-based methods such as ViBe keep a set of past values per pixel and update them randomly, which makes them fast and robust.

MethodBackground representationStrengthWeakness
Frame differencingPrevious frameExtremely fastMisses slow or stopped objects
Running averageExponentially weighted meanSimple, low memoryPoor with multimodal backgrounds
Gaussian mixture modelSeveral Gaussians per pixelHandles repetitive motionSensitive to parameter tuning
KNN and ViBeStored samples per pixelRobust and fast to adaptMore memory per pixel
Deep learning modelsLearned features from videoHigh accuracy on hard scenesNeeds training data and GPU power
Common background models and their trade-offs

Learned approaches based on convolutional networks can reach high accuracy on benchmark datasets, but they often generalize poorly to unseen cameras unless retrained. For a fixed installation with modest hardware, a well-tuned classical model frequently remains the practical choice.

Background subtraction answers the question which pixels have changed, not what the changed pixels are. That distinguishes it from object detection and segmentation, which recognize object classes, and from optical flow, which estimates how pixels move.

TechniqueMain question answeredRequires a static camera
Background subtractionWhich pixels differ from the scene modelYes, or motion compensation
Object detectionWhich objects are present and whereNo
Semantic segmentationWhich class each pixel belongs toNo
Optical flowHow each pixel moves between framesNo
Video object segmentationWhich pixels belong to a given object over timeNo
Background subtraction versus neighboring computer vision tasks

In practice, the techniques are often combined. A background subtraction stage cheaply finds regions of motion, and an object detector such as YOLO runs only on those regions, which reduces computing cost on long, mostly static footage. Object tracking then links detections across frames.

Configuring background subtraction for real scenes

Configuring background subtraction means balancing adaptation speed against stability. The learning rate controls how quickly the model absorbs changes: a high rate tracks lighting shifts but also absorbs a person who stands still, while a low rate keeps stopped objects visible but reacts slowly to clouds or switched lamps.

The history length, the number of frames that shape the model, plays a similar role. Values of a few hundred frames are common for 25 to 30 frames per second video. The decision table below lists typical adjustments.

SituationRecommended settingReason
Outdoor scene with cloudsModerate learning rate, illumination normalizationFollows gradual light changes
Waving trees or waterMultimodal model such as MOG2 or KNNAccepts repetitive background motion
Parked or waiting objectsLow learning rate or selective updatePrevents objects merging into background
Strong shadowsEnable shadow detection in color spaceAvoids enlarged or false blobs
Pan-tilt or handheld cameraMotion compensation or a detector insteadStatic-scene assumption is violated
Configuration choices for typical scenes

Limitations and common failure modes

Background subtraction fails most often when its core assumption of a fixed camera and stable scene breaks. Practitioners recognize a set of recurring problems that every deployment must anticipate.

  • Sudden illumination changes, such as lights switching on, can turn most of the frame into false foreground for several seconds.
  • Camouflage occurs when an object resembles the background in color, leaving holes or missing detections.
  • Ghosting appears when a background object moves away and its former location is detected as foreground.
  • Sleeping foreground: an object that stops long enough is gradually absorbed into the background model.
  • Camera jitter from wind or vibration produces edge noise unless frames are stabilized first.

Because the output is only a change mask, background subtraction cannot identify object classes on its own. Systems that must tell a person from a dog or a shadow need an additional classifier or detector.

Understanding it simply

Picture a security guard who has looked at the same empty hallway for hours. The guard knows exactly how it should look, so anything new, like a person walking through, stands out immediately.

Background subtraction gives a computer the same habit. It remembers what the empty scene looks like and highlights whatever is different in each new picture. It does not know what the moving thing is, only that something has changed, which makes it a quick and inexpensive first filter for camera systems.

Applied in practice

A city traffic department mounts a fixed camera above a four-lane intersection to count vehicles at 1280 by 720 pixels and 25 frames per second. Running a deep detector on every frame would require a server with a GPU, so engineers use an adaptive Gaussian mixture model on a small embedded computer instead, with a history of 500 frames and shadow detection enabled.

The background model marks moving vehicles as blobs, morphological filtering removes noise from rain, and virtual counting lines register each blob that crosses them. A lightweight classifier runs only on detected regions to separate cars, buses and bicycles.

During validation against manual counts, the system reaches counting accuracy of roughly 95 percent in daylight. Accuracy drops at dusk when headlights create glare, so the team adds illumination normalization and a lower learning rate at night, which restores most of the lost performance.