Image pyramid

On this page 7

An image pyramid is a multi-scale representation of an image built by repeatedly smoothing and downsampling the original, so that each level holds the same scene at a coarser resolution. In computer vision, it lets an algorithm search for structures of unknown size with a fixed-size operator, which is the basis of classic detectors, feature extractors, optical flow and image blending.

The concept predates deep learning by decades, yet it remains central today. Modern detection networks reproduce the same idea inside their layers through feature pyramids, and many production systems still build explicit pyramids when objects vary strongly in apparent size.

How an image pyramid is constructed

Construction repeats two steps. First, the current level is filtered with a low-pass kernel, usually a Gaussian or a small binomial approximation of it. Second, the filtered image is subsampled, most often by keeping every second row and column, which halves width and height and reduces the pixel count to about a quarter.

The smoothing step is essential. Downsampling without it causes aliasing, where fine patterns fold into false coarse patterns such as moire. The Gaussian filter suppresses frequencies above the new Nyquist limit before samples are discarded. Because each level holds a quarter of the previous one, a full pyramid needs only about one third more memory than the original image.

The table below shows the levels of a typical octave pyramid built from a 1920 by 1080 frame.

LevelResolutionShare of original pixelsTypical use
Level 01920 x 1080100%Fine detail and small objects
Level 1960 x 54025%Medium objects and edges
Level 2480 x 2706.25%Large objects and layout
Level 3240 x 1351.56%Coarse motion and global context
Level 4120 x 680.39%Initialization of coarse-to-fine search
Levels of a factor-two pyramid from a full HD frame

Main types of image pyramids

Several pyramid variants exist, and they differ in what each level stores and whether the original can be reconstructed. The choice depends on whether the task needs smoothed appearance, band-limited detail or orientation information.

VariantWhat each level storesInvertibleTypical application
Gaussian pyramidSmoothed, downsampled imageNoDetection, tracking, coarse-to-fine search
Laplacian pyramidDifference between adjacent Gaussian levelsYesImage blending, compression, detail editing
Difference-of-Gaussians stackDifferences between blur levels within octavesNoScale-invariant keypoints as in SIFT
Steerable pyramidOriented band-pass subbandsYesTexture analysis and synthesis
Feature pyramidLearned feature maps at several stridesNoDeep object detection and segmentation
Common pyramid variants

The Laplacian pyramid deserves special mention. Each level stores what is lost between two Gaussian levels, a band-pass detail layer, and adding all layers back to the coarsest image restores the original exactly. This property makes it the standard tool for seamless multi-band blending in panorama stitching.

Why multi-scale analysis matters in computer vision

Objects in images appear at unpredictable sizes because of distance and camera focal length. A detector trained on 64 by 128 pixel windows, such as a pedestrian detector using a histogram of oriented gradients, only finds people of roughly that size. Running the same window over every pyramid level makes the detector effectively scale-invariant without retraining.

Pyramids also enable coarse-to-fine estimation. Optical flow methods such as the pyramidal Lucas-Kanade tracker first estimate large motions on a small image, where displacements are only a few pixels, and then refine the estimate level by level. The same strategy speeds up template matching, image registration and stereo matching by pruning the search space early.

  • Sliding-window detection scans every level with one fixed window.
  • Keypoint detection finds extrema across scale and position, as in SIFT.
  • Motion estimation refines displacements from coarse to fine levels.
  • Image blending merges frequency bands separately to hide seams.

An explicit image pyramid is one of several ways to handle scale. Deep networks often replace it with in-network feature hierarchies, which share computation across scales. The comparison below clarifies the trade-offs.

ApproachCompute costStrengthWeakness
Featurized image pyramidHigh, network runs per levelStrong features at every scaleSlow for real-time inference
Single-scale feature mapLowFast and simpleMisses very small or very large objects
Feature pyramid networkModerateSemantic features at all scalesNeeds careful anchor and level assignment
Multi-scale training and testingHigh at test timeImproves accuracy benchmarksLatency grows with each extra scale
Ways to handle scale in vision models

Feature pyramid networks combine a bottom-up backbone with a top-down path and lateral connections, so even high-resolution levels carry semantic information. Detectors such as YOLO and R-CNN variants use this design, yet test-time image pyramids are still applied when the highest accuracy matters more than speed.

Practical choices and common mistakes

Key parameters include the scale factor between levels, the smoothing kernel and the number of levels. A factor of two is efficient but coarse; detectors often use finer steps of about 1.1 to 1.25, which produce more levels and higher cost but avoid missing objects that fall between scales. The pyramid usually stops when the image becomes smaller than the detection window.

SymptomLikely causeFix
Moire or jagged edgesNo low-pass filter before subsamplingApply Gaussian smoothing first
Objects missed between sizesScale factor too largeUse smaller steps between levels
Duplicate detectionsSame object found at several levelsApply non-maximum suppression across levels
Boxes misaligned after mapping backRounding of level dimensionsTrack exact scale per level
Slow inferenceToo many fine levelsLimit scale range to expected object sizes
Symptoms, causes and fixes

Bounding boxes found on a coarse level must be multiplied by that level’s scale to map back to the original image. Ignoring odd image dimensions introduces small offsets that accumulate across levels and reduce intersection over union with ground-truth boxes.

The idea in brief

An image pyramid is like keeping the same photo in several sizes, from full size down to a small thumbnail. Each smaller copy is gently blurred first so it still looks clean.

Why bother? A computer program that looks for faces or cars often checks for one fixed size. By running the same check on every copy, it can find both near and far objects. Small copies also help with quick, rough searches, and the larger copies then refine the answer.

A practical scenario

A traffic engineering team installs a camera above a busy intersection to count pedestrians crossing at a signal. People near the camera appear about 300 pixels tall, while those on the far side are only about 60 pixels tall. Their detector expects people around 128 pixels tall.

The team builds an image pyramid from each 1920 by 1080 frame with a scale factor of 1.2, upsampling once for the most distant pedestrians and downsampling for the nearest ones. The detector runs on every level, and non-maximum suppression merges duplicate boxes. To keep up with 15 frames per second, they restrict the range to the levels where people can actually appear. The result is a reliable count across the whole crossing, including small figures that a single-scale detector missed.