An image pyramid is a multi-scale representation of an image built by repeatedly smoothing and downsampling the original, so that each level holds the same scene at a coarser resolution. In computer vision, it lets an algorithm search for structures of unknown size with a fixed-size operator, which is the basis of classic detectors, feature extractors, optical flow and image blending.
The concept predates deep learning by decades, yet it remains central today. Modern detection networks reproduce the same idea inside their layers through feature pyramids, and many production systems still build explicit pyramids when objects vary strongly in apparent size.
How an image pyramid is constructed
Construction repeats two steps. First, the current level is filtered with a low-pass kernel, usually a Gaussian or a small binomial approximation of it. Second, the filtered image is subsampled, most often by keeping every second row and column, which halves width and height and reduces the pixel count to about a quarter.
The smoothing step is essential. Downsampling without it causes aliasing, where fine patterns fold into false coarse patterns such as moire. The Gaussian filter suppresses frequencies above the new Nyquist limit before samples are discarded. Because each level holds a quarter of the previous one, a full pyramid needs only about one third more memory than the original image.
The table below shows the levels of a typical octave pyramid built from a 1920 by 1080 frame.
| Level | Resolution | Share of original pixels | Typical use |
|---|---|---|---|
| Level 0 | 1920 x 1080 | 100% | Fine detail and small objects |
| Level 1 | 960 x 540 | 25% | Medium objects and edges |
| Level 2 | 480 x 270 | 6.25% | Large objects and layout |
| Level 3 | 240 x 135 | 1.56% | Coarse motion and global context |
| Level 4 | 120 x 68 | 0.39% | Initialization of coarse-to-fine search |
Main types of image pyramids
Several pyramid variants exist, and they differ in what each level stores and whether the original can be reconstructed. The choice depends on whether the task needs smoothed appearance, band-limited detail or orientation information.
| Variant | What each level stores | Invertible | Typical application |
|---|---|---|---|
| Gaussian pyramid | Smoothed, downsampled image | No | Detection, tracking, coarse-to-fine search |
| Laplacian pyramid | Difference between adjacent Gaussian levels | Yes | Image blending, compression, detail editing |
| Difference-of-Gaussians stack | Differences between blur levels within octaves | No | Scale-invariant keypoints as in SIFT |
| Steerable pyramid | Oriented band-pass subbands | Yes | Texture analysis and synthesis |
| Feature pyramid | Learned feature maps at several strides | No | Deep object detection and segmentation |
The Laplacian pyramid deserves special mention. Each level stores what is lost between two Gaussian levels, a band-pass detail layer, and adding all layers back to the coarsest image restores the original exactly. This property makes it the standard tool for seamless multi-band blending in panorama stitching.
Why multi-scale analysis matters in computer vision
Objects in images appear at unpredictable sizes because of distance and camera focal length. A detector trained on 64 by 128 pixel windows, such as a pedestrian detector using a histogram of oriented gradients, only finds people of roughly that size. Running the same window over every pyramid level makes the detector effectively scale-invariant without retraining.
Pyramids also enable coarse-to-fine estimation. Optical flow methods such as the pyramidal Lucas-Kanade tracker first estimate large motions on a small image, where displacements are only a few pixels, and then refine the estimate level by level. The same strategy speeds up template matching, image registration and stereo matching by pruning the search space early.
- Sliding-window detection scans every level with one fixed window.
- Keypoint detection finds extrema across scale and position, as in SIFT.
- Motion estimation refines displacements from coarse to fine levels.
- Image blending merges frequency bands separately to hide seams.
Image pyramids compared with related multi-scale techniques
An explicit image pyramid is one of several ways to handle scale. Deep networks often replace it with in-network feature hierarchies, which share computation across scales. The comparison below clarifies the trade-offs.
| Approach | Compute cost | Strength | Weakness |
|---|---|---|---|
| Featurized image pyramid | High, network runs per level | Strong features at every scale | Slow for real-time inference |
| Single-scale feature map | Low | Fast and simple | Misses very small or very large objects |
| Feature pyramid network | Moderate | Semantic features at all scales | Needs careful anchor and level assignment |
| Multi-scale training and testing | High at test time | Improves accuracy benchmarks | Latency grows with each extra scale |
Feature pyramid networks combine a bottom-up backbone with a top-down path and lateral connections, so even high-resolution levels carry semantic information. Detectors such as YOLO and R-CNN variants use this design, yet test-time image pyramids are still applied when the highest accuracy matters more than speed.
Practical choices and common mistakes
Key parameters include the scale factor between levels, the smoothing kernel and the number of levels. A factor of two is efficient but coarse; detectors often use finer steps of about 1.1 to 1.25, which produce more levels and higher cost but avoid missing objects that fall between scales. The pyramid usually stops when the image becomes smaller than the detection window.
| Symptom | Likely cause | Fix |
|---|---|---|
| Moire or jagged edges | No low-pass filter before subsampling | Apply Gaussian smoothing first |
| Objects missed between sizes | Scale factor too large | Use smaller steps between levels |
| Duplicate detections | Same object found at several levels | Apply non-maximum suppression across levels |
| Boxes misaligned after mapping back | Rounding of level dimensions | Track exact scale per level |
| Slow inference | Too many fine levels | Limit scale range to expected object sizes |
Bounding boxes found on a coarse level must be multiplied by that level’s scale to map back to the original image. Ignoring odd image dimensions introduces small offsets that accumulate across levels and reduce intersection over union with ground-truth boxes.
The idea in brief
An image pyramid is like keeping the same photo in several sizes, from full size down to a small thumbnail. Each smaller copy is gently blurred first so it still looks clean.
Why bother? A computer program that looks for faces or cars often checks for one fixed size. By running the same check on every copy, it can find both near and far objects. Small copies also help with quick, rough searches, and the larger copies then refine the answer.
A practical scenario
A traffic engineering team installs a camera above a busy intersection to count pedestrians crossing at a signal. People near the camera appear about 300 pixels tall, while those on the far side are only about 60 pixels tall. Their detector expects people around 128 pixels tall.
The team builds an image pyramid from each 1920 by 1080 frame with a scale factor of 1.2, upsampling once for the most distant pedestrians and downsampling for the nearest ones. The detector runs on every level, and non-maximum suppression merges duplicate boxes. To keep up with 15 frames per second, they restrict the range to the levels where people can actually appear. The result is a reliable count across the whole crossing, including small figures that a single-scale detector missed.