Visual wake words

On this page 7

Visual wake words is a binary image classification task in which a very small neural network decides whether a person is present in a camera frame, so that a low-power device can trigger a heavier action only when it matters. In Edge AI, the term names both the use case and a widely used benchmark dataset, which serves as the vision counterpart of keyword spotting in audio: a cheap, always-on detector that wakes the rest of the system.

The concept matters because it defines a realistic workload for microcontroller-class hardware with a few hundred kilobytes of memory. It has become a reference point for comparing compact architectures, quantization schemes and inference runtimes under tight memory, compute and energy budgets.

How a visual wake word detector works

A visual wake word detector runs a compact convolutional network on each low-resolution frame and outputs a single probability that a person is visible. When that probability crosses a decision threshold, the device wakes a radio, a larger processor, a recording pipeline or a cloud request.

The pipeline is deliberately minimal. A low-power image sensor captures a small frame, often 96 by 96 pixels in grayscale or RGB, the microcontroller normalizes it, and an INT8-quantized model such as a reduced-width MobileNet produces the score. Everything happens in on-chip SRAM and flash, with no network connection required for the decision itself.

The following table lists the stages of a typical pipeline and the resource each one consumes.

StageWhat happensMain resource consumed
Image captureLow-resolution sensor grabs a frameSensor power and frame buffer
PreprocessingCrop, resize and scale pixel valuesA few kilobytes of RAM
InferenceQuantized CNN computes person scoreCompute cycles and peak activation memory
ThresholdingScore compared with a tuned cut-offNegligible
Wake actionLarger subsystem or radio is enabledMost of the energy budget when triggered
Stages of a typical visual wake word pipeline

The economics rest on duty cycling. The detector stays cheap enough to run continuously, while the expensive components remain asleep most of the time. This is the same cascade principle behind keyword spotting and other forms of always-on sensing.

The Visual Wake Words dataset and benchmark

The Visual Wake Words dataset was introduced in 2019 by Google researchers as a standard task for tiny vision models. It relabels images from the COCO dataset into two classes, person and not-person, and marks an image as containing a person when at least one person bounding box covers more than a small fraction of the image area, commonly 0.5 percent.

The original challenge attached explicit hardware limits to the task, which is what makes it useful for comparison. Later, MLPerf Tiny adopted visual wake words as one of its four benchmarks, with a reference MobileNetV1 model at reduced width and a quality target of roughly 80 percent top-1 accuracy.

The next table summarizes the commonly cited parameters of the benchmark; exact settings vary between the original challenge, MLPerf Tiny and individual papers.

ParameterTypical valueWhy it matters
Source imagesCOCO, relabeled into two classesRealistic, cluttered scenes
Input resolution96 x 96 pixels, sometimes up to 224 x 224Drives activation memory and compute
Model and peak memory limitAbout 250 KB each in the original challengeFits typical microcontroller SRAM and flash
Compute limitAbout 60 million multiply-accumulatesKeeps latency and energy low
Accuracy targetAbout 80 percent in MLPerf TinyDefines a useful quality floor
Commonly cited visual wake words benchmark parameters

Because the dataset is public and the task is simple to state, it has become a common test for hardware-aware neural architecture search, quantization-aware training and new inference runtimes.

Visual wake words sits between audio keyword spotting and full object detection in difficulty. It answers a single yes-or-no question, but it must do so on images with varied lighting, scale, pose and occlusion, which is harder than recognizing a spoken word.

The comparison below places it among neighboring workloads that run on constrained devices.

WorkloadOutputTypical hardwareRelative cost
Keyword spottingOne of a few spoken wordsMicrocontroller with microphoneLowest
Visual wake wordsPerson present or absentMicrocontroller with small cameraLow
Image classificationOne of many classesMicrocontroller or mobile NPUMedium
Object detectionBoxes and labels for many objectsMobile NPU or edge acceleratorHigh
Face recognitionIdentity of a personMobile NPU or edge GPUHigh, with privacy obligations
Visual wake words compared with neighboring edge workloads

A visual wake word model does not locate or identify anyone. It only signals presence, which keeps the model small and limits the personal information the device processes, a point that matters for privacy assessments of camera-based products.

Designing a model that fits the budget

Fitting a person detector into a few hundred kilobytes requires co-design of model, runtime and hardware. The usual starting point is a depthwise separable architecture such as MobileNet with a width multiplier of 0.25 to 0.5, followed by full INT8 post-training quantization or quantization-aware training.

The binding constraint is often peak activation memory, not parameter count. Early layers at full resolution produce the largest feature maps, so reducing input size, using patch-based inference or reordering operators can matter more than pruning weights.

The decision table below links common design situations with a recommended approach.

SituationRecommended approachReason
SRAM below 256 KBLower input resolution or patch-based inferenceCuts peak activation memory
Flash below 300 KBINT8 weights and narrower channelsShrinks the model footprint
Accuracy below targetQuantization-aware training and augmentationRecovers quantization loss
Many false wake-upsRaise the threshold or add temporal smoothingTrades recall for fewer triggers
New camera or siteFine-tune on local imagesReduces domain shift
Design choices for a visual wake word model

Runtimes such as TensorFlow Lite Micro and vendor-optimized kernel libraries execute the result. Operator fusion and hardware delegates on chips with vector extensions or small NPUs can reduce latency from hundreds of milliseconds to tens of milliseconds per frame.

Limitations and common mistakes

Visual wake word models are brittle outside their training distribution. Benchmark accuracy on COCO says little about performance with a specific lens, mounting height, infrared illumination or crowd density, so field validation is essential.

  • Treating benchmark accuracy as deployment accuracy: COCO photos differ from fixed low-angle security or appliance cameras.
  • Ignoring small or distant people: the area threshold used in labeling means very small figures may count as absent.
  • Optimizing only for model size: a model that fits in flash can still overflow SRAM during inference.
  • Skipping threshold tuning: the default 0.5 cut-off rarely matches the real cost of false wake-ups versus misses.
  • Assuming privacy by default: presence detection still processes images, and data protection rules may apply if frames are stored or transmitted.

In simple terms

Visual wake words is a tiny program on a small chip that looks at a camera picture and answers one question: is there a person here or not? It does not know who the person is. It only notices that someone is there.

Think of a motion-sensor light that is smarter than usual. Instead of switching on for a cat or a moving branch, it waits for a human. The small detector stays awake so the big system can sleep, which saves battery and avoids sending pictures to the internet.

A practical example

A manufacturer of battery-powered doorbell cameras wants a device that lasts six months on one charge. Streaming video to the cloud all day would drain the battery within days, so the team adds a visual wake word model to the doorbell’s microcontroller.

The model uses a MobileNet variant at 0.25 width, takes 96 by 96 grayscale frames and is quantized to INT8, fitting in about 250 KB of flash. It checks one frame per second at a few milliwatts. Only when it reports a person with confidence above a tuned threshold of 0.7 does the device wake its main processor and Wi-Fi radio to record and upload a clip.

After field testing at 200 homes, the team retrains the model on images from the real camera angle, which cuts false wake-ups caused by passing cars substantially. The result is a product that records visitors reliably while keeping the main radio off for most of the day.