Visual wake words is a binary image classification task in which a very small neural network decides whether a person is present in a camera frame, so that a low-power device can trigger a heavier action only when it matters. In Edge AI, the term names both the use case and a widely used benchmark dataset, which serves as the vision counterpart of keyword spotting in audio: a cheap, always-on detector that wakes the rest of the system.
The concept matters because it defines a realistic workload for microcontroller-class hardware with a few hundred kilobytes of memory. It has become a reference point for comparing compact architectures, quantization schemes and inference runtimes under tight memory, compute and energy budgets.
How a visual wake word detector works
A visual wake word detector runs a compact convolutional network on each low-resolution frame and outputs a single probability that a person is visible. When that probability crosses a decision threshold, the device wakes a radio, a larger processor, a recording pipeline or a cloud request.
The pipeline is deliberately minimal. A low-power image sensor captures a small frame, often 96 by 96 pixels in grayscale or RGB, the microcontroller normalizes it, and an INT8-quantized model such as a reduced-width MobileNet produces the score. Everything happens in on-chip SRAM and flash, with no network connection required for the decision itself.
The following table lists the stages of a typical pipeline and the resource each one consumes.
| Stage | What happens | Main resource consumed |
|---|---|---|
| Image capture | Low-resolution sensor grabs a frame | Sensor power and frame buffer |
| Preprocessing | Crop, resize and scale pixel values | A few kilobytes of RAM |
| Inference | Quantized CNN computes person score | Compute cycles and peak activation memory |
| Thresholding | Score compared with a tuned cut-off | Negligible |
| Wake action | Larger subsystem or radio is enabled | Most of the energy budget when triggered |
The economics rest on duty cycling. The detector stays cheap enough to run continuously, while the expensive components remain asleep most of the time. This is the same cascade principle behind keyword spotting and other forms of always-on sensing.
The Visual Wake Words dataset and benchmark
The Visual Wake Words dataset was introduced in 2019 by Google researchers as a standard task for tiny vision models. It relabels images from the COCO dataset into two classes, person and not-person, and marks an image as containing a person when at least one person bounding box covers more than a small fraction of the image area, commonly 0.5 percent.
The original challenge attached explicit hardware limits to the task, which is what makes it useful for comparison. Later, MLPerf Tiny adopted visual wake words as one of its four benchmarks, with a reference MobileNetV1 model at reduced width and a quality target of roughly 80 percent top-1 accuracy.
The next table summarizes the commonly cited parameters of the benchmark; exact settings vary between the original challenge, MLPerf Tiny and individual papers.
| Parameter | Typical value | Why it matters |
|---|---|---|
| Source images | COCO, relabeled into two classes | Realistic, cluttered scenes |
| Input resolution | 96 x 96 pixels, sometimes up to 224 x 224 | Drives activation memory and compute |
| Model and peak memory limit | About 250 KB each in the original challenge | Fits typical microcontroller SRAM and flash |
| Compute limit | About 60 million multiply-accumulates | Keeps latency and energy low |
| Accuracy target | About 80 percent in MLPerf Tiny | Defines a useful quality floor |
Because the dataset is public and the task is simple to state, it has become a common test for hardware-aware neural architecture search, quantization-aware training and new inference runtimes.
How visual wake words compares with related edge tasks
Visual wake words sits between audio keyword spotting and full object detection in difficulty. It answers a single yes-or-no question, but it must do so on images with varied lighting, scale, pose and occlusion, which is harder than recognizing a spoken word.
The comparison below places it among neighboring workloads that run on constrained devices.
| Workload | Output | Typical hardware | Relative cost |
|---|---|---|---|
| Keyword spotting | One of a few spoken words | Microcontroller with microphone | Lowest |
| Visual wake words | Person present or absent | Microcontroller with small camera | Low |
| Image classification | One of many classes | Microcontroller or mobile NPU | Medium |
| Object detection | Boxes and labels for many objects | Mobile NPU or edge accelerator | High |
| Face recognition | Identity of a person | Mobile NPU or edge GPU | High, with privacy obligations |
A visual wake word model does not locate or identify anyone. It only signals presence, which keeps the model small and limits the personal information the device processes, a point that matters for privacy assessments of camera-based products.
Designing a model that fits the budget
Fitting a person detector into a few hundred kilobytes requires co-design of model, runtime and hardware. The usual starting point is a depthwise separable architecture such as MobileNet with a width multiplier of 0.25 to 0.5, followed by full INT8 post-training quantization or quantization-aware training.
The binding constraint is often peak activation memory, not parameter count. Early layers at full resolution produce the largest feature maps, so reducing input size, using patch-based inference or reordering operators can matter more than pruning weights.
The decision table below links common design situations with a recommended approach.
| Situation | Recommended approach | Reason |
|---|---|---|
| SRAM below 256 KB | Lower input resolution or patch-based inference | Cuts peak activation memory |
| Flash below 300 KB | INT8 weights and narrower channels | Shrinks the model footprint |
| Accuracy below target | Quantization-aware training and augmentation | Recovers quantization loss |
| Many false wake-ups | Raise the threshold or add temporal smoothing | Trades recall for fewer triggers |
| New camera or site | Fine-tune on local images | Reduces domain shift |
Runtimes such as TensorFlow Lite Micro and vendor-optimized kernel libraries execute the result. Operator fusion and hardware delegates on chips with vector extensions or small NPUs can reduce latency from hundreds of milliseconds to tens of milliseconds per frame.
Limitations and common mistakes
Visual wake word models are brittle outside their training distribution. Benchmark accuracy on COCO says little about performance with a specific lens, mounting height, infrared illumination or crowd density, so field validation is essential.
- Treating benchmark accuracy as deployment accuracy: COCO photos differ from fixed low-angle security or appliance cameras.
- Ignoring small or distant people: the area threshold used in labeling means very small figures may count as absent.
- Optimizing only for model size: a model that fits in flash can still overflow SRAM during inference.
- Skipping threshold tuning: the default 0.5 cut-off rarely matches the real cost of false wake-ups versus misses.
- Assuming privacy by default: presence detection still processes images, and data protection rules may apply if frames are stored or transmitted.
In simple terms
Visual wake words is a tiny program on a small chip that looks at a camera picture and answers one question: is there a person here or not? It does not know who the person is. It only notices that someone is there.
Think of a motion-sensor light that is smarter than usual. Instead of switching on for a cat or a moving branch, it waits for a human. The small detector stays awake so the big system can sleep, which saves battery and avoids sending pictures to the internet.
A practical example
A manufacturer of battery-powered doorbell cameras wants a device that lasts six months on one charge. Streaming video to the cloud all day would drain the battery within days, so the team adds a visual wake word model to the doorbell’s microcontroller.
The model uses a MobileNet variant at 0.25 width, takes 96 by 96 grayscale frames and is quantized to INT8, fitting in about 250 KB of flash. It checks one frame per second at a few milliwatts. Only when it reports a person with confidence above a tuned threshold of 0.7 does the device wake its main processor and Wi-Fi radio to record and upload a clip.
After field testing at 200 homes, the team retrains the model on images from the real camera angle, which cuts false wake-ups caused by passing cars substantially. The result is a product that records visitors reliably while keeping the main radio off for most of the day.