MLPerf Tiny

On this page 8

MLPerf Tiny is an industry benchmark suite from MLCommons that measures how fast and how efficiently ultra-low-power devices, typically microcontrollers with kilobytes of memory, run small neural network inference workloads. Within Edge AI and the broader field of artificial intelligence, it gives chip vendors, compiler teams and product engineers a common, reproducible yardstick for comparing TinyML hardware and software stacks.

The suite fixes the tasks, datasets, reference models and minimum quality targets, then reports latency per inference and, optionally, energy per inference measured under a controlled harness. Because every submission follows the same rules, published results let readers compare systems that would otherwise be described with incompatible marketing figures.

How the MLPerf Tiny benchmark suite is structured

MLPerf Tiny consists of a small set of representative tasks, each defined by a dataset, a reference model, a quality target and a measurement procedure. The tasks were chosen because they reflect what always-on sensing products actually do: listen for a word, detect a person, classify a small image or spot a machine fault.

The table below summarizes the four original benchmarks that have formed the core of the suite since its first published round in 2021.

BenchmarkDatasetReference modelQuality target
Keyword spottingGoogle Speech CommandsDS-CNN90% top-1 accuracy
Visual wake wordsVisual Wake Words (COCO-derived)MobileNetV1, width 0.2580% top-1 accuracy
Image classificationCIFAR-10ResNet-885% top-1 accuracy
Anomaly detectionToyADMOS (toy car)Fully connected autoencoder0.85 AUC
Core benchmarks of MLPerf Tiny

All reference models are deliberately compact, with tens to a few hundred thousand parameters, so that they fit in the flash and SRAM budgets of mainstream microcontrollers once they are quantized. Later rounds have extended the suite with additional scenarios, such as a streaming wake-word workload, while keeping the original four tasks for continuity.

Metrics, harness and measurement method

MLPerf Tiny reports single-stream latency as its primary performance metric: the time the device under test needs to process one input, measured over many repeated inferences and summarized as a median. Energy per inference, expressed in microjoules, is an optional but highly valued second metric because battery life is usually the deciding constraint for tiny devices.

Measurements run through a test harness developed with EEMBC, the embedded benchmarking consortium. A host computer sends inputs through an interface board, the device runs inference, and in energy mode an external energy monitor powers the device and records its consumption, while optocouplers or level shifters isolate the device so that the harness itself does not distort the reading.

MetricUnitWhat it revealsMain influences
LatencyMillisecondsResponsiveness of one inferenceClock speed, kernels, accelerator use
EnergyMicrojoules per inferenceBattery cost of each resultVoltage, memory traffic, idle power
Accuracy or AUCPercent or ratioWhether the model still worksQuantization, training, model changes
What each MLPerf Tiny metric captures

Accuracy is not a ranked metric: it is a pass-or-fail gate. A submission that misses the quality target is not valid, which prevents vendors from buying speed by quietly degrading the model below a useful level.

Closed and open divisions

MLPerf Tiny separates submissions into two divisions so that results answer different questions. The closed division compares hardware and software stacks running the same reference workload, while the open division shows what is possible when the model itself may change.

AspectClosed divisionOpen division
Model architectureReference model requiredAny architecture allowed
RetrainingNot permitted beyond allowed quantizationPermitted
Quality targetFixed per benchmarkMust be reported, may differ
Best use of resultsFair hardware comparisonShowcase of co-design
Comparison of the two submission divisions

Closed-division submitters may still apply post-training quantization or equivalent numeric changes, which is why almost every closed result runs INT8 arithmetic rather than 32-bit floating point. Open-division entries often combine hardware-aware neural architecture search, pruning or binarized layers, and they can be much faster, but they are not directly comparable with closed results.

How MLPerf Tiny relates to other edge benchmarks

MLPerf Tiny occupies the lowest power tier of the MLPerf family. MLPerf Mobile targets smartphones with neural processing units, and MLPerf Inference Edge targets larger edge servers and embedded GPUs, where workloads include object detection and language models that would never fit on a microcontroller.

Compared with vendor-specific figures such as peak operations per second, MLPerf Tiny measures end-to-end inference on a real model, including memory movement, kernel efficiency and runtime overhead. Peak throughput figures say little about a microcontroller, because small models are frequently limited by memory bandwidth and by operator support in the edge inference runtime rather than by raw arithmetic.

Reading results and common mistakes

A published MLPerf Tiny result describes a complete system: the chip, its clock configuration, the compiler or runtime, the kernel library and the exact model build. Changing any one of these can move latency by a large factor, so the system description matters as much as the headline number.

MistakeWhy it misleadsBetter practice
Comparing latency across divisionsModels differ in open entriesCompare closed results only
Ignoring clock frequencyFaster clocks raise power drawCheck the system description
Treating latency as energyA fast chip can still be power hungryUse the energy result where available
Assuming the result fits a productReal firmware adds sensors and radiosPrototype the full application
Typical misreadings of MLPerf Tiny results

Another frequent misunderstanding is that benchmark results reflect production-ready accuracy. The quality targets are deliberately modest, and a commercial keyword spotter usually needs more classes, noise robustness and far lower false-accept rates than the benchmark model provides.

Why MLPerf Tiny matters for TinyML engineering

MLPerf Tiny matters because it turned TinyML performance claims into verifiable, peer-reviewed numbers. Every result passes a review period in which other submitters can challenge it, which discourages benchmark-specific shortcuts that would not help real applications.

  • Silicon vendors use it to demonstrate microcontrollers and neural accelerators against competitors.
  • Compiler and runtime teams use the reference models as regression tests for kernel optimization and operator fusion.
  • Product engineers use published latency and energy figures to shortlist chips before building prototypes.
  • Researchers use the open division to show the gains of new compression and architecture methods.

The main limitation is coverage: a handful of tasks cannot represent every sensor workload, and many products run models that differ substantially from the references. The benchmark is therefore best treated as a well-controlled first filter, not as a substitute for profiling the actual application.

Explained without jargon

Many small gadgets, such as smart doorbells, fitness bands and factory sensors, now contain tiny chips that run artificial intelligence on their own. Each chip maker says its product is the fastest or the most efficient, but they often measure in different ways, which makes the claims hard to compare.

MLPerf Tiny works like a standardized fitness test for these chips. Every competitor runs the same exercises, such as recognizing a spoken word or spotting a person in a picture, under the same rules and with the same referees. The test records how quickly each chip answers and how much battery it uses, and it only counts answers that are correct often enough. The result is a fair scoreboard instead of a collection of advertising claims.

A typical use case

An engineering team designing a battery-powered voice-controlled light switch needs a microcontroller that can listen for a wake word around the clock and still run for about two years on two AA cells. The team shortlists three candidate chips and, instead of trusting datasheet figures, opens the latest MLPerf Tiny closed-division results for the keyword spotting benchmark.

One candidate reports a latency of a few milliseconds but a comparatively high energy figure per inference, while another is slower yet uses roughly a third of the energy. Because the switch only needs to classify about one audio window per second, latency is not critical, and the team selects the lower-energy chip. A two-week prototype with the real microphone and firmware confirms that the published ranking holds, and the team saves months of comparative testing by starting from a benchmark everyone can verify.