TinyML

Also known as: Tiny machine learning

On this page 7

TinyML, short for tiny machine learning, is the practice of running trained machine learning models directly on microcontrollers and other ultra-low-power hardware, typically with less than 1 MB of flash, a few hundred kilobytes of RAM and a power budget measured in milliwatts or less. Within Edge AI, it occupies the most constrained end of the spectrum: where phones and PCs run models with gigabytes of memory, TinyML devices must perform inference with resources roughly a thousand times smaller.

The goal is always-on intelligence at the sensor. A TinyML device listens, watches or measures continuously, interprets the signal locally and only wakes a larger processor or sends data to the cloud when something relevant happens. This makes artificial intelligence possible in battery-powered and energy-harvesting products where a radio link or a full application processor would be too costly in energy, latency or privacy.

How TinyML runs a model on a microcontroller

A TinyML system runs a pre-trained, heavily compressed model inside the firmware of a microcontroller unit (MCU). Training happens on a workstation or in the cloud; the device only performs inference, usually with 8-bit integer arithmetic because most MCUs lack a fast floating-point unit for neural workloads.

The typical pipeline has four stages. Raw sensor data is captured by an analog front end, converted into compact features such as a spectrogram of one second of audio, passed through a small neural network and turned into a decision. Because the model weights live in flash memory and the intermediate activations must fit in SRAM, the peak activation size often limits the design more than the parameter count.

Execution relies on a lightweight interpreter or compiled runtime. Interpreter-based runtimes such as TensorFlow Lite for Microcontrollers parse a serialized model and call optimized kernels, while ahead-of-time compilers generate plain C code for the specific network, removing interpreter overhead and saving memory. Vendor kernel libraries such as CMSIS-NN supply hand-optimized operators for Arm Cortex-M cores.

The table below summarizes the resource envelope that distinguishes TinyML hardware from other inference targets.

ResourceTypical TinyML rangeWhat it constrains
Flash memory128 KB to 2 MBModel weights and firmware code
SRAM16 KB to 512 KBPeak activations and input buffers
Clock speed32 MHz to 480 MHzInference latency
Active power1 mW to 100 mWBattery life and duty cycle
Model size10 KB to 500 KBNetwork depth and width
Typical resource envelope of TinyML hardware

Model compression techniques that make TinyML possible

TinyML depends on aggressive model compression, because even a small mobile network is too large for an MCU without it. Practitioners combine several techniques and measure the effect of each on accuracy, memory and latency.

Quantization is almost always applied first. Converting 32-bit floating-point weights and activations to INT8 cuts model size by about four times and enables integer-only kernels. Quantization-aware training recovers most of the accuracy lost by post-training quantization when a network is sensitive. Pruning, weight clustering and compact architectures designed through hardware-aware neural architecture search reduce the remaining footprint further.

The following table compares the main compression methods used for microcontroller targets.

TechniqueTypical savingMain trade-off
INT8 quantizationAbout 4x smaller, 2-4x fasterSmall accuracy loss, calibration needed
Structured pruning30-70% fewer operationsRequires fine-tuning to recover accuracy
Weight clusteringBetter compression of stored weightsNo speed gain without special kernels
Compact architectures10-100x fewer parameters than standard networksLower ceiling on task complexity
Binarized networksUp to 32x smaller weightsSignificant accuracy loss on hard tasks
Compression methods used in TinyML

TinyML compared with other edge inference approaches

TinyML differs from general edge inference mainly in scale and in the operating model. A smartphone running a local language model has a neural processing unit, an operating system and gigabytes of memory; a TinyML device often has no operating system, a fixed firmware image and a single dedicated task.

The comparison below places TinyML among neighboring deployment patterns in the Edge AI landscape.

ApproachTypical hardwareModel scaleTypical task
TinyMLMicrocontroller, DSP, micro-NPUKilobytesKeyword spotting, anomaly detection
Mobile edge inferencePhone or laptop SoC with NPUMegabytes to gigabytesImage enhancement, local LLM
Edge server inferenceGateway or GPU box on siteGigabytesMulti-camera video analytics
Cloud offloadingRemote data centerUnlimited in practiceLarge generative models
TinyML among edge deployment options

TinyML is often combined with these approaches in a cascade: the microcontroller filters the signal around the clock, and only confirmed events trigger a heavier model on a phone, a gateway or a cloud service. This pattern underlies AIoT products and many wake-word systems.

Typical TinyML applications and how to choose a target

TinyML suits problems where the input is a continuous sensor stream and the output is a short classification or alarm. Audio, vibration, motion and low-resolution vision dominate, because these signals compress well into small feature maps.

Choosing a target device starts from the latency, energy and accuracy requirements, not from the model. The decision table below summarizes common situations.

SituationRecommended setupReason
Wake word on a coin-cell deviceCortex-M4 or M33 with INT8 modelSub-milliwatt always-on listening
Predictive maintenance on motorsMCU with accelerometer and small 1D CNNLocal vibration analysis avoids constant radio use
Person detection at 96×96 pixelsMCU with micro-NPU or Cortex-M55Vision needs vector acceleration
Gesture recognition in wearablesLow-power MCU with IMU featuresSmall time-series model, long battery life
Task needs open-ended languageNot TinyML; use phone or cloudModel far exceeds MCU memory
Decision guide for TinyML deployments

Benchmark suites such as MLPerf Tiny give comparable numbers for latency and energy per inference across hardware, which helps teams avoid relying on vendor claims measured under different conditions.

Limitations and common mistakes in TinyML projects

The central limitation of TinyML is model capacity. Networks with tens of thousands of parameters handle narrow tasks well but cannot generalize like large models, so dataset quality and domain match matter more than architecture tweaks. A keyword spotter trained on clean studio recordings often fails in a noisy factory.

  • Ignoring peak RAM: a model that fits in flash can still fail because one layer’s activations exceed available SRAM.
  • Mismatched preprocessing: feature extraction on the device must match training exactly, or accuracy collapses silently.
  • Measuring only inference energy: sensor sampling, memory access and radio use often dominate the total power budget.
  • Skipping update planning: without a secure over-the-air model update path, field errors cannot be corrected.
  • Overlooking security: model files and firmware on unprotected MCUs can be extracted or altered.

On-device training remains limited on microcontrollers. Most TinyML products ship a frozen model, and personalization, when present, adjusts only a small final layer or a threshold.

Stripped to the essentials

TinyML puts a very small piece of artificial intelligence inside a cheap chip, the kind that runs a thermostat or a fitness band. The chip has little memory and almost no power to spare, so the model must be shrunk until it fits, much like packing for a long trip with only a small backpack.

Once it fits, the chip can listen, feel or watch on its own, all day, on a small battery. It does not need the internet to decide whether it heard a command or felt an unusual vibration. The key idea is simple: decide locally, send only what matters.

Where it is applied

A water utility installs battery-powered vibration sensors on pumps at remote pumping stations. Each sensor contains a Cortex-M4 microcontroller running a small 1D convolutional network of about 40 KB, quantized to INT8, which analyzes one second of accelerometer data every minute.

The model classifies the vibration pattern as normal operation, bearing wear or cavitation. Only when it detects an anomaly several times in a row does the device switch on its low-power radio and send a short alert with a compact summary of the signal. Raw data never leaves the sensor in normal operation.

Because the radio is idle most of the time, the sensors run for several years on two AA batteries. Maintenance crews receive targeted alerts days before failures, and unplanned pump outages fall noticeably compared with fixed inspection schedules.