A mobile NPU (neural processing unit) is a dedicated AI accelerator integrated into a smartphone or tablet system-on-chip (SoC) that executes neural network inference, mainly matrix and convolution operations, with far higher energy efficiency than the CPU or GPU. In mobile hardware it sits alongside the CPU cores, the mobile GPU, the image signal processor and the modem on the same die, sharing memory with them.
Vendors use different names for the same idea, such as Neural Engine, Hexagon NPU or AI processing unit. Whatever the name, its purpose is to run features like on-device speech recognition, photo processing, translation and small language models locally, quickly and within a phone’s strict power and thermal budget of only a few watts.
How a mobile NPU accelerates neural networks
Neural network inference is dominated by multiply-accumulate (MAC) operations on large tensors. A mobile NPU contains arrays of MAC units, often organized as systolic arrays or vector-matrix engines, that perform thousands of these operations per clock cycle while keeping data in fast local SRAM instead of fetching it repeatedly from main memory.
The efficiency comes from specialization and low precision. NPUs are optimized for INT8 and INT4 integer arithmetic and FP16, which use less energy and memory bandwidth than the FP32 math common on CPUs. Moving data, not computing it, is the largest energy cost, so NPU designs focus on data reuse and on-chip buffers.
The table shows the main building blocks found in typical mobile NPU designs.
| Block | Function | Effect on performance |
|---|---|---|
| MAC array | Parallel multiply-accumulate on tensors | Sets peak operations per second |
| Local SRAM | Holds weights and activations near compute | Cuts memory traffic and energy |
| Vector or scalar unit | Activation functions, normalization | Handles operations outside matrix math |
| DMA engine | Moves data between DRAM and SRAM | Hides memory latency |
| Microcontroller and firmware | Schedules layers and operations | Determines utilization in practice |
Key parameters of a mobile NPU
NPU performance is usually advertised in TOPS (trillions of operations per second). The figure is useful only with context, because vendors may count INT8 or INT4 operations, sparse or dense math, and peak rather than sustained throughput. TOPS per watt and real benchmark results are more meaningful.
| Parameter | Typical range in recent phones | What it affects |
|---|---|---|
| Peak throughput | Tens of TOPS, varying by precision | Maximum model size and speed |
| Supported precisions | INT4, INT8, FP16, sometimes FP8 | Model accuracy and efficiency |
| Power draw | Roughly one to a few watts under load | Battery life and heat |
| Memory bandwidth | Shared LPDDR5 or LPDDR5X | Speed of large language models |
| On-device model size | Up to a few billion parameters, quantized | Which generative features run locally |
For generative AI, memory bandwidth often matters more than TOPS. Generating each token of a language model requires reading most of the model weights from DRAM, so a phone with faster LPDDR memory may produce text faster even if its NPU has a lower peak rating.
Mobile NPU compared with CPU and mobile GPU
A smartphone can run machine learning on three main processors, and frameworks often split a model across them. The choice depends on the model’s operators, precision and latency needs.
| Processor | Strength | Weakness | Typical ML role |
|---|---|---|---|
| CPU | Flexible, supports every operator | Low efficiency for large tensors | Small models, unsupported layers |
| Mobile GPU | High parallel throughput, FP16 | Higher power, shared with graphics | Vision models, flexible acceleration |
| Mobile NPU | Best performance per watt | Limited operator and format support | Sustained or always-on inference |
| DSP or ISP | Very low power for signals | Narrow programmability | Audio wake words, camera pipelines |
The NPU wins clearly for sustained workloads, such as continuous camera effects or transcription, where heat would force the GPU to throttle. The CPU remains the fallback for unsupported operations, and frequent fallbacks can erase the NPU’s advantage because data must cross between processors.
Software access and practical use
Developers rarely program a mobile NPU directly. Instead they use frameworks and delegates that compile a trained model for the available hardware: Core ML on Apple devices, LiteRT (formerly TensorFlow Lite) with hardware delegates and vendor SDKs on Android, and ONNX Runtime with execution providers on several platforms.
- Train or select a model with operators known to be supported on the target NPUs.
- Quantize the model to INT8 or INT4, using calibration data to limit accuracy loss.
- Convert and compile it with the platform tool, checking which layers fall back to the CPU.
- Profile on real devices, measuring latency, power and temperature over minutes, not seconds.
- Keep a fallback path on the GPU or CPU for devices without a capable NPU.
Because NPU architectures differ between chip vendors and generations, performance portability is limited. A model that runs well on one phone may fall back heavily on another, so testing across a representative device matrix is essential.
Limitations and common misconceptions
The mobile NPU is often marketed as a single number, which hides important practical constraints.
| Misconception | Reality | Practical consequence |
|---|---|---|
| Higher TOPS means faster AI | Precision, sparsity and bandwidth differ | Compare real benchmarks instead |
| Every model runs on the NPU | Unsupported operators fall back | Design models for the hardware |
| Quantization is free | Accuracy can drop noticeably | Validate quality after conversion |
| Peak speed is sustained | Thermal limits reduce clocks | Test long sessions, not short runs |
| NPU replaces cloud AI | Memory limits model size | Large models still run in data centers |
In simple terms
A mobile NPU is a small, specialized brain for AI inside your phone’s main chip. The phone’s general processor can do almost anything, but slowly for AI tasks. The NPU does one kind of math very quickly and with little power.
It is like using a dishwasher instead of washing every plate by hand. Both get the job done, but the dishwasher is built for that one task. That is why your phone can blur photo backgrounds, translate speech and transcribe voice notes without sending everything to the internet or draining the battery.
A practical example
A developer builds a note-taking app that transcribes meetings on the device for privacy reasons. The first version runs a speech recognition model on the CPU, and a one-hour meeting makes the phone noticeably warm and drains a large share of the battery.
The developer quantizes the model to INT8, converts it with the platform’s NPU tools and replaces two unsupported layers with supported equivalents. On recent phones, the model now runs almost entirely on the NPU, transcription keeps pace with live speech, and power use falls substantially during long recordings. Older devices without a capable NPU automatically use the GPU path instead.