An NPU, or neural processing unit, is a specialized processor designed to execute neural network inference with far higher energy efficiency than a general-purpose CPU. Among processors, the NPU is a domain-specific accelerator: it gives up flexibility in exchange for very fast, low-power matrix multiplication and convolution on low-precision numbers.
In consumer hardware, an NPU is usually an integrated block on a system on a chip, sitting beside the CPU cores and the GPU in smartphones, laptops, cameras and cars. It handles tasks such as background blur in video calls, speech recognition, photo enhancement and on-device language models without draining the battery.
How an NPU accelerates neural networks
Neural network inference consists mostly of multiply-accumulate (MAC) operations on large arrays of weights and activations. An NPU contains hundreds or thousands of MAC units arranged in arrays or systolic structures, so that a single clock cycle performs many multiplications in parallel while data flows between neighboring units.
Two design choices give the NPU its efficiency. First, it uses low-precision arithmetic such as INT8, INT4 or FP16 instead of 32-bit floating point, which shrinks both the circuits and the memory traffic. Second, it keeps data in large on-chip SRAM buffers close to the compute units, because moving data from external DRAM costs far more energy than the arithmetic itself.
The result is a block that is excellent at dense tensor operations but cannot run arbitrary code. Unsupported layers fall back to the CPU or GPU, which is why software support matters as much as raw performance.
Key NPU parameters and what they mean
NPU performance is usually quoted in TOPS, trillions of operations per second, but this single number hides several assumptions. The table explains the parameters that matter when comparing NPUs.
| Parameter | Typical range | What it affects |
|---|---|---|
| Peak TOPS | About 10 to 80 in phones and laptops | Theoretical upper limit of throughput |
| Numeric precision | INT4, INT8, FP16, BF16 | Speed, memory use and model accuracy |
| TOPS per watt | Often several times a CPU or GPU | Battery life during sustained AI work |
| On-chip memory | A few megabytes to tens of megabytes | How much of a model avoids DRAM access |
| Memory bandwidth | Shared with the system memory | Speed of large models such as LLMs |
| Operator support | Varies by vendor and driver | Whether a model runs fully on the NPU |
TOPS figures are almost always given for INT8 or lower precision at peak clock. Two NPUs with the same TOPS can differ greatly in real speed if one is limited by memory bandwidth or by poor compiler support. The Copilot+ PC category, for example, set a threshold of 40 or more NPU TOPS, which is a useful floor but not a guarantee of application performance.
NPU compared with CPU, GPU and TPU
Each processor type handles AI workloads with a different balance of flexibility, throughput and power. Modern systems use all of them together and assign each task to the most suitable unit.
| Processor | Strength | Weakness | Typical AI role |
|---|---|---|---|
| CPU | Runs any code, low latency for small tasks | Low parallel throughput | Control logic, small models, fallback |
| GPU | Massive parallelism, mature software | High power draw | Training, large models, bursts of inference |
| NPU | Very high efficiency on fixed tensor tasks | Limited operator support | Sustained, low-power on-device inference |
| TPU | Very high throughput at data-center scale | Available mainly in the cloud | Training and serving large models |
| DSP | Efficient signal processing | Narrow programming model | Audio, sensor and camera pipelines |
The term AI accelerator covers all of these specialized designs. In practice, NPU usually means a client-side, power-constrained accelerator, while TPU and similar names refer to data-center chips. A laptop GPU may still outperform its NPU in raw speed, but the NPU often completes the same task at a fraction of the energy.
Software stack and practical use
An NPU is only useful when applications can reach it through a runtime and compiler that convert a trained model into instructions the hardware understands. Typical routes include ONNX Runtime with vendor execution providers, Windows ML, Core ML on Apple devices, Android NNAPI successors and vendor toolkits such as OpenVINO or QNN.
Before deployment, models are usually quantized from 32-bit or 16-bit floating point to INT8 or INT4. Quantization can reduce accuracy slightly, so developers validate results against the original model. The following table shows which workloads suit an NPU best.
| Workload | Best unit | Reason |
|---|---|---|
| Video call background effects | NPU | Continuous, steady load where power matters |
| Live captions and transcription | NPU | Long-running, battery sensitive |
| Small on-device language model | NPU or GPU | Depends on memory bandwidth and support |
| Image generation | GPU | Large models and high peak compute |
| Model training | GPU or TPU | Needs high precision and flexibility |
Limitations and common misconceptions
The NPU is frequently overestimated in marketing and underused in practice. Experienced engineers watch for these issues:
- TOPS is not real speed: sustained performance depends on memory, precision and software.
- Not all layers run on the NPU: unsupported operators trigger slow transfers back to the CPU or GPU.
- Fragmented software: each vendor exposes its NPU differently, so portable applications remain harder to build.
- Large models hit memory limits: a language model with billions of parameters is bound by system memory bandwidth, not by NPU compute.
- An idle NPU adds nothing: without applications that use it, the block simply sits unused.
A plain-language summary
An NPU is a small specialist inside the main chip of a phone or laptop. The CPU is like a skilled generalist who can do any job, and the GPU is a large crew that handles heavy visual work. The NPU is a worker trained for one kind of repetitive math used by artificial intelligence.
Because it does only that job, it finishes AI tasks quickly while using very little battery, which lets features like voice typing and camera effects run all day without the device getting hot.
An example from practice
A sales consultant spends most of the day in video calls on a thin laptop with an NPU rated at about 45 TOPS. Background blur, eye contact correction, noise suppression and live captions all run continuously during each call.
When these effects run on the NPU, CPU load stays low and the fans remain quiet. Measured over a working day, the laptop lasts noticeably longer than when the same effects are forced onto the integrated GPU, because the NPU completes the work at much lower power.
The consultant also uses a small on-device assistant to summarize meeting notes. That model fits in memory and runs partly on the NPU, so confidential notes never leave the laptop.