On-device AI is the practice of running machine learning inference locally on the end-user device, such as a smartphone, laptop, car, camera or sensor, rather than sending input data to a remote server. Within AI infrastructure, it is the deployment option at the opposite end from cloud model serving: the model weights, the computation and usually the data never leave the device.
The approach is closely related to edge AI, a broader term that also includes gateways and local servers near the data source. On-device AI is attractive for privacy, latency, offline operation and cost, but it forces models to fit within tight limits of memory, power and heat.
How on-device AI executes a model locally
On-device AI executes a pre-trained, compressed model with a local runtime that maps neural network operations onto the available processors. Training almost always happens in a data center; the device performs inference only, sometimes with light personalization.
Modern devices contain several compute units, and the runtime decides which one runs each part of the model. The table below compares them.
| Processor | Strength | Limitation | Typical workload |
|---|---|---|---|
| CPU | Universal support for every operation | Lowest efficiency per watt | Small models and fallback layers |
| GPU | High parallel throughput | Higher power draw on mobile | Vision models and image generation |
| NPU | Best performance per watt | Supports a limited operator set | Continuous speech, camera and language tasks |
| DSP | Very low power for signal tasks | Narrow programming model | Wake-word detection and audio filtering |
A neural processing unit (NPU) is often the decisive component. When a model contains operators the NPU does not support, the runtime must split the graph and fall back to the CPU, and this graph partitioning can erase most of the expected speed gain.
Making models fit on the device
Models must be made smaller and cheaper through model compression before they run well on a device. Memory bandwidth, rather than raw compute, is usually the bottleneck for language models, because every generated token requires reading most of the weights.
The following table summarizes the main compression techniques and their trade-offs.
| Technique | What it does | Typical saving | Main risk |
|---|---|---|---|
| Quantization | Stores weights in 8, 4 or fewer bits | About 2 to 4 times smaller than 16-bit | Accuracy loss at very low bit widths |
| Pruning | Removes low-importance weights or channels | Varies widely by model | Speedup only with hardware support for sparsity |
| Knowledge distillation | Trains a small student to imitate a large teacher | Often an order of magnitude fewer parameters | Student loses rare skills |
| Efficient architectures | Uses designs built for mobile hardware | Lower compute per inference | Requires retraining from scratch |
As a rough guide, a language model with 3 billion parameters at 4-bit precision needs about 1.5 to 2 GB of memory for weights alone, plus space for the KV cache. That is feasible on recent flagship phones but leaves little room for other applications.
On-device AI compared with cloud inference
On-device AI and cloud inference represent a trade-off between capability and locality. Cloud servers run the largest models on GPU clusters, while devices run smaller models with no network dependency.
| Factor | On-device AI | Cloud inference |
|---|---|---|
| Latency | No network round trip | Adds network delay and queueing |
| Privacy | Data can stay on the device | Data travels to and is processed by a provider |
| Model size | Limited by device memory | Very large models are possible |
| Operating cost | Paid by the user hardware and battery | Paid per token or per GPU hour |
| Updates | Model shipped with app or OS updates | Model changed centrally at any time |
| Offline use | Works without connectivity | Requires a network connection |
Many products use a hybrid design: a local model handles frequent, simple or sensitive requests, and a model routing layer forwards hard requests to the cloud. The routing decision itself can run on the device, which keeps sensitive requests local by default.
Where on-device AI is used and how to choose it
On-device AI is the right choice when latency, privacy or connectivity matter more than maximum model quality. Typical applications include keyboard prediction, speech recognition, camera processing, translation, health sensing and local assistants.
| Situation | Recommended choice | Reason |
|---|---|---|
| Real-time camera or audio processing | On-device | Frame rates cannot tolerate network delay |
| Health or biometric data | On-device | Reduces exposure of sensitive data |
| Complex reasoning over long documents | Cloud | Needs a larger model and more memory |
| Mixed everyday assistant requests | Hybrid routing | Balances quality, cost and privacy |
| Industrial sensors with poor connectivity | On-device or edge gateway | Must keep working offline |
Before committing, teams should profile the model on the slowest supported device, not on a developer flagship. Measured tokens per second, peak memory and sustained temperature decide whether the feature is usable.
Limitations and common mistakes
On-device AI shifts several problems from the data center to millions of heterogeneous devices. The most serious limitation is hardware fragmentation: the same model can run quickly on one chipset and fail to load on another.
- Thermal throttling: sustained inference heats the device, and the system lowers clock speeds, so benchmark numbers from short runs overstate real performance.
- Battery drain: continuous inference on the CPU or GPU can consume power noticeably faster than on the NPU.
- Model extraction: weights shipped on the device can be copied and analyzed, which matters for proprietary models.
- Limited monitoring: without telemetry, model drift and failures are hard to detect, and collecting telemetry partly undermines the privacy benefit.
- Privacy overstatement: local inference does not guarantee privacy if results, logs or embeddings are later uploaded.
A common mistake is to evaluate only the full-precision model and assume the quantized version behaves the same. Accuracy must be re-measured after compression on the target runtime, because rounding errors and operator fallbacks can change outputs in ways that are invisible on a server.
Explained without jargon
On-device AI means the smart features on your phone do their thinking inside the device itself. Your photos, voice and text do not travel to a distant computer. It is like carrying a pocket dictionary instead of phoning a library for every word.
The benefits are speed, privacy and working without internet. The catch is size. A phone has far less memory and power than a data center, so the AI must be smaller and simpler. Very demanding tasks may still go to the cloud.
A typical use case
A hospital equips its nurses with tablets that transcribe spoken shift notes. Because the notes contain patient information, the IT team chooses an on-device speech recognition model quantized to 8 bits, which runs on the tablet NPU and needs no network connection on the wards, where wireless coverage is unreliable.
During a pilot on the oldest tablet model in use, the team finds that transcription slows down after twenty minutes because of thermal throttling, so it limits continuous dictation to short segments. After rollout, audio never leaves the tablet, transcripts appear in under a second, and only the reviewed text notes are synchronized with the records system, which satisfies the privacy review without a cloud processing agreement.