On-device AI

Also known as: edge AI

On this page 7

On-device AI is the practice of running machine learning inference locally on the end-user device, such as a smartphone, laptop, car, camera or sensor, rather than sending input data to a remote server. Within AI infrastructure, it is the deployment option at the opposite end from cloud model serving: the model weights, the computation and usually the data never leave the device.

The approach is closely related to edge AI, a broader term that also includes gateways and local servers near the data source. On-device AI is attractive for privacy, latency, offline operation and cost, but it forces models to fit within tight limits of memory, power and heat.

How on-device AI executes a model locally

On-device AI executes a pre-trained, compressed model with a local runtime that maps neural network operations onto the available processors. Training almost always happens in a data center; the device performs inference only, sometimes with light personalization.

Modern devices contain several compute units, and the runtime decides which one runs each part of the model. The table below compares them.

ProcessorStrengthLimitationTypical workload
CPUUniversal support for every operationLowest efficiency per wattSmall models and fallback layers
GPUHigh parallel throughputHigher power draw on mobileVision models and image generation
NPUBest performance per wattSupports a limited operator setContinuous speech, camera and language tasks
DSPVery low power for signal tasksNarrow programming modelWake-word detection and audio filtering
Processors used for on-device inference

A neural processing unit (NPU) is often the decisive component. When a model contains operators the NPU does not support, the runtime must split the graph and fall back to the CPU, and this graph partitioning can erase most of the expected speed gain.

Making models fit on the device

Models must be made smaller and cheaper through model compression before they run well on a device. Memory bandwidth, rather than raw compute, is usually the bottleneck for language models, because every generated token requires reading most of the weights.

The following table summarizes the main compression techniques and their trade-offs.

TechniqueWhat it doesTypical savingMain risk
QuantizationStores weights in 8, 4 or fewer bitsAbout 2 to 4 times smaller than 16-bitAccuracy loss at very low bit widths
PruningRemoves low-importance weights or channelsVaries widely by modelSpeedup only with hardware support for sparsity
Knowledge distillationTrains a small student to imitate a large teacherOften an order of magnitude fewer parametersStudent loses rare skills
Efficient architecturesUses designs built for mobile hardwareLower compute per inferenceRequires retraining from scratch
Compression techniques for on-device models

As a rough guide, a language model with 3 billion parameters at 4-bit precision needs about 1.5 to 2 GB of memory for weights alone, plus space for the KV cache. That is feasible on recent flagship phones but leaves little room for other applications.

On-device AI compared with cloud inference

On-device AI and cloud inference represent a trade-off between capability and locality. Cloud servers run the largest models on GPU clusters, while devices run smaller models with no network dependency.

FactorOn-device AICloud inference
LatencyNo network round tripAdds network delay and queueing
PrivacyData can stay on the deviceData travels to and is processed by a provider
Model sizeLimited by device memoryVery large models are possible
Operating costPaid by the user hardware and batteryPaid per token or per GPU hour
UpdatesModel shipped with app or OS updatesModel changed centrally at any time
Offline useWorks without connectivityRequires a network connection
On-device and cloud inference compared

Many products use a hybrid design: a local model handles frequent, simple or sensitive requests, and a model routing layer forwards hard requests to the cloud. The routing decision itself can run on the device, which keeps sensitive requests local by default.

Where on-device AI is used and how to choose it

On-device AI is the right choice when latency, privacy or connectivity matter more than maximum model quality. Typical applications include keyboard prediction, speech recognition, camera processing, translation, health sensing and local assistants.

SituationRecommended choiceReason
Real-time camera or audio processingOn-deviceFrame rates cannot tolerate network delay
Health or biometric dataOn-deviceReduces exposure of sensitive data
Complex reasoning over long documentsCloudNeeds a larger model and more memory
Mixed everyday assistant requestsHybrid routingBalances quality, cost and privacy
Industrial sensors with poor connectivityOn-device or edge gatewayMust keep working offline
Decision guide for on-device deployment

Before committing, teams should profile the model on the slowest supported device, not on a developer flagship. Measured tokens per second, peak memory and sustained temperature decide whether the feature is usable.

Limitations and common mistakes

On-device AI shifts several problems from the data center to millions of heterogeneous devices. The most serious limitation is hardware fragmentation: the same model can run quickly on one chipset and fail to load on another.

  • Thermal throttling: sustained inference heats the device, and the system lowers clock speeds, so benchmark numbers from short runs overstate real performance.
  • Battery drain: continuous inference on the CPU or GPU can consume power noticeably faster than on the NPU.
  • Model extraction: weights shipped on the device can be copied and analyzed, which matters for proprietary models.
  • Limited monitoring: without telemetry, model drift and failures are hard to detect, and collecting telemetry partly undermines the privacy benefit.
  • Privacy overstatement: local inference does not guarantee privacy if results, logs or embeddings are later uploaded.

A common mistake is to evaluate only the full-precision model and assume the quantized version behaves the same. Accuracy must be re-measured after compression on the target runtime, because rounding errors and operator fallbacks can change outputs in ways that are invisible on a server.

Explained without jargon

On-device AI means the smart features on your phone do their thinking inside the device itself. Your photos, voice and text do not travel to a distant computer. It is like carrying a pocket dictionary instead of phoning a library for every word.

The benefits are speed, privacy and working without internet. The catch is size. A phone has far less memory and power than a data center, so the AI must be smaller and simpler. Very demanding tasks may still go to the cloud.

A typical use case

A hospital equips its nurses with tablets that transcribe spoken shift notes. Because the notes contain patient information, the IT team chooses an on-device speech recognition model quantized to 8 bits, which runs on the tablet NPU and needs no network connection on the wards, where wireless coverage is unreliable.

During a pilot on the oldest tablet model in use, the team finds that transcription slows down after twenty minutes because of thermal throttling, so it limits continuous dictation to short segments. After rollout, audio never leaves the tablet, transcripts appear in under a second, and only the reviewed text notes are synchronized with the records system, which satisfies the privacy review without a cloud processing agreement.