A hardware delegate is a pluggable component of an on-device inference runtime that takes over execution of supported operations from the default CPU kernels and runs them on a specialized accelerator such as a GPU, NPU or DSP. In Edge AI, where models run on phones, PCs and embedded boards instead of in the cloud, the delegate is the mechanism that turns a portable model file into hardware-accelerated inference without rewriting the model for every chip.
The term comes from the delegate API of TensorFlow Lite (now LiteRT), where it is also called an inference delegate. Other runtimes use different names for the same idea: ONNX Runtime calls it an execution provider, and some vendor toolkits call it a backend or plugin.
How a hardware delegate partitions the model graph
A hardware delegate works by claiming the subgraphs it can run and leaving the rest to the CPU. When the interpreter is initialized, it asks the delegate which operators and data types it supports. The delegate then groups consecutive supported nodes into delegated partitions, and each partition is replaced in the graph by a single delegate node that the runtime calls like any other operator.
Inside a delegated partition, the delegate compiles the operations for the target hardware, often through a vendor driver such as the Android Neural Networks API, Apple Core ML, Qualcomm QNN or OpenCL. Unsupported operations fall back to CPU kernels, so a model always runs, but every boundary between partitions forces a tensor copy between memory domains and a synchronization point.
The following table summarizes the stages a typical delegate goes through from model load to the first result.
| Stage | What the delegate does | Typical cost |
|---|---|---|
| Capability query | Reports supported operators, data types and shapes | Microseconds |
| Graph partitioning | Groups supported nodes into delegated subgraphs | Milliseconds |
| Kernel compilation | Builds shaders, NPU programs or driver graphs | Tens of ms to several seconds |
| Memory binding | Allocates device buffers and maps input and output tensors | Milliseconds |
| Invocation | Runs the subgraph on the accelerator and returns tensors | Per-inference latency |
Compilation is the expensive step, which is why mature delegates offer serialization or kernel caching. Caching the compiled program on disk can cut the cold-start time of the second launch from seconds to a fraction of a second.
Common types of hardware delegates
Hardware delegates differ mainly in which accelerator they target and which numeric precision they prefer. The CPU itself can also be wrapped by a delegate when an optimized kernel library replaces the reference implementation.
| Delegate | Target hardware | Preferred precision | Typical use |
|---|---|---|---|
| GPU delegate | Mobile GPUs via OpenCL, OpenGL ES, Metal or Vulkan | FP16, FP32 | Vision models, image segmentation |
| NNAPI delegate | Android NPUs, DSPs and GPUs through vendor drivers | INT8, FP16 | Broad Android coverage, now deprecated |
| Core ML delegate | Apple Neural Engine and GPU | FP16 | Models shipped in iOS apps |
| QNN or Hexagon delegate | Qualcomm Hexagon NPU and DSP | INT8, INT16 | Low-power always-on sensing |
| XNNPACK | Arm and x86 CPU cores with SIMD | FP32, FP16, INT8 | Default fast CPU path |
| Microcontroller kernels | Arm Cortex-M with CMSIS-NN or Ethos-U | INT8 | TinyML and keyword spotting |
Because Android deprecated the Neural Networks API in recent releases, developers increasingly use vendor-specific delegates directly. This improves operator coverage on one chip family but reduces portability, which is a recurring tension in edge deployment.
Hardware delegate compared with related concepts
A hardware delegate is a runtime-level acceleration hook, which distinguishes it from the broader runtime and from the offline steps that prepare a model. Several neighboring concepts in edge inference are often confused with it.
| Concept | What it is | Relation to the delegate |
|---|---|---|
| Edge inference runtime | The engine that loads and executes the model | Hosts one or more delegates |
| Execution provider | ONNX Runtime name for an accelerator backend | Functional equivalent |
| Model conversion | Offline translation to a runtime format | Happens before delegation |
| Operator fusion | Merging several operators into one kernel | Often performed inside the delegate |
| Post-training quantization | Reducing weights and activations to INT8 or lower | Determines which delegates can run the model |
| Model offloading | Moving work to another device or the cloud | Delegation stays on the same device |
The key difference from model offloading is locality: a delegate moves computation between processors of one system on a chip, so data never leaves the device and network latency does not apply.
Choosing and configuring a hardware delegate
The right hardware delegate depends on the model’s operator set, its precision and the target device fleet. Engineers usually benchmark several options on real devices, because published peak TOPS figures rarely predict end-to-end latency.
| Situation | Recommended choice | Reason |
|---|---|---|
| FP16 vision model on many Android phones | GPU delegate | Wide availability and good float coverage |
| Fully INT8 model on a known Qualcomm device | QNN delegate | Highest efficiency on the NPU |
| iOS app with a convolutional model | Core ML delegate | Access to the Neural Engine |
| Transformer with dynamic shapes | XNNPACK on CPU | Accelerators often reject dynamic shapes |
| Microcontroller under 1 MB RAM | CMSIS-NN kernels | No separate accelerator memory |
Common configuration options include allowing FP16 precision loss on GPUs, setting a maximum number of delegated partitions, enabling a kernel cache directory and choosing a power or performance preference. A robust application also implements a fallback chain: try the NPU delegate, then the GPU delegate, then the CPU, and record which path succeeded.
Limitations and common mistakes
The most frequent problem is graph fragmentation. If a single unsupported operator sits in the middle of a network, the delegate splits the model into two partitions with a CPU island between them, and the transfer overhead can exceed the speedup. In such cases a delegated model may run slower than plain CPU inference.
- Numerical drift: FP16 or vendor INT8 kernels may produce slightly different outputs than the reference CPU path, so accuracy must be validated per delegate.
- Driver variability: the same delegate can behave differently across chipsets and operating system versions, which makes device-farm testing essential.
- Initialization latency: compiling kernels at startup can freeze the first frame if caching is disabled.
- Thermal throttling: sustained accelerator load raises temperature, so short benchmarks overstate long-run throughput.
- Silent fallback: a runtime may quietly execute everything on the CPU when delegation fails, so logging the number of delegated nodes is a basic safeguard.
Mitigations include replacing unsupported operators during model conversion, applying quantization that matches the accelerator’s native precision and using hardware-aware architecture choices such as MobileNet-style blocks that delegates support well.
In plain words
A phone contains several kinds of processors. The main processor can do almost anything, but special chips for graphics or AI can do certain math much faster while using less battery. A hardware delegate is the go-between that hands suitable pieces of an AI model to those special chips.
Think of a general contractor who passes the wiring to an electrician and the plumbing to a plumber. The job gets done faster, but every handover takes time, so the work should be split into as few handovers as possible. When a specialist cannot do a task, the contractor simply does it alone.
A real-world example
A team building a plant-disease detection app for farmers ships a quantized image classifier of about 4 MB. On a mid-range Android phone, the CPU path with four threads needs roughly 45 ms per photo, which is acceptable for single shots but too slow for a live camera preview.
The engineers enable the GPU delegate with FP16 precision and discover that one custom resize operator keeps falling back to the CPU, creating three separate partitions. After replacing that operator with a standard one during conversion, the whole graph is delegated as a single partition. Latency drops to around 12 ms, the preview runs smoothly at 30 frames per second, and kernel caching reduces app startup delay from 1.8 seconds to under 0.3 seconds. The team keeps the CPU path as a fallback for older phones whose GPU drivers fail validation.