Hardware delegate

Also known as: inference delegate

On this page 7

A hardware delegate is a pluggable component of an on-device inference runtime that takes over execution of supported operations from the default CPU kernels and runs them on a specialized accelerator such as a GPU, NPU or DSP. In Edge AI, where models run on phones, PCs and embedded boards instead of in the cloud, the delegate is the mechanism that turns a portable model file into hardware-accelerated inference without rewriting the model for every chip.

The term comes from the delegate API of TensorFlow Lite (now LiteRT), where it is also called an inference delegate. Other runtimes use different names for the same idea: ONNX Runtime calls it an execution provider, and some vendor toolkits call it a backend or plugin.

How a hardware delegate partitions the model graph

A hardware delegate works by claiming the subgraphs it can run and leaving the rest to the CPU. When the interpreter is initialized, it asks the delegate which operators and data types it supports. The delegate then groups consecutive supported nodes into delegated partitions, and each partition is replaced in the graph by a single delegate node that the runtime calls like any other operator.

Inside a delegated partition, the delegate compiles the operations for the target hardware, often through a vendor driver such as the Android Neural Networks API, Apple Core ML, Qualcomm QNN or OpenCL. Unsupported operations fall back to CPU kernels, so a model always runs, but every boundary between partitions forces a tensor copy between memory domains and a synchronization point.

The following table summarizes the stages a typical delegate goes through from model load to the first result.

StageWhat the delegate doesTypical cost
Capability queryReports supported operators, data types and shapesMicroseconds
Graph partitioningGroups supported nodes into delegated subgraphsMilliseconds
Kernel compilationBuilds shaders, NPU programs or driver graphsTens of ms to several seconds
Memory bindingAllocates device buffers and maps input and output tensorsMilliseconds
InvocationRuns the subgraph on the accelerator and returns tensorsPer-inference latency
Lifecycle of a hardware delegate in an inference runtime

Compilation is the expensive step, which is why mature delegates offer serialization or kernel caching. Caching the compiled program on disk can cut the cold-start time of the second launch from seconds to a fraction of a second.

Common types of hardware delegates

Hardware delegates differ mainly in which accelerator they target and which numeric precision they prefer. The CPU itself can also be wrapped by a delegate when an optimized kernel library replaces the reference implementation.

DelegateTarget hardwarePreferred precisionTypical use
GPU delegateMobile GPUs via OpenCL, OpenGL ES, Metal or VulkanFP16, FP32Vision models, image segmentation
NNAPI delegateAndroid NPUs, DSPs and GPUs through vendor driversINT8, FP16Broad Android coverage, now deprecated
Core ML delegateApple Neural Engine and GPUFP16Models shipped in iOS apps
QNN or Hexagon delegateQualcomm Hexagon NPU and DSPINT8, INT16Low-power always-on sensing
XNNPACKArm and x86 CPU cores with SIMDFP32, FP16, INT8Default fast CPU path
Microcontroller kernelsArm Cortex-M with CMSIS-NN or Ethos-UINT8TinyML and keyword spotting
Representative delegates and execution providers

Because Android deprecated the Neural Networks API in recent releases, developers increasingly use vendor-specific delegates directly. This improves operator coverage on one chip family but reduces portability, which is a recurring tension in edge deployment.

A hardware delegate is a runtime-level acceleration hook, which distinguishes it from the broader runtime and from the offline steps that prepare a model. Several neighboring concepts in edge inference are often confused with it.

ConceptWhat it isRelation to the delegate
Edge inference runtimeThe engine that loads and executes the modelHosts one or more delegates
Execution providerONNX Runtime name for an accelerator backendFunctional equivalent
Model conversionOffline translation to a runtime formatHappens before delegation
Operator fusionMerging several operators into one kernelOften performed inside the delegate
Post-training quantizationReducing weights and activations to INT8 or lowerDetermines which delegates can run the model
Model offloadingMoving work to another device or the cloudDelegation stays on the same device
How the delegate relates to neighboring concepts

The key difference from model offloading is locality: a delegate moves computation between processors of one system on a chip, so data never leaves the device and network latency does not apply.

Choosing and configuring a hardware delegate

The right hardware delegate depends on the model’s operator set, its precision and the target device fleet. Engineers usually benchmark several options on real devices, because published peak TOPS figures rarely predict end-to-end latency.

SituationRecommended choiceReason
FP16 vision model on many Android phonesGPU delegateWide availability and good float coverage
Fully INT8 model on a known Qualcomm deviceQNN delegateHighest efficiency on the NPU
iOS app with a convolutional modelCore ML delegateAccess to the Neural Engine
Transformer with dynamic shapesXNNPACK on CPUAccelerators often reject dynamic shapes
Microcontroller under 1 MB RAMCMSIS-NN kernelsNo separate accelerator memory
Decision guide for selecting a delegate

Common configuration options include allowing FP16 precision loss on GPUs, setting a maximum number of delegated partitions, enabling a kernel cache directory and choosing a power or performance preference. A robust application also implements a fallback chain: try the NPU delegate, then the GPU delegate, then the CPU, and record which path succeeded.

Limitations and common mistakes

The most frequent problem is graph fragmentation. If a single unsupported operator sits in the middle of a network, the delegate splits the model into two partitions with a CPU island between them, and the transfer overhead can exceed the speedup. In such cases a delegated model may run slower than plain CPU inference.

  • Numerical drift: FP16 or vendor INT8 kernels may produce slightly different outputs than the reference CPU path, so accuracy must be validated per delegate.
  • Driver variability: the same delegate can behave differently across chipsets and operating system versions, which makes device-farm testing essential.
  • Initialization latency: compiling kernels at startup can freeze the first frame if caching is disabled.
  • Thermal throttling: sustained accelerator load raises temperature, so short benchmarks overstate long-run throughput.
  • Silent fallback: a runtime may quietly execute everything on the CPU when delegation fails, so logging the number of delegated nodes is a basic safeguard.

Mitigations include replacing unsupported operators during model conversion, applying quantization that matches the accelerator’s native precision and using hardware-aware architecture choices such as MobileNet-style blocks that delegates support well.

In plain words

A phone contains several kinds of processors. The main processor can do almost anything, but special chips for graphics or AI can do certain math much faster while using less battery. A hardware delegate is the go-between that hands suitable pieces of an AI model to those special chips.

Think of a general contractor who passes the wiring to an electrician and the plumbing to a plumber. The job gets done faster, but every handover takes time, so the work should be split into as few handovers as possible. When a specialist cannot do a task, the contractor simply does it alone.

A real-world example

A team building a plant-disease detection app for farmers ships a quantized image classifier of about 4 MB. On a mid-range Android phone, the CPU path with four threads needs roughly 45 ms per photo, which is acceptable for single shots but too slow for a live camera preview.

The engineers enable the GPU delegate with FP16 precision and discover that one custom resize operator keeps falling back to the CPU, creating three separate partitions. After replacing that operator with a standard one during conversion, the whole graph is delegated as a single partition. Latency drops to around 12 ms, the preview runs smoothly at 30 frames per second, and kernel caching reduces app startup delay from 1.8 seconds to under 0.3 seconds. The team keeps the CPU path as a fallback for older phones whose GPU drivers fail validation.