ONNX Runtime

On this page 8

ONNX Runtime is an open-source, cross-platform inference engine that loads models in the ONNX format and executes them efficiently on CPUs, GPUs and dedicated accelerators. Among AI frameworks & tools, it occupies the deployment end of the pipeline: models are trained in PyTorch, TensorFlow, scikit-learn or other libraries, exported to ONNX, and then served by ONNX Runtime without the original training framework.

The engine was created by Microsoft and is now maintained as an open project used in cloud services, desktop applications, mobile apps and browsers. Its value rests on two ideas: graph-level optimization of the model before execution, and execution providers that hand parts of the graph to the best available hardware library.

How ONNX Runtime executes a model

ONNX Runtime executes a model by loading the ONNX graph, transforming it into an optimized form, partitioning it across hardware backends and running the resulting kernels in dependency order. The ONNX file itself is a protobuf description of operators, tensors and weights, versioned by an opset number that fixes the semantics of each operator.

During session creation, the runtime applies graph transformations such as constant folding, redundant node elimination and operator fusion, for example merging a convolution, batch normalization and activation into one kernel. It then asks each registered execution provider which nodes it can handle, assigns subgraphs to providers in priority order and falls back to the default CPU provider for anything left over.

The main stages of a typical inference session are summarized below, together with what each stage controls.

StageWhat happensMain setting
Model loadingParses the ONNX protobuf and validates the opsetModel path or byte buffer
Graph optimizationFolds constants, fuses operators, removes dead nodesOptimization level
PartitioningAssigns subgraphs to execution providersProvider list and order
Memory planningReuses buffers and pre-allocates arenasArena and memory pattern options
ExecutionRuns kernels sequentially or in parallelThread counts and execution mode
Stages of an ONNX Runtime inference session

Because session creation performs this work once, the first run is slower than later ones. Production services therefore create a session at startup, keep it alive and send repeated requests through it; ONNX Runtime sessions are thread-safe for concurrent inference calls.

Execution providers and supported hardware

An execution provider is a plug-in that maps ONNX operators to a specific hardware library, and the choice of provider usually determines performance more than any other setting. A model can use several providers at once, which is useful when an accelerator supports most operators but not all of them.

The following table lists widely used providers and the hardware each one targets; availability depends on the package build installed.

Execution providerTarget hardwareTypical use
CPUx86 and Arm processorsDefault fallback, servers without accelerators
CUDANVIDIA GPUsCloud inference with moderate latency demands
TensorRTNVIDIA GPUs with TensorRTLowest latency on NVIDIA hardware
DirectMLDirectX 12 GPUs on WindowsDesktop applications across GPU vendors
OpenVINOIntel CPUs, GPUs and NPUsEdge and client devices with Intel silicon
CoreMLApple Neural Engine and GPUiOS and macOS applications
QNNQualcomm NPUsAndroid devices and Arm laptops
Common execution providers in ONNX Runtime

Every boundary between providers forces a tensor copy between memory spaces, so a graph that switches between GPU and CPU many times can run slower than a pure CPU graph. Profiling the partition, which ONNX Runtime can log at session creation, is the first step when an accelerated model underperforms.

Optimization levels and quantization

ONNX Runtime improves speed through graph optimization levels and quantization, and both can be applied offline so that the optimized model is saved and reused. Offline optimization shortens session startup, which matters for serverless functions and mobile apps.

The optimization levels build on each other, as the next table shows.

LevelTransformations includedHardware dependence
DisabledNone, graph runs as exportedNone
BasicConstant folding and redundant node removalHardware independent
ExtendedComplex fusions such as attention and GELUPartly provider specific
LayoutChanges tensor memory layout, for example NCHWcCPU specific
Graph optimization levels

Quantization converts weights and often activations from 32-bit floating point to 8-bit integers, cutting model size by roughly four times and accelerating inference on hardware with integer instructions. Dynamic quantization computes activation scales at run time and suits transformer models, while static quantization relies on a calibration dataset and suits convolutional networks. Accuracy loss is usually small but must be measured on a validation set, because some layers are sensitive to reduced precision.

ONNX Runtime is best understood as a framework-neutral runtime, distinct both from the ONNX format it reads and from vendor-specific inference engines. The comparison below clarifies where it fits among neighboring tools.

ToolRoleKey difference from ONNX Runtime
ONNXOpen model exchange formatDefines the file, does not execute it
TensorRTNVIDIA inference optimizerSingle vendor, can serve as an ONNX Runtime provider
PyTorch eager modeTraining and research frameworkHeavier dependency, less deployment focused
LiteRTOn-device runtime for TensorFlow modelsUses its own flat buffer format
llama.cppSpecialized engine for language modelsNarrow model scope, GGUF format
ONNX Runtime and neighboring inference options

The decisive advantage of ONNX Runtime is portability with one API: the same model and nearly identical code run on a Linux GPU server, a Windows laptop and an Android phone. The trade-off is that a dedicated engine tuned for one hardware family or one model class can reach higher peak throughput.

Language bindings and deployment targets

ONNX Runtime offers bindings for Python, C, C++, C#, Java and JavaScript, so inference can live inside existing applications instead of separate Python services. A web build runs models in the browser through WebAssembly or WebGPU, and mobile packages with a reduced operator set keep app size small.

  • Server inference: a Python or C++ service wraps a long-lived session behind an HTTP or gRPC endpoint.
  • Desktop applications: C# and C++ programs call the runtime directly, often through DirectML on Windows.
  • Mobile and edge: minimal builds include only the operators a given model needs.
  • Generative models: the ONNX Runtime GenAI extension adds token generation loops and key-value cache handling for language models.

Limitations and common mistakes

Most problems with ONNX Runtime originate in the export step rather than in the runtime itself. Exporters can fail on dynamic control flow, custom layers or unsupported operators, and an export with fixed input shapes silently rejects other batch sizes.

SymptomLikely causeFix
Shape mismatch errorExported with static dimensionsDeclare dynamic axes during export
Unsupported operatorOpset too old or custom layerRaise the opset or register a custom operator
GPU slower than CPUFrequent provider switchesCheck partitioning and remove fallback nodes
Different outputsPrecision or fusion differencesCompare with tolerances, not exact equality
Typical problems and fixes

A frequent misconception is that conversion alone makes a model faster. Speedups come from the provider, optimization level and quantization, and a careless export can even slow a model down. Teams should validate numerical parity against the original framework and benchmark with realistic batch sizes before switching production traffic.

The simple explanation

Think of an AI model as a recipe written by one chef in one kitchen. ONNX is a standard way to write that recipe down, and ONNX Runtime is the universal kitchen that can cook it on almost any stove, whether that is a big server, a laptop or a phone.

Before cooking, the runtime reorganizes the steps so that less time is wasted, and it hands each step to the best appliance available. The result is the same dish, served faster, without needing the original chef’s kitchen.

How it is used in practice

A logistics company trains a document classification model in PyTorch to sort scanned delivery notes into twelve categories. Running the model inside the original Python environment requires a large container and about 140 milliseconds per page on the CPU servers in its warehouse.

The engineers export the model to ONNX with a dynamic batch dimension, apply dynamic 8-bit quantization and load it into ONNX Runtime from the existing C# warehouse application. After checking that accuracy on a held-out set of 5,000 pages changed by less than half a percentage point, they deploy it. Latency falls to roughly 45 milliseconds per page, the Python service is retired, and the same model file is later reused on handheld Android scanners through the mobile package.