ONNX

Also known as: Open Neural Network Exchange

On this page 7

ONNX, short for Open Neural Network Exchange, is an open, vendor-neutral format for trained machine learning models that describes a model as a computation graph of standardized operators together with its learned weights. Among AI frameworks & tools in artificial intelligence, it acts as the interchange layer between training frameworks and inference engines: a model trained in PyTorch, TensorFlow or scikit-learn can be exported once and executed by many runtimes and hardware back ends.

The format was introduced in 2017 by Microsoft and Facebook and is now governed as an open project under the LF AI & Data Foundation. ONNX itself is a specification plus reference tooling; execution is the job of a separate engine, most often ONNX Runtime.

How an ONNX model is structured

An ONNX file is a Protocol Buffers (protobuf) message, usually saved with the .onnx extension, that contains a top-level ModelProto. Inside it, a GraphProto lists nodes, each of which calls one operator such as Conv, MatMul, Relu or Softmax, and connects named tensors as inputs and outputs. The graph is static and acyclic, which lets tools analyze it without running the original Python code.

The learned parameters are stored as initializers, constant tensors attached to the graph. Because protobuf messages are limited to 2 GB, larger models keep their weights in external data files referenced from the main file. Inputs and outputs carry element types and shapes, and any dimension can be symbolic, for example batch or sequence, so one exported model accepts variable batch sizes and sequence lengths.

The table below summarizes the main building blocks of a model file and the role each one plays.

ComponentWhat it holdsWhy it matters
IR versionVersion of the file format itselfDetermines which readers can parse the file
Opset importsOperator set domain and version, such as ai.onnx 17Fixes the exact semantics of every operator
Graph nodesOperator calls wired by tensor namesDefine the computation to execute
InitializersWeights and constant tensorsCarry the trained parameters
Inputs and outputsNames, data types and shapesForm the contract with calling code
MetadataProducer name, version, custom key-value pairsSupport traceability and model registries
Main components of an ONNX model file

Operator sets and versioning in ONNX

Operator sets (opsets) are the core of ONNX compatibility: each opset version defines a fixed list of operators with precise inputs, attributes and behavior. When an operator changes, for example when a new attribute is added, the change appears in a new opset version while older versions remain valid. A model declares the opset it targets, and a runtime must support that version to execute it correctly.

Besides the default ai.onnx domain for neural network operators, the ai.onnx.ml domain covers classical machine learning, such as tree ensembles, linear classifiers and label encoders. Vendors can also register custom operator domains, which keeps the format extensible but reduces portability, because only runtimes that implement the custom operator can load the model.

The following table compares the main ways a model can reach the ONNX format and the typical issues each path raises.

Source frameworkTypical exporterFrequent difficulty
PyTorchtorch.onnx.export, TorchScript or Dynamo basedData-dependent control flow and dynamic shapes
TensorFlow and Kerastf2onnxUnsupported TensorFlow operations
scikit-learnskl2onnxCustom transformers without converters
XGBoost and LightGBMonnxmltoolsFeature naming and type mismatches
Hugging Face TransformersOptimum exportersCache inputs for autoregressive decoding
Common export paths into ONNX

ONNX is best understood as a framework-neutral graph format, unlike formats that store only weights or that belong to one runtime. A safetensors file, for instance, stores tensors without any computation graph, so the loading code must rebuild the model architecture. Formats such as Core ML, LiteRT or a TensorRT engine are tied to a specific runtime or hardware family, so teams often export to ONNX first and compile from that one portable artifact.

FormatContains a graphPortabilityTypical role
ONNXYesMany runtimes and vendorsExchange and cross-platform inference
safetensorsNo, weights onlyAny framework with matching codeSafe, fast weight distribution
TorchScriptYesMainly PyTorch runtimesPyTorch deployment without Python
TensorFlow SavedModelYesMainly TensorFlow ecosystemTensorFlow Serving and conversion
TensorRT engineYes, compiledOne GPU model and TensorRT versionMaximum NVIDIA inference speed
Core ML modelYesApple devicesOn-device inference on Apple hardware
ONNX and neighboring model formats

Running and optimizing ONNX models

An ONNX model runs inside an inference engine that parses the graph, applies optimizations and maps operators to hardware kernels. ONNX Runtime handles this through execution providers for CPU, CUDA, TensorRT, DirectML, OpenVINO, Core ML and others, while engines such as TensorRT or OpenVINO can import ONNX files directly. The same file can therefore serve a cloud GPU server, a desktop application and an edge device.

Typical optimizations include operator fusion (for example merging convolution, batch normalization and activation), constant folding, removal of unused nodes and quantization to INT8 or lower precision. Quantized ONNX models use either the QDQ representation, with explicit QuantizeLinear and DequantizeLinear nodes, or integer operators such as QLinearConv; QDQ is generally the more portable choice.

The decision table below shows how deployment needs usually translate into ONNX choices.

SituationRecommended approachReason
Same model on Windows, Linux and macOSONNX Runtime with CPU providerOne binary format and stable API
High-throughput NVIDIA servingONNX as input to TensorRTKernel tuning and reduced precision
Intel CPUs or integrated GPUsOpenVINO importing ONNXHardware-specific graph optimization
Browser inferenceONNX Runtime Web with WebGPU or WebAssemblyNo server round trip
Memory-constrained edge deviceINT8 QDQ quantized modelRoughly four times smaller than FP32 weights
Choosing an ONNX deployment approach

Limitations and common mistakes with ONNX

The main limitation of ONNX is that export captures a traced or compiled graph, not the Python program. Data-dependent branches, loops written in Python and unusual tensor manipulations may be frozen to the values seen during export or may fail to convert. Small numerical differences between frameworks are normal, so outputs should be compared with a tolerance rather than for exact equality.

SymptomLikely causeFix
Model fails to loadOpset newer than the runtime supportsExport with a lower opset or upgrade the runtime
Only batch size 1 worksShapes fixed during exportDeclare dynamic axes for variable dimensions
Outputs differ from the originalPreprocessing or precision mismatchCompare with tolerances and reproduce preprocessing
File cannot be savedModel exceeds the 2 GB protobuf limitStore weights as external data
Unsupported operator errorCustom or missing operatorRewrite the layer or register a custom operator
Frequent ONNX problems and their fixes
  • ONNX is not a training framework; training support exists in some runtimes but remains a niche use.
  • Preprocessing is usually outside the graph, so tokenization or image normalization must be reproduced exactly in the serving code.
  • Validation should always follow export: run the ONNX checker, shape inference and a numerical comparison against the source model.
  • An ONNX file can still contain untrusted logic through custom operators, so models from unknown sources deserve review.

Stripped to the essentials

ONNX is a common file format for trained AI models. Think of it like PDF for documents: whoever made the file, many different programs can open it and show the same result. With ONNX, a model built in one tool can run in many other programs and on many devices.

The file describes the steps the model performs and stores the numbers it learned. A separate program reads that file and runs it fast on a phone, a laptop or a server. The key point is simple: build once, run almost anywhere, as long as the target program understands the version of the format used.

Where it is applied

A manufacturing company trains an image classifier for defect detection in PyTorch on a GPU workstation. The production line, however, runs Windows industrial PCs with Intel processors, and a second site uses NVIDIA Jetson modules. Porting the model twice would be slow and error-prone.

The team exports the model to ONNX with opset 17 and a dynamic batch dimension, checks it with the ONNX checker and confirms that outputs match PyTorch within a tolerance of 1e-4 on 500 test images. They then quantize it to INT8 in QDQ format for the Intel PCs and build a TensorRT engine from the same file for the Jetson modules.

The result is one validated model artifact serving both sites. Inference on the industrial PCs drops from about 90 ms to about 35 ms per image, accuracy changes by less than half a percentage point, and future retraining only requires a new export rather than two separate ports.