ONNX, short for Open Neural Network Exchange, is an open, vendor-neutral format for trained machine learning models that describes a model as a computation graph of standardized operators together with its learned weights. Among AI frameworks & tools in artificial intelligence, it acts as the interchange layer between training frameworks and inference engines: a model trained in PyTorch, TensorFlow or scikit-learn can be exported once and executed by many runtimes and hardware back ends.
The format was introduced in 2017 by Microsoft and Facebook and is now governed as an open project under the LF AI & Data Foundation. ONNX itself is a specification plus reference tooling; execution is the job of a separate engine, most often ONNX Runtime.
How an ONNX model is structured
An ONNX file is a Protocol Buffers (protobuf) message, usually saved with the .onnx extension, that contains a top-level ModelProto. Inside it, a GraphProto lists nodes, each of which calls one operator such as Conv, MatMul, Relu or Softmax, and connects named tensors as inputs and outputs. The graph is static and acyclic, which lets tools analyze it without running the original Python code.
The learned parameters are stored as initializers, constant tensors attached to the graph. Because protobuf messages are limited to 2 GB, larger models keep their weights in external data files referenced from the main file. Inputs and outputs carry element types and shapes, and any dimension can be symbolic, for example batch or sequence, so one exported model accepts variable batch sizes and sequence lengths.
The table below summarizes the main building blocks of a model file and the role each one plays.
| Component | What it holds | Why it matters |
|---|---|---|
| IR version | Version of the file format itself | Determines which readers can parse the file |
| Opset imports | Operator set domain and version, such as ai.onnx 17 | Fixes the exact semantics of every operator |
| Graph nodes | Operator calls wired by tensor names | Define the computation to execute |
| Initializers | Weights and constant tensors | Carry the trained parameters |
| Inputs and outputs | Names, data types and shapes | Form the contract with calling code |
| Metadata | Producer name, version, custom key-value pairs | Support traceability and model registries |
Operator sets and versioning in ONNX
Operator sets (opsets) are the core of ONNX compatibility: each opset version defines a fixed list of operators with precise inputs, attributes and behavior. When an operator changes, for example when a new attribute is added, the change appears in a new opset version while older versions remain valid. A model declares the opset it targets, and a runtime must support that version to execute it correctly.
Besides the default ai.onnx domain for neural network operators, the ai.onnx.ml domain covers classical machine learning, such as tree ensembles, linear classifiers and label encoders. Vendors can also register custom operator domains, which keeps the format extensible but reduces portability, because only runtimes that implement the custom operator can load the model.
The following table compares the main ways a model can reach the ONNX format and the typical issues each path raises.
| Source framework | Typical exporter | Frequent difficulty |
|---|---|---|
| PyTorch | torch.onnx.export, TorchScript or Dynamo based | Data-dependent control flow and dynamic shapes |
| TensorFlow and Keras | tf2onnx | Unsupported TensorFlow operations |
| scikit-learn | skl2onnx | Custom transformers without converters |
| XGBoost and LightGBM | onnxmltools | Feature naming and type mismatches |
| Hugging Face Transformers | Optimum exporters | Cache inputs for autoregressive decoding |
How ONNX compares with related formats
ONNX is best understood as a framework-neutral graph format, unlike formats that store only weights or that belong to one runtime. A safetensors file, for instance, stores tensors without any computation graph, so the loading code must rebuild the model architecture. Formats such as Core ML, LiteRT or a TensorRT engine are tied to a specific runtime or hardware family, so teams often export to ONNX first and compile from that one portable artifact.
| Format | Contains a graph | Portability | Typical role |
|---|---|---|---|
| ONNX | Yes | Many runtimes and vendors | Exchange and cross-platform inference |
| safetensors | No, weights only | Any framework with matching code | Safe, fast weight distribution |
| TorchScript | Yes | Mainly PyTorch runtimes | PyTorch deployment without Python |
| TensorFlow SavedModel | Yes | Mainly TensorFlow ecosystem | TensorFlow Serving and conversion |
| TensorRT engine | Yes, compiled | One GPU model and TensorRT version | Maximum NVIDIA inference speed |
| Core ML model | Yes | Apple devices | On-device inference on Apple hardware |
Running and optimizing ONNX models
An ONNX model runs inside an inference engine that parses the graph, applies optimizations and maps operators to hardware kernels. ONNX Runtime handles this through execution providers for CPU, CUDA, TensorRT, DirectML, OpenVINO, Core ML and others, while engines such as TensorRT or OpenVINO can import ONNX files directly. The same file can therefore serve a cloud GPU server, a desktop application and an edge device.
Typical optimizations include operator fusion (for example merging convolution, batch normalization and activation), constant folding, removal of unused nodes and quantization to INT8 or lower precision. Quantized ONNX models use either the QDQ representation, with explicit QuantizeLinear and DequantizeLinear nodes, or integer operators such as QLinearConv; QDQ is generally the more portable choice.
The decision table below shows how deployment needs usually translate into ONNX choices.
| Situation | Recommended approach | Reason |
|---|---|---|
| Same model on Windows, Linux and macOS | ONNX Runtime with CPU provider | One binary format and stable API |
| High-throughput NVIDIA serving | ONNX as input to TensorRT | Kernel tuning and reduced precision |
| Intel CPUs or integrated GPUs | OpenVINO importing ONNX | Hardware-specific graph optimization |
| Browser inference | ONNX Runtime Web with WebGPU or WebAssembly | No server round trip |
| Memory-constrained edge device | INT8 QDQ quantized model | Roughly four times smaller than FP32 weights |
Limitations and common mistakes with ONNX
The main limitation of ONNX is that export captures a traced or compiled graph, not the Python program. Data-dependent branches, loops written in Python and unusual tensor manipulations may be frozen to the values seen during export or may fail to convert. Small numerical differences between frameworks are normal, so outputs should be compared with a tolerance rather than for exact equality.
| Symptom | Likely cause | Fix |
|---|---|---|
| Model fails to load | Opset newer than the runtime supports | Export with a lower opset or upgrade the runtime |
| Only batch size 1 works | Shapes fixed during export | Declare dynamic axes for variable dimensions |
| Outputs differ from the original | Preprocessing or precision mismatch | Compare with tolerances and reproduce preprocessing |
| File cannot be saved | Model exceeds the 2 GB protobuf limit | Store weights as external data |
| Unsupported operator error | Custom or missing operator | Rewrite the layer or register a custom operator |
- ONNX is not a training framework; training support exists in some runtimes but remains a niche use.
- Preprocessing is usually outside the graph, so tokenization or image normalization must be reproduced exactly in the serving code.
- Validation should always follow export: run the ONNX checker, shape inference and a numerical comparison against the source model.
- An ONNX file can still contain untrusted logic through custom operators, so models from unknown sources deserve review.
Stripped to the essentials
ONNX is a common file format for trained AI models. Think of it like PDF for documents: whoever made the file, many different programs can open it and show the same result. With ONNX, a model built in one tool can run in many other programs and on many devices.
The file describes the steps the model performs and stores the numbers it learned. A separate program reads that file and runs it fast on a phone, a laptop or a server. The key point is simple: build once, run almost anywhere, as long as the target program understands the version of the format used.
Where it is applied
A manufacturing company trains an image classifier for defect detection in PyTorch on a GPU workstation. The production line, however, runs Windows industrial PCs with Intel processors, and a second site uses NVIDIA Jetson modules. Porting the model twice would be slow and error-prone.
The team exports the model to ONNX with opset 17 and a dynamic batch dimension, checks it with the ONNX checker and confirms that outputs match PyTorch within a tolerance of 1e-4 on 500 test images. They then quantize it to INT8 in QDQ format for the Intel PCs and build a TensorRT engine from the same file for the Jetson modules.
The result is one validated model artifact serving both sites. Inference on the industrial PCs drops from about 90 ms to about 35 ms per image, accuracy changes by less than half a percentage point, and future retraining only requires a new export rather than two separate ports.