TensorRT (also called NVIDIA TensorRT) is a software development kit for high-performance deep learning inference that takes a trained neural network and compiles it into an optimized engine for a specific NVIDIA GPU. Among AI frameworks & tools, it sits at the deployment end of the pipeline: models are trained in PyTorch, TensorFlow or JAX, then handed to TensorRT to run with lower latency and higher throughput in production.
TensorRT does not train models. Its job is to remove everything that training needs but inference does not, to fuse operations, to choose the fastest kernel for every layer and, when allowed, to run the network at reduced numerical precision such as FP16, INT8 or FP8.
How TensorRT builds an inference engine
TensorRT works in two distinct phases: build and run. In the build phase, the builder parses a network definition, most often an ONNX file exported from a training framework, and produces a serialized engine (also called a plan file). In the runtime phase, the application deserializes that engine and executes it on the GPU through an execution context, usually inside a CUDA stream.
The build phase is where the optimization happens. The builder applies a sequence of graph transformations and then measures candidate implementations on the actual hardware, a step known as kernel auto-tuning or tactic selection. Because the result is tuned to one device, an engine is tied to the GPU architecture and TensorRT version that built it.
The following table summarizes the main optimizations the builder performs and what each one improves.
| Optimization | What it does | Main benefit |
|---|---|---|
| Layer and tensor fusion | Merges convolution, bias and activation into one kernel | Fewer memory round trips |
| Precision calibration | Runs layers in FP16, BF16, INT8 or FP8 | Higher throughput, smaller memory use |
| Kernel auto-tuning | Times several tactics per layer and keeps the fastest | Best speed on the target GPU |
| Dead layer elimination | Removes unused outputs and constant subgraphs | Smaller, simpler graph |
| Memory reuse | Assigns tensor buffers by lifetime | Lower activation memory |
| Dynamic shape profiles | Optimizes for declared minimum, optimal and maximum shapes | Variable batch and sequence sizes |
Precision modes and quantization in TensorRT
Reduced precision is the largest single source of speedup in TensorRT, because Tensor Cores process lower-precision data at much higher rates. A model trained in FP32 can usually run in FP16 with negligible accuracy change, while integer and 8-bit floating-point modes require calibration or quantization-aware training to keep accuracy within acceptable limits.
INT8 engines need scale factors that map floating-point ranges to integers. TensorRT obtains them either through post-training calibration with a few hundred representative inputs or from explicit quantize and dequantize nodes inserted during quantization-aware training. The table below compares the common modes; the speedups are typical ranges that depend heavily on the model and GPU generation.
| Precision | Typical speedup vs FP32 | Accuracy risk | Requirement |
|---|---|---|---|
| FP32 | Baseline | None | Any supported GPU |
| TF32 | Moderate on Ampere and later | Very low | Ampere or newer |
| FP16 or BF16 | About 2x to 4x | Low | Tensor Core GPU |
| INT8 | About 3x to 6x | Moderate without calibration | Calibration data or QAT |
| FP8 | Higher than FP16 on recent GPUs | Moderate | Hopper or newer |
| INT4 weights | Large gains for memory-bound LLMs | Model dependent | Weight-only quantization |
Precision can be mixed per layer. Sensitive layers such as the final classifier or normalization layers are often pinned to higher precision while the bulk of the network runs in INT8 or FP16.
TensorRT compared with related inference tools
TensorRT is a vendor-specific compiler and runtime, which distinguishes it from portable inference engines in the same toolchain. In practice it is frequently combined with them rather than chosen instead of them, for example as the TensorRT execution provider inside ONNX Runtime.
| Tool | Role | Hardware scope | Relation to TensorRT |
|---|---|---|---|
| ONNX | Model exchange format | Hardware neutral | Main input format for the parser |
| ONNX Runtime | Portable inference engine | CPU, GPU, NPU | Can delegate subgraphs to TensorRT |
| PyTorch | Training and eager inference | Broad | Exports models; Torch-TensorRT compiles them |
| TensorFlow | Training and serving | Broad | TF-TRT integration converts subgraphs |
| llama.cpp | Local LLM runtime | CPU and many GPUs | Portable alternative, less GPU-specific tuning |
| TensorRT-LLM | LLM-specific library on TensorRT | NVIDIA GPUs | Adds paged KV cache and in-flight batching |
For large language models, plain TensorRT is rarely used directly. TensorRT-LLM adds attention kernels, paged key-value caching, tensor parallelism and request batching, and it is commonly served through the Triton Inference Server.
Deploying models with TensorRT in practice
A typical workflow is export, build, validate and serve. Engineers export the model to ONNX, build an engine with the trtexec command-line tool or the Python API, compare outputs against the original framework, and then load the engine in a C++ or Python service. Validation against reference outputs is essential, because fusion and reduced precision change numerical results slightly.
The right integration path depends on how much control and portability the team needs, as the decision table shows.
| Situation | Recommended path | Reason |
|---|---|---|
| Maximum performance, fixed model | Native engine via ONNX parser | Full control over precision and profiles |
| PyTorch code with unsupported operators | Torch-TensorRT | Falls back to PyTorch for unsupported parts |
| Mixed hardware fleet | ONNX Runtime with TensorRT provider | One model file, GPU acceleration where available |
| Large language model serving | TensorRT-LLM with Triton | Batching and KV cache management |
| Edge device such as Jetson | Engine built on the device | Engines are not portable across GPUs |
Limitations and common mistakes with TensorRT
The main trade-off of TensorRT is speed in exchange for portability and flexibility. Engines run only on NVIDIA hardware, and an engine built for one GPU generation or TensorRT release generally must be rebuilt for another, although newer versions offer limited hardware-compatibility and version-compatibility modes at some performance cost.
- Unsupported operators in the ONNX graph stop the parser; custom layers then require a plugin written in CUDA.
- Uncalibrated INT8 with unrepresentative data can cause severe accuracy loss that only appears on real traffic.
- Overly wide dynamic shape ranges reduce the optimizations the builder can apply.
- Long build times, from seconds to tens of minutes, make engines unsuitable for building at request time.
- Benchmarking with warm-up excluded or with synthetic inputs overstates real-world latency gains.
The simple explanation
Training a neural network is like writing a long recipe that works in any kitchen. TensorRT takes that finished recipe and rewrites it for one particular kitchen, combining steps, using the quickest tools on hand and measuring ingredients a little less precisely where it does not change the taste.
The result is a version of the model that answers much faster on an NVIDIA graphics card. The cost is that this rewritten version only fits that type of kitchen, so moving to different hardware means preparing it again.
How it is used in practice
A logistics company runs an object detection model that reads parcel labels from 40 camera feeds in a sorting center. In PyTorch on a single data center GPU, the model handles each frame in about 18 milliseconds, which is too slow to keep up with the conveyor at peak hours.
The engineering team exports the model to ONNX, builds a TensorRT engine with FP16 precision and a dynamic batch range of 1 to 16, and calibrates an INT8 variant with 500 real label images. After checking that detection accuracy drops by less than half a percentage point, they deploy the INT8 engine behind Triton.
Latency per frame falls to roughly 5 milliseconds, and one GPU now serves all 40 cameras instead of the three that had been budgeted, without retraining the model.