TensorRT

Also known as: NVIDIA TensorRT

On this page 7

TensorRT (also called NVIDIA TensorRT) is a software development kit for high-performance deep learning inference that takes a trained neural network and compiles it into an optimized engine for a specific NVIDIA GPU. Among AI frameworks & tools, it sits at the deployment end of the pipeline: models are trained in PyTorch, TensorFlow or JAX, then handed to TensorRT to run with lower latency and higher throughput in production.

TensorRT does not train models. Its job is to remove everything that training needs but inference does not, to fuse operations, to choose the fastest kernel for every layer and, when allowed, to run the network at reduced numerical precision such as FP16, INT8 or FP8.

How TensorRT builds an inference engine

TensorRT works in two distinct phases: build and run. In the build phase, the builder parses a network definition, most often an ONNX file exported from a training framework, and produces a serialized engine (also called a plan file). In the runtime phase, the application deserializes that engine and executes it on the GPU through an execution context, usually inside a CUDA stream.

The build phase is where the optimization happens. The builder applies a sequence of graph transformations and then measures candidate implementations on the actual hardware, a step known as kernel auto-tuning or tactic selection. Because the result is tuned to one device, an engine is tied to the GPU architecture and TensorRT version that built it.

The following table summarizes the main optimizations the builder performs and what each one improves.

OptimizationWhat it doesMain benefit
Layer and tensor fusionMerges convolution, bias and activation into one kernelFewer memory round trips
Precision calibrationRuns layers in FP16, BF16, INT8 or FP8Higher throughput, smaller memory use
Kernel auto-tuningTimes several tactics per layer and keeps the fastestBest speed on the target GPU
Dead layer eliminationRemoves unused outputs and constant subgraphsSmaller, simpler graph
Memory reuseAssigns tensor buffers by lifetimeLower activation memory
Dynamic shape profilesOptimizes for declared minimum, optimal and maximum shapesVariable batch and sequence sizes
Core optimizations applied during the build phase

Precision modes and quantization in TensorRT

Reduced precision is the largest single source of speedup in TensorRT, because Tensor Cores process lower-precision data at much higher rates. A model trained in FP32 can usually run in FP16 with negligible accuracy change, while integer and 8-bit floating-point modes require calibration or quantization-aware training to keep accuracy within acceptable limits.

INT8 engines need scale factors that map floating-point ranges to integers. TensorRT obtains them either through post-training calibration with a few hundred representative inputs or from explicit quantize and dequantize nodes inserted during quantization-aware training. The table below compares the common modes; the speedups are typical ranges that depend heavily on the model and GPU generation.

PrecisionTypical speedup vs FP32Accuracy riskRequirement
FP32BaselineNoneAny supported GPU
TF32Moderate on Ampere and laterVery lowAmpere or newer
FP16 or BF16About 2x to 4xLowTensor Core GPU
INT8About 3x to 6xModerate without calibrationCalibration data or QAT
FP8Higher than FP16 on recent GPUsModerateHopper or newer
INT4 weightsLarge gains for memory-bound LLMsModel dependentWeight-only quantization
Precision modes supported by TensorRT

Precision can be mixed per layer. Sensitive layers such as the final classifier or normalization layers are often pinned to higher precision while the bulk of the network runs in INT8 or FP16.

TensorRT is a vendor-specific compiler and runtime, which distinguishes it from portable inference engines in the same toolchain. In practice it is frequently combined with them rather than chosen instead of them, for example as the TensorRT execution provider inside ONNX Runtime.

ToolRoleHardware scopeRelation to TensorRT
ONNXModel exchange formatHardware neutralMain input format for the parser
ONNX RuntimePortable inference engineCPU, GPU, NPUCan delegate subgraphs to TensorRT
PyTorchTraining and eager inferenceBroadExports models; Torch-TensorRT compiles them
TensorFlowTraining and servingBroadTF-TRT integration converts subgraphs
llama.cppLocal LLM runtimeCPU and many GPUsPortable alternative, less GPU-specific tuning
TensorRT-LLMLLM-specific library on TensorRTNVIDIA GPUsAdds paged KV cache and in-flight batching
How TensorRT relates to neighboring tools

For large language models, plain TensorRT is rarely used directly. TensorRT-LLM adds attention kernels, paged key-value caching, tensor parallelism and request batching, and it is commonly served through the Triton Inference Server.

Deploying models with TensorRT in practice

A typical workflow is export, build, validate and serve. Engineers export the model to ONNX, build an engine with the trtexec command-line tool or the Python API, compare outputs against the original framework, and then load the engine in a C++ or Python service. Validation against reference outputs is essential, because fusion and reduced precision change numerical results slightly.

The right integration path depends on how much control and portability the team needs, as the decision table shows.

SituationRecommended pathReason
Maximum performance, fixed modelNative engine via ONNX parserFull control over precision and profiles
PyTorch code with unsupported operatorsTorch-TensorRTFalls back to PyTorch for unsupported parts
Mixed hardware fleetONNX Runtime with TensorRT providerOne model file, GPU acceleration where available
Large language model servingTensorRT-LLM with TritonBatching and KV cache management
Edge device such as JetsonEngine built on the deviceEngines are not portable across GPUs
Choosing a TensorRT integration path

Limitations and common mistakes with TensorRT

The main trade-off of TensorRT is speed in exchange for portability and flexibility. Engines run only on NVIDIA hardware, and an engine built for one GPU generation or TensorRT release generally must be rebuilt for another, although newer versions offer limited hardware-compatibility and version-compatibility modes at some performance cost.

  • Unsupported operators in the ONNX graph stop the parser; custom layers then require a plugin written in CUDA.
  • Uncalibrated INT8 with unrepresentative data can cause severe accuracy loss that only appears on real traffic.
  • Overly wide dynamic shape ranges reduce the optimizations the builder can apply.
  • Long build times, from seconds to tens of minutes, make engines unsuitable for building at request time.
  • Benchmarking with warm-up excluded or with synthetic inputs overstates real-world latency gains.

The simple explanation

Training a neural network is like writing a long recipe that works in any kitchen. TensorRT takes that finished recipe and rewrites it for one particular kitchen, combining steps, using the quickest tools on hand and measuring ingredients a little less precisely where it does not change the taste.

The result is a version of the model that answers much faster on an NVIDIA graphics card. The cost is that this rewritten version only fits that type of kitchen, so moving to different hardware means preparing it again.

How it is used in practice

A logistics company runs an object detection model that reads parcel labels from 40 camera feeds in a sorting center. In PyTorch on a single data center GPU, the model handles each frame in about 18 milliseconds, which is too slow to keep up with the conveyor at peak hours.

The engineering team exports the model to ONNX, builds a TensorRT engine with FP16 precision and a dynamic batch range of 1 to 16, and calibrates an INT8 variant with 500 real label images. After checking that detection accuracy drops by less than half a percentage point, they deploy the INT8 engine behind Triton.

Latency per frame falls to roughly 5 milliseconds, and one GPU now serves all 40 cameras instead of the three that had been budgeted, without retraining the model.