Operator fusion

Also known as: kernel fusion

On this page 7

Operator fusion, also called kernel fusion, is a compiler optimization that merges several consecutive neural network operations into a single kernel, so intermediate results stay in registers or on-chip memory instead of being written to and read back from main memory. In Edge AI, where phones, PCs and embedded boards have limited memory bandwidth and tight power budgets, it is one of the most effective ways to make a model run faster without changing its weights or its accuracy.

Fusion is performed by inference runtimes and graph compilers during model conversion or at load time. It works alongside quantization and pruning: those techniques shrink the model, while fusion reduces the cost of executing the graph that remains.

How operator fusion removes memory round trips

Operator fusion works by eliminating intermediate tensors between operations. In an unfused graph, a convolution writes its full output tensor to memory, a batch normalization reads it and writes another tensor, and an activation such as ReLU reads that tensor again. Each step is a separate kernel launch with its own memory traffic.

A fused kernel computes the convolution, applies the normalization scale and shift, and clamps negative values in one pass, writing only the final result. For many layers on edge hardware, the arithmetic is cheap and the memory transfers dominate latency and energy, so removing them has an outsized effect. Reading a value from DRAM can cost one to two orders of magnitude more energy than a multiply-accumulate operation, which is why fusion matters so much on battery-powered devices.

The following table traces what happens to a common three-operation block with and without fusion.

AspectUnfused executionFused execution
Kernel launchesThreeOne
Intermediate tensors in main memoryTwo full tensorsNone
Memory reads and writesAbout three times the output sizeAbout one times the output size
Numerical resultReference outputSame output, within rounding
Peak activation memoryHigherLower
Execution of a convolution, batch normalization and ReLU block

Common fusion patterns in edge inference

Most edge runtimes recognize a fixed catalog of fusion patterns that cover the bulk of real models. Some patterns are algebraic folds applied once at conversion time, while others are runtime kernel fusions that combine operations into one compute routine.

The table below lists the patterns a practitioner encounters most often and where each one delivers its benefit.

PatternWhat is mergedMain benefit
Batch norm foldingNormalization scale and bias into convolution weightsRemoves an entire layer at no runtime cost
Conv plus activationConvolution with ReLU, ReLU6 or hardswishAvoids writing pre-activation tensors
Elementwise chainsAdd, multiply, clamp and similar operationsOne pass over the data instead of several
Requantization fusionInteger accumulation with rescale to INT8Keeps quantized pipelines in integer arithmetic
Attention fusionMatrix multiply, softmax and second multiplyAvoids materializing the full attention matrix
Depthwise plus pointwiseTwo convolutions of a separable blockReuses tiles while they sit in cache
Typical operator fusion patterns

Batch norm folding is technically a graph rewrite rather than kernel fusion, because the normalization disappears from the graph entirely. Runtimes and documentation often group the two together, so the distinction matters mainly when debugging a converted model.

Vertical and horizontal fusion

Fusion strategies divide into vertical fusion, which merges operations along a data dependency chain, and horizontal fusion, which merges independent operations that read the same input. Vertical fusion is the classic case described above; horizontal fusion appears when, for example, three parallel projection layers share one input and are executed as a single wider matrix multiplication.

Compilers also differ in how they find fusion opportunities. Rule-based fusion matches hand-written patterns and is predictable, while automatic fusion in tensor compilers such as TVM or XLA searches the graph for any legal grouping and generates code for it. Automatic approaches cover unusual architectures better but make performance harder to predict across hardware targets.

ApproachHow fusions are chosenStrengthWeakness
Rule-basedFixed pattern libraryStable, well tested kernelsMisses uncommon layer sequences
AutomaticGraph search and code generationCovers novel architecturesLonger compile time, less predictable
Vendor-specificDelegate or driver decidesTuned for one acceleratorBehavior differs between devices
Comparison of fusion approaches

How operator fusion relates to other edge optimizations

Operator fusion is lossless by design: it changes how the computation is scheduled, not what is computed. This sets it apart from quantization, pruning and low-rank factorization, which all trade some accuracy for size or speed and usually require validation against a test set.

Fusion and quantization reinforce each other. A quantized model that dequantizes between every layer loses much of its speed advantage, so INT8 pipelines depend on fused requantization to stay in integer arithmetic from input to output. When a hardware delegate cannot fuse a pattern, the runtime may fall back to the CPU for that segment, and the data transfer between processors can erase the gain from acceleration.

TechniqueChanges weightsAffects accuracyMain gain
Operator fusionNo, except foldingNoLower latency and memory traffic
Post-training quantizationYesSlightlySmaller model, faster integer math
Structured pruningYesPossiblyFewer channels and operations
Low-rank factorizationYesPossiblyFewer parameters per layer
Operator fusion compared with related techniques

Limitations and common mistakes

Fusion is not always possible or beneficial. Unsupported operator sequences, dynamic shapes and custom layers often block it, and a fused kernel that uses too many registers can lower hardware occupancy and run slower than the separate kernels it replaced.

SymptomLikely causeFix
Latency barely improves after conversionCustom layer breaks the fusion chainReplace it with a supported equivalent
Model partly runs on CPUDelegate lacks a fused kernelCheck the runtime profile and adjust the graph
Small output differencesChanged order of floating-point operationsCompare within a tolerance, not exactly
Fused model slower on one deviceKernel too large for local memoryDisable that fusion or retile
Symptoms, causes and fixes
  • Training-time layers such as dropout must be removed or set to inference mode before conversion, or patterns will not match.
  • Profiling on the target device is the only reliable check, because desktop results rarely transfer to mobile accelerators.
  • Folding batch normalization into weights before quantization calibration usually gives better integer accuracy than folding afterward.

Explained without jargon

Imagine cooking a meal where every step uses a separate pan: you chop vegetables, put them in a bowl, carry the bowl across the kitchen, fry them, put them in another bowl, carry it back, and then season them. Most of the time goes into moving bowls around, not into cooking.

Operator fusion is like doing all three steps in one pan without leaving the stove. The phone or chip that runs an AI model does the same math as before, but it stops shuffling data back and forth to its memory between steps. The result is identical, yet it arrives sooner and uses less battery.

A typical use case

A team building a real-time object detector for a handheld barcode and parcel scanner converts its trained model into an INT8 format for the device’s neural processing unit. The first test shows about 38 milliseconds per frame, too slow for the 30 frames per second the product requires.

Profiling reveals that a custom activation function prevents the runtime from fusing convolutions with their activations, and that several layers dequantize back to floating point. The engineers replace the custom activation with the standard hardswish, fold batch normalization before calibration, and reconvert. The runtime now fuses convolution, activation and requantization into single kernels across the whole backbone.

Latency drops to about 21 milliseconds per frame with no measurable change in detection accuracy, and the device runs noticeably cooler during a full warehouse shift. No retraining was needed; the gain came entirely from executing the same computation more efficiently.