Early exit network

Also known as: early exiting

On this page 7

An early exit network is a neural network that contains intermediate classifiers, called exits, attached to hidden layers, so that inference can stop before the final layer whenever an intermediate prediction is already confident enough. In Edge AI, where models run on phones, PCs and embedded devices with tight budgets for latency and energy, early exiting turns a fixed-cost model into one whose compute adapts to the difficulty of each input.

The idea rests on a simple observation: most real inputs are easy. A clear photograph of a cat, a loud wake word or a short and unambiguous sentence can often be classified correctly by the first third of a network. Spending the full depth on such samples wastes cycles, battery and thermal headroom. Early exiting, also known simply as early exiting or adaptive depth inference, keeps the full model available for hard samples while letting easy samples leave early.

How an early exit network decides when to stop

An early exit network decides to stop by comparing a confidence score at each exit with a threshold. The backbone, for example a ResNet, a MobileNet or a transformer encoder, is divided into stages. After selected stages, a small exit head (typically global pooling followed by one or two fully connected layers) produces a full prediction. If the score passes the threshold, the result is returned and the remaining layers are never executed.

The most common confidence criteria are the maximum softmax probability, the entropy of the output distribution and patience-based agreement, in which inference stops once several consecutive exits predict the same class. Learned criteria also exist: a tiny gating network predicts whether continuing is likely to change the answer. The following table compares the main exit criteria.

CriterionExit conditionStrengthWeakness
Max softmax probabilityTop-class probability above thresholdTrivial to computeOverconfident heads exit wrongly
Output entropyEntropy below thresholdUses the whole distributionThreshold depends on class count
Patience-basedSame prediction at k consecutive exitsRobust to single-head noiseDelays exit by k stages
Learned gateGate network outputs exitCan optimize a cost objectiveExtra training and parameters
Common exit criteria and their behavior

Thresholds are not universal constants. They are calibrated on a validation set to reach a target accuracy or a target average cost, and a single trained model can serve several operating points simply by changing the threshold at deployment time.

Training the exits and placing them in the backbone

Early exit networks are trained either jointly, with a weighted sum of losses from every exit, or in two stages, where a pretrained backbone is frozen and only the exit heads are trained. Joint training usually gives better early exits but can slightly degrade the final exit, because early layers are pulled toward producing linearly separable features too soon. Weighting the deeper losses more heavily and using self-distillation, in which the final exit teaches the earlier ones, reduces this conflict.

Exit placement is a design trade-off. Each head adds parameters, memory traffic and a small amount of latency even when it does not fire. The table below summarizes the parameters an engineer tunes when building an early exit network for a device.

ParameterTypical rangeWhat it affects
Number of exits2 to 6 exitsGranularity of cost savings and overhead
Exit positionAfter 25-75% of depthAccuracy of early predictions
Head size1-5% of backbone parametersExit accuracy versus model footprint
Confidence threshold0.5 to 0.99 probabilityBalance of speed and accuracy
Loss weightsRising with depthQuality of final versus early exits
Design parameters of an early exit network

In practice, published image and language results commonly report average compute reductions of roughly 30-60% with an accuracy loss of under one percentage point, but the savings depend heavily on how easy the deployment data is. A dataset dominated by hard or ambiguous samples yields little benefit.

Early exiting compared with other edge compression methods

Early exiting is a form of dynamic inference, whereas most compression techniques are static: they make every inference cheaper by the same amount. The methods are complementary and are frequently combined, for example an INT8 quantized backbone with early exits on top. The comparison below positions early exiting among its neighbors in the Edge AI toolbox.

TechniqueCost reductionVaries per inputMain risk
Early exit networkSkips later layersYesUnpredictable latency
Post-training quantizationLower bit width for all layersNoAccuracy drop at low bits
Structured pruningRemoves channels or blocksNoCapacity loss if overdone
Knowledge distillationTrains a smaller student modelNoStudent cannot match teacher
Split computingMoves later layers off deviceOptionalNetwork delay and privacy
Early exiting versus related efficiency techniques

Early exits also pair naturally with split computing and hybrid inference. A device runs the first stages locally and returns the answer when an early exit is confident; only uncertain samples send intermediate features to an edge server or the cloud, which cuts both bandwidth and offloading frequency.

Deploying early exits on real edge hardware

Deployment requires a runtime that supports conditional control flow or a model split into separate subgraphs. Many edge inference runtimes and hardware delegates favor static graphs, so engineers often export each stage as its own model and let the application code evaluate the exit rule between calls. This adds a small scheduling overhead but keeps each subgraph eligible for accelerator offload and operator fusion.

Batching is the other practical complication. On a server, samples in one batch may exit at different stages, which breaks dense matrix operations. On a phone or microcontroller that processes one input at a time, this problem largely disappears, which is one reason early exiting suits single-stream edge workloads such as keyword spotting, visual wake words and on-device text classification.

Limitations and common mistakes with early exit networks

The central limitation is variable latency. Average cost falls, but the worst-case latency stays equal to, or slightly above, that of the original model because hard inputs still traverse every stage and every unused head. Real-time systems must therefore be budgeted for the full depth. The following table lists frequent problems.

SymptomLikely causeRemedy
Early exits wrong but confidentPoor calibration of exit headsTemperature scaling per exit
Final accuracy droppedJoint loss favors early layersIncrease deep-exit weights or distill
No measured speedupHeads too large or runtime overheadSmaller heads, fewer exits
Savings vanish in the fieldDistribution shift to harder dataRecalibrate thresholds on field data
Missed real-time deadlinesBudgeted for average latencyPlan for worst-case path
Typical problems, causes and remedies
  • Calibration matters more than accuracy at each exit: a head that is accurate but overconfident on its errors silently lowers final quality.
  • Exit decisions can be exploited: adversarial inputs can be crafted to force every sample to the deepest exit, eroding energy savings.
  • Energy savings must be measured on the target device, because memory access and wake-up costs do not scale linearly with skipped FLOPs.

A plain-language summary

An early exit network is a model that is allowed to answer before it has finished thinking when the question is easy. It has several checkpoints along the way. At each checkpoint it asks itself whether it is sure. If it is, it stops and gives the answer; if not, it keeps working.

Think of a librarian who recognizes popular books at a glance but walks to the catalog only for rare titles. Most visitors are served in seconds, and the hard requests still get full attention. For a phone or a small sensor, this means less battery used and faster answers on average, without giving up the ability to handle difficult cases.

An example from practice

A team building a battery-powered wildlife camera wants to classify every motion-triggered frame on the device and transmit only images of target species. Their MobileNet-style classifier takes about 40 ms per frame on the camera’s microcontroller-class processor, which limits battery life to a few weeks.

They add two exit heads, after roughly one third and two thirds of the network, train them with self-distillation from the final layer, and calibrate an entropy threshold on 5,000 labeled field images. Most frames show empty vegetation moved by wind, and the first exit handles them confidently. In field tests, about 70% of frames leave at the first exit and the average inference time falls to roughly 18 ms, while species accuracy stays within half a percentage point of the original model.

The team still sizes the processing schedule for the full 40 ms path, because a burst of ambiguous night frames can push every sample to the final exit. The result is a camera that runs nearly twice as long between battery changes.