An early exit network is a neural network that contains intermediate classifiers, called exits, attached to hidden layers, so that inference can stop before the final layer whenever an intermediate prediction is already confident enough. In Edge AI, where models run on phones, PCs and embedded devices with tight budgets for latency and energy, early exiting turns a fixed-cost model into one whose compute adapts to the difficulty of each input.
The idea rests on a simple observation: most real inputs are easy. A clear photograph of a cat, a loud wake word or a short and unambiguous sentence can often be classified correctly by the first third of a network. Spending the full depth on such samples wastes cycles, battery and thermal headroom. Early exiting, also known simply as early exiting or adaptive depth inference, keeps the full model available for hard samples while letting easy samples leave early.
How an early exit network decides when to stop
An early exit network decides to stop by comparing a confidence score at each exit with a threshold. The backbone, for example a ResNet, a MobileNet or a transformer encoder, is divided into stages. After selected stages, a small exit head (typically global pooling followed by one or two fully connected layers) produces a full prediction. If the score passes the threshold, the result is returned and the remaining layers are never executed.
The most common confidence criteria are the maximum softmax probability, the entropy of the output distribution and patience-based agreement, in which inference stops once several consecutive exits predict the same class. Learned criteria also exist: a tiny gating network predicts whether continuing is likely to change the answer. The following table compares the main exit criteria.
| Criterion | Exit condition | Strength | Weakness |
|---|---|---|---|
| Max softmax probability | Top-class probability above threshold | Trivial to compute | Overconfident heads exit wrongly |
| Output entropy | Entropy below threshold | Uses the whole distribution | Threshold depends on class count |
| Patience-based | Same prediction at k consecutive exits | Robust to single-head noise | Delays exit by k stages |
| Learned gate | Gate network outputs exit | Can optimize a cost objective | Extra training and parameters |
Thresholds are not universal constants. They are calibrated on a validation set to reach a target accuracy or a target average cost, and a single trained model can serve several operating points simply by changing the threshold at deployment time.
Training the exits and placing them in the backbone
Early exit networks are trained either jointly, with a weighted sum of losses from every exit, or in two stages, where a pretrained backbone is frozen and only the exit heads are trained. Joint training usually gives better early exits but can slightly degrade the final exit, because early layers are pulled toward producing linearly separable features too soon. Weighting the deeper losses more heavily and using self-distillation, in which the final exit teaches the earlier ones, reduces this conflict.
Exit placement is a design trade-off. Each head adds parameters, memory traffic and a small amount of latency even when it does not fire. The table below summarizes the parameters an engineer tunes when building an early exit network for a device.
| Parameter | Typical range | What it affects |
|---|---|---|
| Number of exits | 2 to 6 exits | Granularity of cost savings and overhead |
| Exit position | After 25-75% of depth | Accuracy of early predictions |
| Head size | 1-5% of backbone parameters | Exit accuracy versus model footprint |
| Confidence threshold | 0.5 to 0.99 probability | Balance of speed and accuracy |
| Loss weights | Rising with depth | Quality of final versus early exits |
In practice, published image and language results commonly report average compute reductions of roughly 30-60% with an accuracy loss of under one percentage point, but the savings depend heavily on how easy the deployment data is. A dataset dominated by hard or ambiguous samples yields little benefit.
Early exiting compared with other edge compression methods
Early exiting is a form of dynamic inference, whereas most compression techniques are static: they make every inference cheaper by the same amount. The methods are complementary and are frequently combined, for example an INT8 quantized backbone with early exits on top. The comparison below positions early exiting among its neighbors in the Edge AI toolbox.
| Technique | Cost reduction | Varies per input | Main risk |
|---|---|---|---|
| Early exit network | Skips later layers | Yes | Unpredictable latency |
| Post-training quantization | Lower bit width for all layers | No | Accuracy drop at low bits |
| Structured pruning | Removes channels or blocks | No | Capacity loss if overdone |
| Knowledge distillation | Trains a smaller student model | No | Student cannot match teacher |
| Split computing | Moves later layers off device | Optional | Network delay and privacy |
Early exits also pair naturally with split computing and hybrid inference. A device runs the first stages locally and returns the answer when an early exit is confident; only uncertain samples send intermediate features to an edge server or the cloud, which cuts both bandwidth and offloading frequency.
Deploying early exits on real edge hardware
Deployment requires a runtime that supports conditional control flow or a model split into separate subgraphs. Many edge inference runtimes and hardware delegates favor static graphs, so engineers often export each stage as its own model and let the application code evaluate the exit rule between calls. This adds a small scheduling overhead but keeps each subgraph eligible for accelerator offload and operator fusion.
Batching is the other practical complication. On a server, samples in one batch may exit at different stages, which breaks dense matrix operations. On a phone or microcontroller that processes one input at a time, this problem largely disappears, which is one reason early exiting suits single-stream edge workloads such as keyword spotting, visual wake words and on-device text classification.
Limitations and common mistakes with early exit networks
The central limitation is variable latency. Average cost falls, but the worst-case latency stays equal to, or slightly above, that of the original model because hard inputs still traverse every stage and every unused head. Real-time systems must therefore be budgeted for the full depth. The following table lists frequent problems.
| Symptom | Likely cause | Remedy |
|---|---|---|
| Early exits wrong but confident | Poor calibration of exit heads | Temperature scaling per exit |
| Final accuracy dropped | Joint loss favors early layers | Increase deep-exit weights or distill |
| No measured speedup | Heads too large or runtime overhead | Smaller heads, fewer exits |
| Savings vanish in the field | Distribution shift to harder data | Recalibrate thresholds on field data |
| Missed real-time deadlines | Budgeted for average latency | Plan for worst-case path |
- Calibration matters more than accuracy at each exit: a head that is accurate but overconfident on its errors silently lowers final quality.
- Exit decisions can be exploited: adversarial inputs can be crafted to force every sample to the deepest exit, eroding energy savings.
- Energy savings must be measured on the target device, because memory access and wake-up costs do not scale linearly with skipped FLOPs.
A plain-language summary
An early exit network is a model that is allowed to answer before it has finished thinking when the question is easy. It has several checkpoints along the way. At each checkpoint it asks itself whether it is sure. If it is, it stops and gives the answer; if not, it keeps working.
Think of a librarian who recognizes popular books at a glance but walks to the catalog only for rare titles. Most visitors are served in seconds, and the hard requests still get full attention. For a phone or a small sensor, this means less battery used and faster answers on average, without giving up the ability to handle difficult cases.
An example from practice
A team building a battery-powered wildlife camera wants to classify every motion-triggered frame on the device and transmit only images of target species. Their MobileNet-style classifier takes about 40 ms per frame on the camera’s microcontroller-class processor, which limits battery life to a few weeks.
They add two exit heads, after roughly one third and two thirds of the network, train them with self-distillation from the final layer, and calibrate an entropy threshold on 5,000 labeled field images. Most frames show empty vegetation moved by wind, and the first exit handles them confidently. In field tests, about 70% of frames leave at the first exit and the average inference time falls to roughly 18 ms, while species accuracy stays within half a percentage point of the original model.
The team still sizes the processing schedule for the full 40 ms path, because a burst of ambiguous night frames can push every sample to the final exit. The result is a camera that runs nearly twice as long between battery changes.