Inference optimization

On this page 7

Inference optimization is the discipline of making a trained model produce predictions faster, cheaper and at larger scale without an unacceptable loss of quality. In AI infrastructure, it covers changes to the model itself, to the numerical kernels that execute it and to the serving system that schedules requests across GPUs or other accelerators.

For large language models, inference rather than training usually dominates lifetime compute spending, because a model trained once may answer billions of requests. The goals are expressed through measurable targets such as time to first token, time per output token, throughput in tokens per second and cost per token.

Where inference time and cost come from

Inference cost is determined by how much computation and memory traffic each token requires and by how well the hardware is kept busy. Transformer inference has two phases with very different profiles, so optimization starts by identifying which phase limits a given workload.

The prefill phase processes the whole prompt in parallel and is usually compute-bound. The decode phase generates one token at a time and is usually memory-bandwidth-bound, because every step must read the model weights and the growing KV cache from accelerator memory while performing relatively little arithmetic.

The following table summarizes the two phases and the metrics they drive.

PhaseMain bottleneckMetric it drivesTypical remedies
PrefillArithmetic throughputTime to first tokenFlashAttention, prompt caching, chunked prefill
DecodeMemory bandwidthTime per output tokenQuantization, batching, speculative decoding
Both phasesMemory capacity for KV cacheMaximum concurrencyPagedAttention, KV cache compression
Prefill and decode phases compared

This distinction explains a common surprise: a GPU may report low utilization during decoding even when it is the bottleneck, because it spends most of each step waiting on memory rather than computing.

Main families of optimization techniques

Inference optimization techniques fall into three layers: model, kernel and system. Gains from different layers usually multiply, which is why mature serving stacks apply several at once.

LayerTechniqueTypical effectMain risk
ModelQuantization to 8 or 4 bitsLess memory, faster decodeAccuracy loss on hard tasks
ModelDistillation and pruningSmaller, cheaper modelRetraining effort, capability gaps
KernelFused attention kernelsFewer memory round tripsHardware-specific code
SystemContinuous batchingMuch higher throughputHigher tail latency if overloaded
SystemSpeculative decodingFewer large-model stepsGains depend on draft acceptance
SystemPrompt and prefix cachingSkips repeated prefillCache invalidation and memory use
Optimization techniques by layer

Quantization is often the first lever because it reduces both memory footprint and bandwidth per token. Moving from 16-bit to 8-bit weights roughly halves weight memory, and well-calibrated 8-bit formats usually show small quality changes, while 4-bit formats require more careful evaluation.

Continuous batching replaces fixed batches with a scheduler that adds and removes sequences at every decoding step. Combined with paged KV cache management, as popularized by vLLM, it can raise throughput several-fold on the same hardware for mixed-length traffic.

Latency and throughput trade-offs

The central tension in inference optimization is that larger batches raise throughput but slow each individual request. Serving systems therefore tune for a service-level objective, such as a maximum time to first token, and then maximize throughput within it.

WorkloadPrimary targetPreferred techniques
Interactive chatLow time to first tokenPrefix caching, moderate batches, speculative decoding
Voice agentsVery low end-to-end latencySmall or quantized models, streaming output
Offline batch jobsMaximum tokens per dollarLarge batches, aggressive quantization
Long-document analysisFit long context in memoryKV cache compression, context parallelism
Code completionVery fast short outputsSpeculative decoding, small specialized models
Optimization goals by workload

Disaggregated serving addresses the trade-off structurally by running prefill and decode on separate pools of accelerators, so that long prompts do not stall the token stream of other users.

Inference optimization is an objective that spans several neighboring practices rather than a single tool. Model compression, model serving and model routing each contribute to it from a different angle.

  • Model compression changes the model weights or architecture; inference optimization also includes runtime and scheduling changes that leave the model untouched.
  • Model serving provides the infrastructure that hosts models; optimization tunes how that infrastructure executes them.
  • Model routing sends easy requests to cheaper models, which lowers average cost without changing any single model.
  • Training optimization targets gradient computation and communication, which follow different bottlenecks from token generation.

Common mistakes and limitations

The most frequent error is optimizing a benchmark instead of the real traffic profile. Results measured on short prompts with a single user rarely predict behavior under hundreds of concurrent long-context sessions.

SymptomLikely causeCorrection
Quality drops after quantizationNo task-specific evaluationEvaluate on representative tasks before rollout
Good averages, poor tail latencyOverloaded batching schedulerTrack p95 and p99, cap batch size
Out-of-memory errors at peakKV cache sized for short promptsPlan memory for realistic context lengths
Small gain from speculative decodingLow draft acceptance rateUse a better aligned draft model
Typical pitfalls and corrections

Every technique has a quality, latency or engineering cost, and gains are workload-specific. Reliable teams measure accuracy, tail latency and cost together and treat each optimization as a reversible experiment.

The short version

Inference optimization is about making an AI model answer faster and for less money once it has already been trained. Think of a busy kitchen: the recipes stay the same, but the cooks prepare several orders at once, reuse ingredients that are already chopped and use lighter equipment where it does not hurt the taste.

The challenge is balance. Serving more people at once can make each person wait a little longer, and shrinking the model can slightly lower quality. Good optimization finds the point where users stay happy and costs fall.

Where it makes a difference

A customer support platform runs a 70-billion-parameter language model on a cluster of eight high-memory GPUs. At peak hours, responses take several seconds to start, and the monthly accelerator bill keeps rising as more clients join.

The engineering team first measures real traffic and finds that most conversations share a long system prompt. It enables prefix caching for that prompt, switches to a serving engine with continuous batching and paged KV cache memory, and quantizes the weights to 8 bits after confirming on a set of past tickets that answer quality is unchanged. Simple questions are routed to a smaller model.

After these changes, time to first token falls well under one second for most requests, and the same cluster serves roughly three times as many concurrent conversations. The team keeps monitoring p99 latency and answer quality weekly so that later tuning does not quietly erode either one.