GPU inference is the execution of a trained machine learning model on graphics processing units to produce predictions, embeddings, images or generated text for real requests. In AI infrastructure, it is the production side of the model lifecycle: training happens once or occasionally, while inference runs continuously for every user request and usually accounts for most of a deployed model’s long-term compute cost.
GPUs dominate inference for large models because neural network workloads consist mainly of matrix multiplications, which GPUs execute across thousands of parallel cores, and because they provide high-bandwidth memory to hold model weights close to the compute units. The engineering challenge is to keep these expensive processors busy while meeting latency targets.
How inference runs on a GPU
During inference, model weights are loaded into GPU memory (VRAM), inputs are batched and sent to the device, and optimized compute kernels perform the layer operations. For large language models, generation has two distinct phases with very different performance profiles.
| Phase | What happens | Main bottleneck |
|---|---|---|
| Prefill | The whole prompt is processed in parallel and the KV cache is built | Compute throughput |
| Decode | Tokens are generated one at a time, each reading all weights | Memory bandwidth |
In the decode phase, every new token requires reading the full set of model weights from memory. A model with 70 billion parameters stored in 16-bit precision occupies about 140 GB, so per-token speed for a single request is limited by how fast memory can be read, not by arithmetic. Batching many requests lets the GPU reuse each weight read for many tokens, which is the key to efficient throughput.
The KV cache stores attention keys and values for previous tokens so they are not recomputed. It grows with context length and the number of concurrent requests and often consumes as much memory as the weights themselves in long-context workloads.
Performance metrics and their trade-offs
GPU inference is tuned against several metrics that pull in different directions. The central tension is between latency for each user and throughput for the whole system.
| Metric | Definition | Interactive target range |
|---|---|---|
| Time to first token (TTFT) | Delay before the first output token appears | Typically under 0.5-2 seconds |
| Inter-token latency | Time between consecutive output tokens | Often 10-50 ms, faster than reading speed |
| Throughput | Tokens or requests processed per second per GPU | Maximized within latency limits |
| GPU utilization | Share of compute capacity in use | Higher means lower cost per token |
| Cost per million tokens | Hardware cost divided by tokens served | Depends on all the above |
Larger batches raise throughput and lower cost per token but increase latency for each request. Production systems set a latency budget, often expressed as a service level objective, and then fit the largest batch that respects it.
Optimization techniques
Most of the efficiency in GPU inference comes from software optimization rather than faster hardware. Inference servers combine several techniques, each addressing a different bottleneck.
| Technique | What it does | Main benefit |
|---|---|---|
| Continuous batching | Adds and removes requests from a running batch at each step | Much higher utilization |
| Paged KV cache | Stores the cache in fixed-size blocks like virtual memory | Less fragmentation, more concurrent users |
| Quantization | Stores weights in 8-bit or 4-bit formats | Less memory, faster decoding |
| Fused attention kernels | Computes attention without materializing large matrices | Faster prefill, less memory |
| Speculative decoding | A small draft model proposes tokens the large model verifies | Lower latency per output |
| Prompt caching | Reuses the KV cache for repeated prompt prefixes | Faster and cheaper repeated prompts |
When a model does not fit on one GPU, it is split. Tensor parallelism divides each layer across GPUs connected by fast links, while pipeline parallelism places different layers on different GPUs. Tensor parallelism lowers latency but needs high-bandwidth interconnects between devices.
Choosing hardware and deployment
Hardware selection starts with memory capacity: the GPU or group of GPUs must hold the weights, the KV cache for the expected concurrency and runtime overhead. Compute speed and memory bandwidth then determine latency and throughput.
| Workload | Typical choice | Reason |
|---|---|---|
| Small models and embeddings | Single mid-range data center or workstation GPU | Low memory needs, high request volume |
| Mid-size LLM, quantized | One GPU with 24-80 GB of memory | Weights and cache fit on one device |
| Large LLM at full precision | Multiple high-memory GPUs with fast interconnect | Weights exceed one device |
| Image and video generation | High-compute GPUs | Diffusion steps are compute-bound |
| Spiky, low-volume traffic | Serverless or shared GPU capacity | Avoids paying for idle hardware |
GPU inference is usually deployed behind an inference server that handles batching, scheduling and the model API, as part of a broader model serving platform. Model routing can send simple requests to smaller, cheaper models and reserve large GPUs for difficult ones.
Common pitfalls
Many inference deployments are more expensive than necessary because of avoidable configuration mistakes.
- Sizing memory for weights only and running out when the KV cache grows with long contexts.
- Benchmarking with a single request, which hides how latency behaves under real concurrency.
- Leaving GPUs idle between requests because static batching waits for full batches.
- Quantizing without evaluation, which can quietly degrade accuracy on specific tasks.
- Ignoring cold starts: loading tens of gigabytes of weights can take minutes when scaling out.
The reliable approach is to load test with realistic prompt and output lengths, measure latency percentiles such as p95 and p99 rather than averages, and validate output quality after every optimization.
Explained without jargon
GPU inference is the moment an AI model actually does its job for someone. Training teaches the model; inference is when it answers a question, writes a reply or creates a picture.
Graphics chips are used because they can do thousands of small calculations at the same time, which is exactly what AI models need. The hard part is serving many people at once. Nobody should wait too long. It is a bit like a busy kitchen that cooks many orders together, so each meal still arrives hot and on time.
A typical use case
An online retailer runs a customer-service chatbot on an open 8-billion-parameter language model. The first deployment uses a basic server with static batching on two GPUs, and during evening peaks users wait several seconds for the first token.
The infrastructure team moves the model to an inference server with continuous batching and a paged KV cache, quantizes the weights to 8 bits after confirming answer quality on 500 test conversations and enables prompt caching for the long system prompt shared by every session.
Under the same peak load, time to first token falls to well under one second, and one GPU now handles the traffic that previously needed two. The team uses the freed capacity to serve a second model for order-status lookups, routed automatically by request type.