A KV cache (key-value cache) is the memory buffer in which a transformer language model stores the attention keys and values of already processed tokens, so that each newly generated token only needs to compute its own keys, values and queries instead of reprocessing the entire sequence. In AI infrastructure, the KV cache is the main reason autoregressive generation is fast enough to serve, and at the same time the main consumer of GPU memory during inference.
Because the cache grows with every token of every active request, its size decides how many users a GPU can serve at once and how long a context it can hold. Much of modern inference engineering, from paged memory managers to cache quantization, exists to manage this one data structure.
How the KV cache speeds up token generation
In self-attention, every token attends to all previous tokens through their key and value vectors. Without a cache, generating token number n would require recomputing keys and values for all n minus one earlier tokens in every layer, so the work per step would grow with the sequence length. With a cache, those vectors are computed once and reused, which turns each decoding step into a single-token forward pass plus a read of the stored context.
Inference therefore splits into two phases. In the prefill phase, the whole prompt is processed in parallel and its keys and values are written into the cache; this phase is compute-bound. In the decode phase, tokens are generated one at a time, each step reading the full cache; this phase is memory-bandwidth-bound, because the GPU spends most of its time moving cached tensors rather than doing arithmetic.
| Property | Prefill | Decode |
|---|---|---|
| Tokens per step | Entire prompt at once | One token per sequence |
| Cache operation | Writes keys and values | Reads all, appends one entry |
| Bottleneck | Compute throughput | Memory bandwidth |
| Latency metric | Time to first token | Time per output token |
| Benefit of batching | Moderate | Large, amortizes cache reads |
Estimating KV cache memory
KV cache size can be estimated before deployment with a simple product: 2 (keys and values) × layers × KV heads × head dimension × bytes per value × tokens. The total is multiplied by the number of tokens across all concurrent sequences, so memory grows linearly with both context length and batch size.
For a model with 32 layers, 32 key-value heads of dimension 128 and 16-bit values, each token needs about 0.5 MB of cache. A single 4,096-token sequence then occupies roughly 2 GB, and 32 such sequences need around 64 GB, which can exceed the weights of the model itself.
| Parameter | Typical range | Effect on memory |
|---|---|---|
| Number of layers | About 24 to over 100 | Linear growth |
| KV heads | 1 to the full head count | Linear; reduced by MQA and GQA |
| Head dimension | 64 to 256 | Linear growth |
| Precision | 16-bit, 8-bit or 4-bit | Halves with each step down |
| Context length | Thousands to over 100,000 tokens | Linear growth per sequence |
| Concurrent sequences | 1 to hundreds | Linear growth in total |
Techniques that shrink or manage the cache
Cache optimizations work at two levels: model architecture, which changes how much state each token produces, and serving software, which changes how that state is stored and shared. Both are usually combined in production systems.
| Technique | Level | How it helps | Trade-off |
|---|---|---|---|
| Multi-query attention | Architecture | One shared KV head for all queries | Possible quality loss |
| Grouped-query attention | Architecture | Query groups share KV heads | Smaller savings than MQA |
| Latent attention compression | Architecture | Stores a compressed latent per token | More complex kernels |
| Sliding-window attention | Architecture | Keeps only recent tokens | Distant context is lost |
| Paged allocation | Serving | Fixed-size blocks remove fragmentation | Block-table indirection |
| Cache quantization | Serving | 8-bit or 4-bit keys and values | Small accuracy risk |
| Prefix sharing | Serving | Identical prompt prefixes reuse blocks | Needs exact token match |
| Offloading | Serving | Moves cold blocks to CPU memory | Transfer latency |
Paged allocation, popularized by the vLLM engine, deserves special mention. Earlier servers reserved a contiguous region for the maximum context of each request, wasting much of it; storing the cache in small fixed-size blocks allocated on demand allows far more concurrent sequences on the same GPU.
KV cache and related serving concepts
The KV cache is often confused with prompt caching, but they operate at different time scales. The KV cache lives for the duration of one request; prompt caching keeps the cache of a frequently repeated prefix, such as a long system prompt, across requests, so that later requests skip part of the prefill.
| Concept | Relation to the KV cache |
|---|---|
| Prompt caching | Persists cache entries of shared prefixes between requests |
| Speculative decoding | Draft tokens are verified and accepted cache entries are kept |
| Continuous batching | Admits new requests as cache blocks become free |
| Model compression | Shrinks weights, while cache size depends on context |
| Model routing | Can send requests to replicas that already hold a prefix |
Operational pitfalls
Teams that size inference hardware only by model weights regularly run into out-of-memory errors under load. Typical problems and their remedies are listed below.
- Ignoring concurrency: capacity planning must use realistic batch sizes and context lengths, not a single request.
- Overstated context limits: advertising a very long context without reserving cache memory forces request rejection or preemption.
- Preemption thrashing: when the cache fills, sequences are evicted and recomputed, which can collapse throughput.
- Unvalidated cache quantization: low-bit caches can degrade long-context retrieval more than short benchmarks reveal.
- Prefix mismatches: a changing timestamp at the start of a system prompt silently disables prefix reuse.
A sound practice is to monitor cache utilization and preemption counts alongside latency, because they explain throughput drops that GPU utilization alone does not show.
The short version
When a language model writes an answer, it produces one word piece at a time, and each new piece has to look back at everything written so far. The KV cache is the model’s notepad of what it has already read. Instead of rereading the whole conversation for every new word, it glances at its notes.
The notes make writing fast, but they take up space. Long conversations and many simultaneous users fill the notepad quickly, so managing that memory is a central job of any system that serves language models.
Where it makes a difference
A company runs an internal assistant on a single 80 GB GPU with a model whose 16-bit weights take about 16 GB. Early tests with one user look fine, but at launch, 40 employees send requests with 6,000-token documents, and the server starts rejecting requests. A quick calculation shows that the KV cache for that load would need well over 100 GB.
The team switches to a serving engine with paged cache allocation, enables 8-bit cache quantization after confirming accuracy on its own document questions, and turns on prefix caching for the shared system prompt. The same GPU now holds all concurrent sessions, and time to first token drops noticeably for repeated questions, without buying additional hardware.