Multi-head latent attention (MLA) is a transformer attention variant that compresses keys and values into a shared low-dimensional latent vector for each token, caches only that compact vector during generation, and reconstructs per-head keys and values from it when attention is computed. Among attention designs for neural networks, it targets the memory cost of the KV cache, which dominates inference expenses for long contexts.
MLA was introduced with the DeepSeek-V2 language model in 2024 and was later adopted in DeepSeek-V3 and several other open-weights models. It aims to keep the modeling quality of full multi-head attention while reaching a cache footprint comparable to or smaller than that of grouped-query attention.
How multi-head latent attention compresses keys and values
MLA replaces the separate key and value projections of standard attention with a two-stage low-rank factorization. The hidden state of each token is first multiplied by a down-projection matrix that produces a latent vector of dimension d_c, and up-projection matrices then expand this latent into full keys and values for every attention head.
During autoregressive decoding, only the latent vector is stored in the cache. In DeepSeek-V2, the latent has 512 dimensions, whereas the model has 128 heads with 128 dimensions each, so standard multi-head attention would cache 32,768 key and value elements per token per layer. The compression ratio is therefore dozens of times smaller per layer.
The queries can also be compressed through their own latent, which reduces activation memory during training but does not affect the cache. The main components and their roles are summarized below.
| Component | Function | Cached at inference |
|---|---|---|
| KV down-projection | Maps the hidden state to the latent vector | Output is cached |
| Key up-projection | Expands the latent into per-head keys | No |
| Value up-projection | Expands the latent into per-head values | No |
| Query latent | Optional low-rank path for queries | No |
| Decoupled RoPE key | Carries positional information separately | Yes, small vector |
Weight absorption and decoupled rotary embeddings
MLA avoids rebuilding full keys and values at inference through weight absorption. Because attention scores are bilinear, the key up-projection can be merged algebraically into the query projection, and the value up-projection can be merged into the output projection. Attention then operates directly on the cached latent vectors, which keeps decoding fast.
Absorption breaks when rotary position embedding is applied to keys, because the position-dependent rotation sits between the two matrices and prevents them from being merged. DeepSeek solved this with a decoupled RoPE path: a small extra query and key component, 64 dimensions per token in DeepSeek-V2, carries positional information and is concatenated with the content part.
The decoupled key is shared across all heads and cached alongside the latent. The total cache per token per layer in DeepSeek-V2 is therefore 512 plus 64, or 576 elements, a figure that the designers compared to grouped-query attention with only about two groups.
Multi-head latent attention compared with other attention variants
MLA belongs to a family of designs that trade expressiveness against cache size. Multi-query attention and grouped-query attention reduce memory by sharing heads, while MLA reduces memory by compressing information rather than discarding heads. The table compares cache sizes for a model with n_h heads of dimension d_h.
| Variant | Cached elements | Quality relative to full attention |
|---|---|---|
| Multi-head attention | 2 × n_h × d_h | Reference |
| Grouped-query attention | 2 × n_g × d_h | Slightly lower in many reports |
| Multi-query attention | 2 × d_h | Noticeably lower |
| Multi-head latent attention | d_c + d_h^R | Comparable or higher in published ablations |
MLA is orthogonal to sparse and linear attention. Those methods reduce the number of token pairs that interact, while MLA reduces the size of each cached entry, so the two approaches can be combined in principle.
Key hyperparameters and design choices
The behavior of MLA is set mainly by the latent dimension and the size of the positional path. A larger latent preserves more information but enlarges the cache, while a smaller latent saves memory at the risk of degraded long-range recall.
| Parameter | Symbol | DeepSeek-V2 value | Effect of increasing it |
|---|---|---|---|
| KV latent dimension | d_c | 512 | Better fidelity, larger cache |
| Query latent dimension | d_c’ | 1,536 | More query capacity, more training memory |
| Decoupled RoPE dimension | d_h^R | 64 per head | Finer positional signal, slightly larger cache |
| Number of heads | n_h | 128 | More attention patterns, no cache growth |
A notable property is that adding heads does not grow the cache, because every head reads from the same latent. This lets designers increase head count for representational richness without paying the memory penalty that standard multi-head attention would impose.
Limitations and practical considerations
MLA introduces implementation complexity that simpler variants avoid. Efficient serving requires dedicated kernels that exploit absorption, and inference frameworks needed explicit support before MLA models ran at competitive speed. The table lists common issues.
| Issue | Cause | Remedy |
|---|---|---|
| Slow decoding | Keys and values rebuilt for every step | Use absorbed matrices in the kernel |
| Positional errors | RoPE applied to compressed keys | Keep the decoupled RoPE path |
| Compute-bound attention | Absorbed queries are wider | Tune batch size and kernel tiling |
| Retrofit quality loss | Converting a trained GQA model | Fine-tune after conversion |
Absorption shifts work from memory traffic to arithmetic, so MLA is most beneficial when decoding is memory-bound, as in long-context or high-concurrency serving. Research methods for converting existing models to MLA exist, but retrofitted models usually need additional training to recover quality.
The short version
When a chatbot writes a long answer, it keeps notes about every earlier word so it does not have to reread everything. Those notes take up considerable memory. Multi-head latent attention stores the notes in a compressed shorthand and expands them only when needed.
The result is that the same hardware can handle longer conversations or more users at once. Quality stays close to the full-size method, which is why several recent language models use it.
Where it makes a difference
Consider a company serving a large language model to internal users with documents of up to 100,000 tokens. With standard multi-head attention in a model of DeepSeek-V2 dimensions, the KV cache for a single long request would require hundreds of gigabytes in 16-bit precision, far more than one server can hold alongside the weights.
With MLA, each token stores 576 elements per layer instead of 32,768, so the same request needs less than one fiftieth of that memory. The operations team can keep many long-context sessions on one multi-GPU node, batch more requests together and raise throughput substantially, while users notice no meaningful drop in answer quality.