Multi-head latent attention

Also known as: MLA

On this page 7

Multi-head latent attention (MLA) is a transformer attention variant that compresses keys and values into a shared low-dimensional latent vector for each token, caches only that compact vector during generation, and reconstructs per-head keys and values from it when attention is computed. Among attention designs for neural networks, it targets the memory cost of the KV cache, which dominates inference expenses for long contexts.

MLA was introduced with the DeepSeek-V2 language model in 2024 and was later adopted in DeepSeek-V3 and several other open-weights models. It aims to keep the modeling quality of full multi-head attention while reaching a cache footprint comparable to or smaller than that of grouped-query attention.

How multi-head latent attention compresses keys and values

MLA replaces the separate key and value projections of standard attention with a two-stage low-rank factorization. The hidden state of each token is first multiplied by a down-projection matrix that produces a latent vector of dimension d_c, and up-projection matrices then expand this latent into full keys and values for every attention head.

During autoregressive decoding, only the latent vector is stored in the cache. In DeepSeek-V2, the latent has 512 dimensions, whereas the model has 128 heads with 128 dimensions each, so standard multi-head attention would cache 32,768 key and value elements per token per layer. The compression ratio is therefore dozens of times smaller per layer.

The queries can also be compressed through their own latent, which reduces activation memory during training but does not affect the cache. The main components and their roles are summarized below.

ComponentFunctionCached at inference
KV down-projectionMaps the hidden state to the latent vectorOutput is cached
Key up-projectionExpands the latent into per-head keysNo
Value up-projectionExpands the latent into per-head valuesNo
Query latentOptional low-rank path for queriesNo
Decoupled RoPE keyCarries positional information separatelyYes, small vector
Components of a multi-head latent attention layer

Weight absorption and decoupled rotary embeddings

MLA avoids rebuilding full keys and values at inference through weight absorption. Because attention scores are bilinear, the key up-projection can be merged algebraically into the query projection, and the value up-projection can be merged into the output projection. Attention then operates directly on the cached latent vectors, which keeps decoding fast.

Absorption breaks when rotary position embedding is applied to keys, because the position-dependent rotation sits between the two matrices and prevents them from being merged. DeepSeek solved this with a decoupled RoPE path: a small extra query and key component, 64 dimensions per token in DeepSeek-V2, carries positional information and is concatenated with the content part.

The decoupled key is shared across all heads and cached alongside the latent. The total cache per token per layer in DeepSeek-V2 is therefore 512 plus 64, or 576 elements, a figure that the designers compared to grouped-query attention with only about two groups.

Multi-head latent attention compared with other attention variants

MLA belongs to a family of designs that trade expressiveness against cache size. Multi-query attention and grouped-query attention reduce memory by sharing heads, while MLA reduces memory by compressing information rather than discarding heads. The table compares cache sizes for a model with n_h heads of dimension d_h.

VariantCached elementsQuality relative to full attention
Multi-head attention2 × n_h × d_hReference
Grouped-query attention2 × n_g × d_hSlightly lower in many reports
Multi-query attention2 × d_hNoticeably lower
Multi-head latent attentiond_c + d_h^RComparable or higher in published ablations
KV cache per token per layer for common attention variants

MLA is orthogonal to sparse and linear attention. Those methods reduce the number of token pairs that interact, while MLA reduces the size of each cached entry, so the two approaches can be combined in principle.

Key hyperparameters and design choices

The behavior of MLA is set mainly by the latent dimension and the size of the positional path. A larger latent preserves more information but enlarges the cache, while a smaller latent saves memory at the risk of degraded long-range recall.

ParameterSymbolDeepSeek-V2 valueEffect of increasing it
KV latent dimensiond_c512Better fidelity, larger cache
Query latent dimensiond_c’1,536More query capacity, more training memory
Decoupled RoPE dimensiond_h^R64 per headFiner positional signal, slightly larger cache
Number of headsn_h128More attention patterns, no cache growth
Main MLA hyperparameters with DeepSeek-V2 values

A notable property is that adding heads does not grow the cache, because every head reads from the same latent. This lets designers increase head count for representational richness without paying the memory penalty that standard multi-head attention would impose.

Limitations and practical considerations

MLA introduces implementation complexity that simpler variants avoid. Efficient serving requires dedicated kernels that exploit absorption, and inference frameworks needed explicit support before MLA models ran at competitive speed. The table lists common issues.

IssueCauseRemedy
Slow decodingKeys and values rebuilt for every stepUse absorbed matrices in the kernel
Positional errorsRoPE applied to compressed keysKeep the decoupled RoPE path
Compute-bound attentionAbsorbed queries are widerTune batch size and kernel tiling
Retrofit quality lossConverting a trained GQA modelFine-tune after conversion
Common MLA pitfalls and remedies

Absorption shifts work from memory traffic to arithmetic, so MLA is most beneficial when decoding is memory-bound, as in long-context or high-concurrency serving. Research methods for converting existing models to MLA exist, but retrofitted models usually need additional training to recover quality.

The short version

When a chatbot writes a long answer, it keeps notes about every earlier word so it does not have to reread everything. Those notes take up considerable memory. Multi-head latent attention stores the notes in a compressed shorthand and expands them only when needed.

The result is that the same hardware can handle longer conversations or more users at once. Quality stays close to the full-size method, which is why several recent language models use it.

Where it makes a difference

Consider a company serving a large language model to internal users with documents of up to 100,000 tokens. With standard multi-head attention in a model of DeepSeek-V2 dimensions, the KV cache for a single long request would require hundreds of gigabytes in 16-bit precision, far more than one server can hold alongside the weights.

With MLA, each token stores 576 elements per layer instead of 32,768, so the same request needs less than one fiftieth of that memory. The operations team can keep many long-context sessions on one multi-GPU node, batch more requests together and raise throughput substantially, while users notice no meaningful drop in answer quality.