Multi-head attention is a neural network layer that runs several scaled dot-product attention operations in parallel, each on its own learned projection of the input, and then concatenates and recombines their outputs. It is the central building block of the transformer and one of the most important mechanisms in modern neural networks for language, vision, audio and multimodal data.
Splitting attention into heads lets a model attend to different relationships at the same time, for example syntactic agreement in one head and long-range reference in another, at roughly the same compute cost as a single attention operation of full width.
How multi-head attention computes its output
Each head receives the same input sequence but applies its own query, key and value projection matrices. The head then computes attention as softmax of the query-key dot products, divided by the square root of the head dimension, and uses the resulting weights to mix the value vectors.
- Project the input of width d_model into h sets of queries, keys and values, each of width d_k, usually d_model divided by h.
- Score every query against every key and scale by the square root of d_k to keep the softmax well conditioned.
- Normalize the scores with softmax, optionally after applying a causal or padding mask.
- Mix the value vectors with these weights to produce one output per position per head.
- Concatenate the h head outputs and apply a final output projection back to d_model.
Because all heads are computed as batched matrix multiplications, they run in parallel on GPUs and accelerators. The total parameter count of the projections is about 4 times d_model squared, independent of the number of heads, so adding heads changes how capacity is divided rather than how much capacity exists.
Key parameters and their effects
A small set of hyperparameters defines each attention layer, and their values interact with memory use, speed and model quality. The table lists the main ones with typical ranges seen in published architectures.
| Parameter | Typical values | What it affects |
|---|---|---|
| Number of heads (h) | 8 to 128 | Diversity of attention patterns |
| Head dimension (d_k) | 64 or 128 | Expressiveness of each head and kernel efficiency |
| Model width (d_model) | 512 to over 16,000 | Total capacity and projection cost |
| Number of key-value heads | 1 up to h | Size of the key-value cache at inference |
| Context length (n) | Hundreds to over a million tokens | Quadratic compute and memory of the score matrix |
| Attention dropout | 0 to 0.1 | Regularization of attention weights |
The quadratic cost in sequence length dominates for long contexts: a sequence of n tokens produces an n by n score matrix per head. Optimized kernels such as FlashAttention avoid materializing that matrix in slow memory, which cuts memory traffic without changing the mathematical result.
What different heads learn
Interpretability studies show that heads often specialize in recognizable functions, although many heads are diffuse or redundant. Specialization emerges from training rather than from any explicit assignment.
- Positional heads attend mainly to the previous or next token.
- Syntactic heads link verbs with subjects or determiners with nouns.
- Induction heads find an earlier occurrence of the current token and copy what followed it, a mechanism tied to in-context learning.
- Attention sinks place most weight on the first token, giving the softmax a place to park probability mass when nothing is relevant.
Head-pruning experiments have found that a substantial fraction of heads can be removed from trained models with only a small loss in accuracy. This redundancy is one motivation for the more economical variants described below, and it is a warning against reading too much into any single head.
Multi-head attention compared with its variants
Large language models frequently replace standard multi-head attention with variants that shrink the key-value cache, which is the main memory bottleneck during autoregressive generation. The comparison shows the main design choices.
| Variant | Key-value heads | Main benefit | Main cost |
|---|---|---|---|
| Multi-head attention | One per query head | Highest flexibility and quality baseline | Largest key-value cache |
| Multi-query attention | One shared | Smallest cache, fastest decoding | Some quality loss, less stable training |
| Grouped-query attention | A few shared groups | Near baseline quality with a smaller cache | Extra design choice for group count |
| Multi-head latent attention | Compressed latent vector | Very small cache with strong quality | More complex implementation |
| Sliding window attention | Any count, local span only | Linear cost in sequence length | No direct long-range access per layer |
Multi-head attention is also distinct from self-attention and cross-attention, which describe where queries and keys come from rather than how many heads exist. In self-attention both come from the same sequence; in cross-attention, as in an encoder-decoder architecture, queries come from the decoder and keys and values from the encoder output.
Practical design decisions
Choosing an attention configuration is a balance between quality, memory and latency, and the right choice depends on whether the model is trained once and served many times. The decision table summarizes common situations.
| Situation | Recommended choice | Reason |
|---|---|---|
| Small research model | Standard multi-head attention | Simplest and well understood |
| Large model served at scale | Grouped-query attention | Cuts cache memory and raises throughput |
| Extremely long contexts | Sliding window or sparse attention layers | Avoids quadratic growth |
| Memory-limited edge deployment | Multi-query attention with quantized cache | Minimal memory per token |
| Encoder-only classification | Standard multi-head attention | No autoregressive cache to optimize |
Head dimension is usually kept at 64 or 128 because hardware kernels are tuned for these sizes. Positional information is supplied separately, most often through rotary position embedding applied to queries and keys inside each head, because attention itself is permutation-invariant.
Limitations and common mistakes
Multi-head attention is powerful but has well-known failure modes that appear in both training and deployment.
| Symptom | Likely cause | Typical fix |
|---|---|---|
| Training loss spikes or diverges | Very large attention logits | QK normalization, lower learning rate |
| Out-of-memory errors on long inputs | Quadratic score matrix | Fused attention kernels, shorter windows |
| Slow token generation | Large key-value cache reads | Grouped-query attention, cache quantization |
| Weak use of distant context | Attention diluted across many tokens | Better positional scaling, retrieval support |
A frequent misconception is that more heads always mean better models. With fixed width, more heads make each head narrower, and beyond a point the per-head dimension becomes too small to represent useful relationships.
The idea in brief
Imagine a team reading the same sentence, where each member looks for something different. One tracks who did what, another tracks when it happened, and a third notices which words refer back to earlier ones. At the end they combine their notes into one richer understanding.
Multi-head attention does this inside an AI model. Each head looks at the same words but focuses on a different kind of connection. Several narrow viewpoints combined give a fuller picture than one broad view would.
A practical scenario
A machine learning team serves a 7-billion-parameter chat model to customer support agents. The original design uses standard multi-head attention with 32 query heads and 32 key-value heads, and at a 32,000-token context the key-value cache consumes most of the GPU memory, limiting each card to a handful of concurrent conversations.
The team retrains the model with grouped-query attention using 8 key-value heads, keeping 32 query heads. Cache size per token falls to one quarter, each GPU serves roughly four times as many simultaneous sessions, and evaluation on the team’s support benchmark shows quality within normal run-to-run variation of the original model.