Multi-head attention

On this page 8

Multi-head attention is a neural network layer that runs several scaled dot-product attention operations in parallel, each on its own learned projection of the input, and then concatenates and recombines their outputs. It is the central building block of the transformer and one of the most important mechanisms in modern neural networks for language, vision, audio and multimodal data.

Splitting attention into heads lets a model attend to different relationships at the same time, for example syntactic agreement in one head and long-range reference in another, at roughly the same compute cost as a single attention operation of full width.

How multi-head attention computes its output

Each head receives the same input sequence but applies its own query, key and value projection matrices. The head then computes attention as softmax of the query-key dot products, divided by the square root of the head dimension, and uses the resulting weights to mix the value vectors.

  1. Project the input of width d_model into h sets of queries, keys and values, each of width d_k, usually d_model divided by h.
  2. Score every query against every key and scale by the square root of d_k to keep the softmax well conditioned.
  3. Normalize the scores with softmax, optionally after applying a causal or padding mask.
  4. Mix the value vectors with these weights to produce one output per position per head.
  5. Concatenate the h head outputs and apply a final output projection back to d_model.

Because all heads are computed as batched matrix multiplications, they run in parallel on GPUs and accelerators. The total parameter count of the projections is about 4 times d_model squared, independent of the number of heads, so adding heads changes how capacity is divided rather than how much capacity exists.

Key parameters and their effects

A small set of hyperparameters defines each attention layer, and their values interact with memory use, speed and model quality. The table lists the main ones with typical ranges seen in published architectures.

ParameterTypical valuesWhat it affects
Number of heads (h)8 to 128Diversity of attention patterns
Head dimension (d_k)64 or 128Expressiveness of each head and kernel efficiency
Model width (d_model)512 to over 16,000Total capacity and projection cost
Number of key-value heads1 up to hSize of the key-value cache at inference
Context length (n)Hundreds to over a million tokensQuadratic compute and memory of the score matrix
Attention dropout0 to 0.1Regularization of attention weights
Main parameters of a multi-head attention layer

The quadratic cost in sequence length dominates for long contexts: a sequence of n tokens produces an n by n score matrix per head. Optimized kernels such as FlashAttention avoid materializing that matrix in slow memory, which cuts memory traffic without changing the mathematical result.

What different heads learn

Interpretability studies show that heads often specialize in recognizable functions, although many heads are diffuse or redundant. Specialization emerges from training rather than from any explicit assignment.

  • Positional heads attend mainly to the previous or next token.
  • Syntactic heads link verbs with subjects or determiners with nouns.
  • Induction heads find an earlier occurrence of the current token and copy what followed it, a mechanism tied to in-context learning.
  • Attention sinks place most weight on the first token, giving the softmax a place to park probability mass when nothing is relevant.

Head-pruning experiments have found that a substantial fraction of heads can be removed from trained models with only a small loss in accuracy. This redundancy is one motivation for the more economical variants described below, and it is a warning against reading too much into any single head.

Multi-head attention compared with its variants

Large language models frequently replace standard multi-head attention with variants that shrink the key-value cache, which is the main memory bottleneck during autoregressive generation. The comparison shows the main design choices.

VariantKey-value headsMain benefitMain cost
Multi-head attentionOne per query headHighest flexibility and quality baselineLargest key-value cache
Multi-query attentionOne sharedSmallest cache, fastest decodingSome quality loss, less stable training
Grouped-query attentionA few shared groupsNear baseline quality with a smaller cacheExtra design choice for group count
Multi-head latent attentionCompressed latent vectorVery small cache with strong qualityMore complex implementation
Sliding window attentionAny count, local span onlyLinear cost in sequence lengthNo direct long-range access per layer
Standard multi-head attention and common variants

Multi-head attention is also distinct from self-attention and cross-attention, which describe where queries and keys come from rather than how many heads exist. In self-attention both come from the same sequence; in cross-attention, as in an encoder-decoder architecture, queries come from the decoder and keys and values from the encoder output.

Practical design decisions

Choosing an attention configuration is a balance between quality, memory and latency, and the right choice depends on whether the model is trained once and served many times. The decision table summarizes common situations.

SituationRecommended choiceReason
Small research modelStandard multi-head attentionSimplest and well understood
Large model served at scaleGrouped-query attentionCuts cache memory and raises throughput
Extremely long contextsSliding window or sparse attention layersAvoids quadratic growth
Memory-limited edge deploymentMulti-query attention with quantized cacheMinimal memory per token
Encoder-only classificationStandard multi-head attentionNo autoregressive cache to optimize
Choosing an attention configuration

Head dimension is usually kept at 64 or 128 because hardware kernels are tuned for these sizes. Positional information is supplied separately, most often through rotary position embedding applied to queries and keys inside each head, because attention itself is permutation-invariant.

Limitations and common mistakes

Multi-head attention is powerful but has well-known failure modes that appear in both training and deployment.

SymptomLikely causeTypical fix
Training loss spikes or divergesVery large attention logitsQK normalization, lower learning rate
Out-of-memory errors on long inputsQuadratic score matrixFused attention kernels, shorter windows
Slow token generationLarge key-value cache readsGrouped-query attention, cache quantization
Weak use of distant contextAttention diluted across many tokensBetter positional scaling, retrieval support
Typical problems with attention layers

A frequent misconception is that more heads always mean better models. With fixed width, more heads make each head narrower, and beyond a point the per-head dimension becomes too small to represent useful relationships.

The idea in brief

Imagine a team reading the same sentence, where each member looks for something different. One tracks who did what, another tracks when it happened, and a third notices which words refer back to earlier ones. At the end they combine their notes into one richer understanding.

Multi-head attention does this inside an AI model. Each head looks at the same words but focuses on a different kind of connection. Several narrow viewpoints combined give a fuller picture than one broad view would.

A practical scenario

A machine learning team serves a 7-billion-parameter chat model to customer support agents. The original design uses standard multi-head attention with 32 query heads and 32 key-value heads, and at a 32,000-token context the key-value cache consumes most of the GPU memory, limiting each card to a handful of concurrent conversations.

The team retrains the model with grouped-query attention using 8 key-value heads, keeping 32 query heads. Cache size per token falls to one quarter, each GPU serves roughly four times as many simultaneous sessions, and evaluation on the team’s support benchmark shows quality within normal run-to-run variation of the original model.