Grouped-query attention

Also known as: GQA

On this page 7

Grouped-query attention (GQA) is a variant of multi-head attention in which the query heads are divided into groups and each group shares a single key head and value head. In transformer neural networks, it sits between standard multi-head attention, where every query head has its own keys and values, and multi-query attention, where all query heads share one pair.

The design exists mainly to reduce the key-value cache that a language model must keep in memory during autoregressive generation. By cutting the number of key and value heads, GQA lowers memory bandwidth per generated token and allows longer contexts or larger batches, while losing far less quality than the more aggressive multi-query approach.

How grouped-query attention computes attention

Grouped-query attention keeps the full number of query heads but projects keys and values into a smaller number of heads, written as G. If a layer has H query heads, each key-value head serves H divided by G query heads, and those query heads compute their attention scores against the same keys and read the same values.

Mathematically, each query head still performs scaled dot-product attention with a softmax over the sequence. The only change is that the key and value projection matrices are smaller, and at run time the shared key-value tensors are broadcast or repeated across the heads of their group. The output projection is unchanged, so the layer has the same interface as ordinary multi-head attention.

The configuration space forms a continuum, shown in the table below for a layer with 32 query heads.

VariantQuery headsKey-value headsQueries per key-value head
Multi-head attention (MHA)32321
GQA with 8 groups3284
GQA with 4 groups3248
Multi-query attention (MQA)32132
Attention variants as settings of the key-value head count

Why the key-value cache drives the design

During generation, a decoder-only model stores the keys and values of every previous token so that it does not recompute them, a structure called the KV cache. Its size grows with layers, key-value heads, head dimension, sequence length and batch size, and for long contexts it can exceed the size of the model weights.

Decoding one token is usually memory-bandwidth bound rather than compute bound: the accelerator spends most of its time reading weights and cache from memory. Shrinking the number of key-value heads by a factor of 4 or 8 shrinks the cache, and the bytes read per step, by the same factor, which directly raises tokens per second and the number of concurrent requests a server can hold.

The following example assumes 32 layers, a head dimension of 128, 16-bit values and a single 32,000-token sequence.

ConfigurationKey-value headsApproximate cache sizeRelative size
MHA32About 16.8 GB100 percent
GQA, 8 groups8About 4.2 GB25 percent
GQA, 4 groups4About 2.1 GB12.5 percent
MQA1About 0.5 GB3 percent
Approximate KV cache size for one long sequence

These figures follow from 2 × layers × heads × head dimension × tokens × bytes. Real deployments also use cache quantization and paged memory, but the proportional saving from GQA remains the same.

Quality trade-offs and uptraining

Grouped-query attention trades a small loss of representational capacity for large inference savings. Sharing keys and values means query heads in the same group cannot attend using independent key spaces, but empirical results show that moderate grouping such as 8 key-value heads keeps quality close to full multi-head attention, while multi-query attention more often shows measurable degradation and less stable training.

GQA was introduced with a method called uptraining: an existing multi-head checkpoint is converted by mean-pooling the key and value projection weights within each group, then trained further for a small fraction of the original compute, reported at roughly 5 percent. Today most large models are instead trained with GQA from the start, which avoids conversion altogether.

PropertyMHAGQAMQA
KV cache sizeLargestReduced by the group factorSmallest
Model qualityReference levelClose to referenceOften slightly lower
Decoding speedSlowest at long contextSubstantially fasterFastest
Tensor parallelism fitSimpleSimple if heads divide evenlyKey-value head must be replicated
Trade-offs across attention variants

Grouped-query attention compared with other efficient attention methods

GQA reduces the width of the cache, not the length of the sequence each token attends to. It is therefore complementary to methods such as sliding window attention and sparse attention, which limit which positions are attended, and to linear attention, which changes the attention computation itself.

MethodWhat it reducesCombines with GQA
Sliding window attentionNumber of past tokens attendedYes, commonly
Multi-head latent attentionCache via a compressed latent vectorAn alternative to GQA
KV cache quantizationBytes per cached valueYes
FlashAttentionMemory traffic inside the attention kernelYes, kernels support grouped heads
What each efficiency method changes

Multi-head latent attention is the closest competitor: instead of sharing heads, it compresses keys and values into a low-rank latent vector and reconstructs per-head projections, which can reach smaller caches than GQA at comparable quality but adds implementation complexity.

Choosing the number of groups

The group count is a hardware and workload decision as much as a modeling one. Practitioners generally choose a key-value head count that divides evenly across the planned tensor-parallel degree, so that each accelerator holds whole key-value heads instead of duplicates.

  • 8 key-value heads is a widespread default for models from several billion to hundreds of billions of parameters.
  • Fewer key-value heads suit long-context or high-concurrency serving, where cache memory is the bottleneck.
  • More key-value heads suit small models, where the cache is modest and quality per parameter matters more.
  • A common mistake is choosing a count smaller than the tensor-parallel degree, which forces replication and erases part of the memory saving.

Explained without jargon

A language model writes text one word at a time, and for each new word it looks back at everything written so far. To avoid rereading from scratch, it keeps a running set of notes about every earlier word. Those notes take up considerable memory.

Grouped-query attention makes the notes smaller. Instead of every reader in the model keeping its own copy, small teams of readers share one copy. Each reader still asks its own questions, but they look things up in the same shared notebook. The model answers almost as well and needs much less memory, so it can respond faster and serve more people at once.

A typical use case

An engineering team serves a 70-billion-parameter chat model on a node with eight GPUs, each with 80 GB of memory. With full multi-head attention, the KV cache for long conversations fills the remaining memory after only a handful of concurrent sessions, and response latency climbs whenever traffic peaks.

The team switches to a model variant trained with 8 key-value heads for 64 query heads, one key-value head per GPU under eight-way tensor parallelism. The cache per session shrinks to one eighth of its former size. On the same hardware, the server now holds several times more concurrent sessions at a 16,000-token context, and per-token latency falls because each decoding step reads far less memory.

Evaluation on the team’s internal question set shows no meaningful drop in answer quality. The outcome is lower serving cost per conversation without a change in hardware.