Self-attention

On this page 7

Self-attention is an operation in neural networks in which every element of a sequence computes a weighted combination of all elements of the same sequence, with the weights learned from the content itself. It lets a model relate each word, image patch or audio frame directly to every other one, regardless of distance.

Self-attention is the core building block of the Transformer architecture and therefore of most modern language, vision and multimodal models in artificial intelligence. Its strengths are direct long-range dependencies and full parallelism during training; its main cost is computation and memory that grow quadratically with sequence length.

How self-attention computes its output

Each input vector is projected by three learned matrices into a query, a key and a value. The query of one position is compared with the keys of all positions by a dot product, which measures how relevant each position is to the current one.

The scores are divided by the square root of the key dimension, which keeps their variance stable, and passed through a softmax to obtain attention weights that sum to one. The output for each position is the weighted sum of all value vectors. In compact form, the operation is softmax(QK^T / sqrt(d_k)) V, known as scaled dot-product attention.

The table below lists the components and their role in a typical model.

ComponentShape for n tokensRole
Query matrix Qn x d_kWhat each token is looking for
Key matrix Kn x d_kWhat each token offers for matching
Value matrix Vn x d_vContent passed on when matched
Score matrixn x nPairwise relevance before softmax
Attention weightsn x nNormalized mixing coefficients
Outputn x d_vContext-aware token representations
Components of scaled dot-product self-attention

Multiple heads, masking and position information

A single attention pattern can capture only one kind of relationship at a time, so Transformers run several attention operations in parallel. In multi-head attention, each head uses its own projections, for example 12 to 128 heads in large models, and the results are concatenated and projected back to the model width.

Decoder models that generate text apply a causal mask, which sets scores for future positions to negative infinity so that each token can attend only to itself and earlier tokens. Encoder models such as those used for classification use bidirectional self-attention without this mask.

Self-attention is permutation-equivariant: shuffling the inputs simply shuffles the outputs. Order must therefore be injected separately through positional encoding, such as sinusoidal vectors, learned position embeddings or rotary position embedding, which rotates queries and keys according to their positions.

Self-attention compared with other sequence operations

Self-attention differs from recurrence and convolution in how far information travels in one layer and how much work that costs. The comparison below uses n for sequence length and d for model width.

OperationCompute per layerPath between distant tokensParallel training
Self-attentionO(n^2 d)1 stepYes
Recurrent layerO(n d^2)n stepsNo
Convolution, kernel kO(k n d^2)About n / k stepsYes
State space modelO(n d)1 step via recurrence stateYes, with scan
Per-layer properties of common sequence operations

Cross-attention, which is closely related, takes queries from one sequence and keys and values from another, as when a decoder attends to encoder outputs. Self-attention is the special case in which all three come from the same sequence. The broader attention mechanism also predates Transformers, where it was first used on top of recurrent networks for translation.

Efficiency variants for long sequences

Because the n x n score matrix dominates cost, doubling the context length roughly quadruples attention memory and compute in a naive implementation. A large family of variants reduces this burden, each with a different trade-off between quality and speed.

ApproachKey ideaTypical trade-off
FlashAttentionTiled exact computation in fast memoryExact results, hardware-specific kernels
Sliding window attentionEach token sees a local windowLong-range links need many layers
Sparse attentionFixed or learned subsets of pairsPatterns may miss relevant tokens
Linear attentionKernel trick avoids the n x n matrixOften lower quality at scale
Grouped-query attentionHeads share key and value projectionsSmaller cache, slight quality loss
Multi-head latent attentionCompresses keys and values into a latentAdded projection complexity
Selected approaches to cheaper self-attention

During text generation, models store past keys and values in a KV cache so that each new token needs only one new row of attention. The cache size, not the arithmetic, often becomes the main memory bottleneck at inference, which is why grouped-query and latent variants are widely adopted.

Limitations and common misconceptions

Self-attention is powerful but not without failure modes, and several popular interpretations overstate what it reveals.

  • Attention weights are not explanations: high weight on a token does not prove that the token caused the prediction.
  • Long context is not uniform recall: models often use information at the start and end of a long input better than material in the middle.
  • Attention sinks form when many heads place large weight on the first token as a default resting point.
  • Numerical instability can arise in low precision if scores grow large, which normalization and scaling help control.

Self-attention also has no built-in notion of locality or order, so it typically needs more data than convolution to learn spatial structure in images. This weak inductive bias is a strength at large scale and a weakness on small datasets.

In plain words

Self-attention lets a model read like a careful reader. Every word looks at every other word to decide what it means here. Take the sentence “The animal did not cross the street because it was tired.” The word “it” checks the other words. It concludes that “it” most likely means the animal, not the street.

Each word asks a question, every word offers an answer, and the best matches contribute the most. The catch is that longer texts mean many more comparisons, which is why very long documents are expensive for these models.

A real-world example

A legal technology team fine-tunes a Transformer-based model to find clauses in commercial contracts. A typical contract runs to about 12,000 tokens, and a definition on page 2 often controls the meaning of an obligation on page 30. Self-attention allows a token in the obligation to attend directly to the distant definition in a single layer, which a recurrent model would struggle to connect.

The first prototype runs out of GPU memory because naive attention over 12,000 tokens needs a score matrix of about 144 million entries per head. The engineers switch to FlashAttention with grouped-query attention, which keeps the computation exact while reducing memory use enough to process each contract in one pass. Clause extraction accuracy improves noticeably compared with the earlier approach of splitting contracts into short chunks, because cross-references are no longer cut apart.