Self-attention is an operation in neural networks in which every element of a sequence computes a weighted combination of all elements of the same sequence, with the weights learned from the content itself. It lets a model relate each word, image patch or audio frame directly to every other one, regardless of distance.
Self-attention is the core building block of the Transformer architecture and therefore of most modern language, vision and multimodal models in artificial intelligence. Its strengths are direct long-range dependencies and full parallelism during training; its main cost is computation and memory that grow quadratically with sequence length.
How self-attention computes its output
Each input vector is projected by three learned matrices into a query, a key and a value. The query of one position is compared with the keys of all positions by a dot product, which measures how relevant each position is to the current one.
The scores are divided by the square root of the key dimension, which keeps their variance stable, and passed through a softmax to obtain attention weights that sum to one. The output for each position is the weighted sum of all value vectors. In compact form, the operation is softmax(QK^T / sqrt(d_k)) V, known as scaled dot-product attention.
The table below lists the components and their role in a typical model.
| Component | Shape for n tokens | Role |
|---|---|---|
| Query matrix Q | n x d_k | What each token is looking for |
| Key matrix K | n x d_k | What each token offers for matching |
| Value matrix V | n x d_v | Content passed on when matched |
| Score matrix | n x n | Pairwise relevance before softmax |
| Attention weights | n x n | Normalized mixing coefficients |
| Output | n x d_v | Context-aware token representations |
Multiple heads, masking and position information
A single attention pattern can capture only one kind of relationship at a time, so Transformers run several attention operations in parallel. In multi-head attention, each head uses its own projections, for example 12 to 128 heads in large models, and the results are concatenated and projected back to the model width.
Decoder models that generate text apply a causal mask, which sets scores for future positions to negative infinity so that each token can attend only to itself and earlier tokens. Encoder models such as those used for classification use bidirectional self-attention without this mask.
Self-attention is permutation-equivariant: shuffling the inputs simply shuffles the outputs. Order must therefore be injected separately through positional encoding, such as sinusoidal vectors, learned position embeddings or rotary position embedding, which rotates queries and keys according to their positions.
Self-attention compared with other sequence operations
Self-attention differs from recurrence and convolution in how far information travels in one layer and how much work that costs. The comparison below uses n for sequence length and d for model width.
| Operation | Compute per layer | Path between distant tokens | Parallel training |
|---|---|---|---|
| Self-attention | O(n^2 d) | 1 step | Yes |
| Recurrent layer | O(n d^2) | n steps | No |
| Convolution, kernel k | O(k n d^2) | About n / k steps | Yes |
| State space model | O(n d) | 1 step via recurrence state | Yes, with scan |
Cross-attention, which is closely related, takes queries from one sequence and keys and values from another, as when a decoder attends to encoder outputs. Self-attention is the special case in which all three come from the same sequence. The broader attention mechanism also predates Transformers, where it was first used on top of recurrent networks for translation.
Efficiency variants for long sequences
Because the n x n score matrix dominates cost, doubling the context length roughly quadruples attention memory and compute in a naive implementation. A large family of variants reduces this burden, each with a different trade-off between quality and speed.
| Approach | Key idea | Typical trade-off |
|---|---|---|
| FlashAttention | Tiled exact computation in fast memory | Exact results, hardware-specific kernels |
| Sliding window attention | Each token sees a local window | Long-range links need many layers |
| Sparse attention | Fixed or learned subsets of pairs | Patterns may miss relevant tokens |
| Linear attention | Kernel trick avoids the n x n matrix | Often lower quality at scale |
| Grouped-query attention | Heads share key and value projections | Smaller cache, slight quality loss |
| Multi-head latent attention | Compresses keys and values into a latent | Added projection complexity |
During text generation, models store past keys and values in a KV cache so that each new token needs only one new row of attention. The cache size, not the arithmetic, often becomes the main memory bottleneck at inference, which is why grouped-query and latent variants are widely adopted.
Limitations and common misconceptions
Self-attention is powerful but not without failure modes, and several popular interpretations overstate what it reveals.
- Attention weights are not explanations: high weight on a token does not prove that the token caused the prediction.
- Long context is not uniform recall: models often use information at the start and end of a long input better than material in the middle.
- Attention sinks form when many heads place large weight on the first token as a default resting point.
- Numerical instability can arise in low precision if scores grow large, which normalization and scaling help control.
Self-attention also has no built-in notion of locality or order, so it typically needs more data than convolution to learn spatial structure in images. This weak inductive bias is a strength at large scale and a weakness on small datasets.
In plain words
Self-attention lets a model read like a careful reader. Every word looks at every other word to decide what it means here. Take the sentence “The animal did not cross the street because it was tired.” The word “it” checks the other words. It concludes that “it” most likely means the animal, not the street.
Each word asks a question, every word offers an answer, and the best matches contribute the most. The catch is that longer texts mean many more comparisons, which is why very long documents are expensive for these models.
A real-world example
A legal technology team fine-tunes a Transformer-based model to find clauses in commercial contracts. A typical contract runs to about 12,000 tokens, and a definition on page 2 often controls the meaning of an obligation on page 30. Self-attention allows a token in the obligation to attend directly to the distant definition in a single layer, which a recurrent model would struggle to connect.
The first prototype runs out of GPU memory because naive attention over 12,000 tokens needs a score matrix of about 144 million entries per head. The engineers switch to FlashAttention with grouped-query attention, which keeps the computation exact while reducing memory use enough to process each contract in one pass. Clause extraction accuracy improves noticeably compared with the earlier approach of splitting contracts into short chunks, because cross-references are no longer cut apart.