An attention mechanism is a component of neural networks that computes, for each element being processed, a weighted combination of other elements, where the weights express how relevant each element is in the current context. Instead of compressing an entire input into one fixed vector, the network learns to look selectively at the parts that matter for each output.
Attention was introduced in neural machine translation around 2014 to let a decoder focus on relevant source words. In 2017 the Transformer architecture showed that attention alone, without recurrence, could model sequences. Today self-attention is the central operation in large language models, vision transformers and many speech and multimodal systems.
How scaled dot-product attention works
The dominant form is scaled dot-product attention. Each input vector is projected by learned weight matrices into three roles: a query, a key and a value. The query asks what information is needed, keys describe what each position offers, and values carry the content that is passed on.
The mechanism proceeds in four steps. It computes the dot product of each query with every key, divides the scores by the square root of the key dimension, applies a softmax so that the weights for each query are positive and sum to one, and finally returns the weighted sum of the values. In compact form, the output is softmax(QKᵀ / √d) · V.
| Component | What it is | Role in the computation |
|---|---|---|
| Query (Q) | Projection of the current position | Determines what to look for |
| Key (K) | Projection of every candidate position | Matched against queries to score relevance |
| Value (V) | Projection carrying content | Mixed according to the weights |
| Scaling factor √d | Square root of the key dimension | Keeps scores in a range where softmax trains well |
| Softmax | Normalizing function | Turns scores into weights that sum to one |
| Mask | Blocked positions set to minus infinity | Prevents attending to future or padding tokens |
Without the scaling factor, dot products grow with the vector dimension, softmax becomes extremely peaked, and gradients vanish during backpropagation. The causal mask used in text generation ensures that each position attends only to earlier positions.
Self-attention, cross-attention and multi-head attention
Attention variants differ mainly in where queries, keys and values come from. In self-attention, all three come from the same sequence, so every token can relate to every other token. In cross-attention, queries come from one sequence and keys and values from another, as when a translation decoder reads the encoded source sentence.
Multi-head attention runs several attention operations in parallel, each with its own projections and a smaller dimension, and concatenates the results. Different heads can specialize, for example in nearby syntax or long-range references, although such specialization is learned and not guaranteed.
| Variant | Source of queries | Source of keys and values | Typical use |
|---|---|---|---|
| Self-attention | Same sequence | Same sequence | Encoders and decoder-only language models |
| Causal self-attention | Same sequence | Earlier positions only | Text generation |
| Cross-attention | Target sequence | Source sequence or other modality | Translation, image captioning, text-to-image |
| Multi-head attention | Split into several heads | Split into several heads | Standard in Transformer layers |
| Grouped-query attention | Many query heads | Fewer shared key-value heads | Reducing memory during inference |
Computational cost and efficient attention
Standard self-attention compares every position with every other, so its time and memory cost grow quadratically with sequence length. Doubling the context length roughly quadruples the attention computation, which is the main reason long context windows are expensive.
Several techniques reduce this cost, either by computing exact attention more efficiently or by approximating it. The table below summarizes the main families.
| Approach | Idea | Trade-off |
|---|---|---|
| FlashAttention | Exact attention computed in GPU-friendly tiles | Same result, less memory traffic |
| Key-value caching | Stores past keys and values during generation | Faster decoding, more memory per token |
| Sliding-window attention | Each token attends to a local window | Long-range links need multiple layers |
| Sparse attention | Attends to selected position patterns | Possible loss of some dependencies |
| Linear attention | Kernel approximations of softmax | Linear cost, often lower accuracy |
In large language model inference, the key-value cache often dominates GPU memory for long prompts, which motivates grouped-query and multi-query attention that share keys and values across heads.
Attention compared with other neural network building blocks
Attention differs from earlier sequence models in how information travels between positions. The comparison shows why it displaced recurrence in most language tasks.
| Building block | Path between distant positions | Parallel training | Cost per layer |
|---|---|---|---|
| Self-attention | One step | Fully parallel | Quadratic in length |
| Recurrent layer (RNN, LSTM) | Proportional to distance | Sequential | Linear in length |
| Convolution | Grows with depth and kernel size | Fully parallel | Linear in length |
| State space models | Through a compressed state | Parallel with special algorithms | Linear in length |
Attention itself has no notion of order, because a weighted sum ignores positions. Transformers therefore add positional encodings, such as sinusoidal encodings or rotary position embeddings, to tell the model where each token sits.
Limitations and common misconceptions
Attention is powerful but often misunderstood. Practitioners should keep the following points in mind.
- Attention weights are not explanations. High weights show where information was mixed, but research shows they do not reliably reveal why a model produced an output.
- Long contexts are not used evenly. Models can retrieve information less reliably from the middle of very long inputs.
- Attention can concentrate on uninformative tokens, such as the first token of a sequence, an effect sometimes called an attention sink.
- Quadratic cost limits context length unless efficient kernels or approximations are used.
Despite these limits, attention remains the most important mechanism in modern deep learning for relating elements of a set or sequence, and most current research modifies rather than abandons it.
In plain words
Attention lets a neural network decide what to focus on. When reading the sentence about a trophy that did not fit in a suitcase because it was too big, a person knows that “it” means the trophy. Attention gives a model a way to make that link.
For every word, the model asks which other words help understand it. It gives those words more weight and the rest less. The model learns these focus patterns by itself from large amounts of examples.
A real-world example
A customer support team deploys a language model to summarize long email threads before an agent replies. A thread may contain 30 messages and several thousand tokens, including quoted earlier replies, signatures and order numbers.
Inside the model, causal self-attention lets each generated summary token draw on the relevant parts of the thread, such as the original complaint, the order number and the latest promise made by a colleague. Engineers enable a key-value cache and an efficient attention kernel, which keeps response time at a few seconds even for long threads. Agents receive a short, accurate summary and spend less time scrolling through repeated quoted text.