A transformer is a neural network architecture built around self-attention, a mechanism that lets every element of a sequence directly weigh every other element when computing its representation. Among neural networks, the transformer replaced recurrence and convolution as the dominant design for language and is now the foundation of large language models, many vision models and multimodal systems.
Introduced in 2017 for machine translation, the transformer processes all positions of a sequence in parallel rather than step by step. This makes training on modern accelerators highly efficient and allows models to capture long-range dependencies that earlier recurrent networks struggled to learn.
How a transformer processes a sequence
A transformer processes a sequence by converting it into tokens, mapping each token to an embedding vector, adding information about position and then passing the vectors through a stack of identical transformer blocks. Each block refines every token’s representation using context from the rest of the sequence.
Inside each block, multi-head self-attention computes, for every token, a query, a key and a value vector. Attention weights come from the scaled dot product of queries and keys, normalized with a softmax, and each token’s output is the weighted sum of value vectors. Multiple heads run in parallel, so different heads can track different relationships, such as syntax or reference.
A position-wise feed-forward network follows the attention layer, usually expanding the dimension about four times before projecting back. Residual connections and layer normalization around both sublayers keep gradients stable, which is what allows stacks of dozens or even more than a hundred layers to train through backpropagation.
| Component | Function | Typical setting |
|---|---|---|
| Token embedding | Maps tokens to vectors | Model width of 512 to over 10,000 dimensions |
| Positional encoding | Injects order information | Sinusoidal, learned or rotary |
| Multi-head attention | Mixes information across tokens | 8 to over 100 heads |
| Feed-forward network | Transforms each token independently | Hidden size about 4 times the width |
| Residual connections | Preserve signal and gradients | Around every sublayer |
| Layer normalization | Stabilizes activations | Pre-norm in most modern models |
Transformer architecture variants
The original transformer had an encoder and a decoder. Later work showed that either half alone is effective for different tasks, which produced three main families.
| Variant | Attention pattern | Training objective | Typical use |
|---|---|---|---|
| Encoder-only | Bidirectional | Masked token prediction | Classification, search embeddings |
| Decoder-only | Causal, left to right | Next-token prediction | Text generation, chat, code |
| Encoder-decoder | Bidirectional plus cross-attention | Sequence-to-sequence | Translation, summarization |
| Vision transformer | Bidirectional over image patches | Classification or self-supervised | Image recognition |
Most large language models are decoder-only transformers. A causal mask prevents each position from attending to later positions, so the model can be trained to predict the next token across billions of text sequences and then generate text one token at a time.
Transformer compared with earlier neural network architectures
The transformer differs from recurrent and convolutional networks mainly in how information travels between positions. In a recurrent network, information passes through every intermediate step; in a transformer, any two tokens are connected in a single attention operation.
| Architecture | Path between distant tokens | Training parallelism | Main limitation |
|---|---|---|---|
| Recurrent network (LSTM) | Grows with distance | Low, sequential steps | Slow training, fading memory |
| Convolutional network | Grows with distance over kernel size | High | Limited receptive field per layer |
| Transformer | One step | High | Quadratic cost in sequence length |
| State space models | Recurrent at inference | High in training | Weaker at precise recall |
The attention mechanism itself predates the transformer, having been used with recurrent translation models. The transformer’s contribution was showing that attention alone, without recurrence, was sufficient and scaled better.
Scaling, cost and efficiency
Transformer performance improves predictably with more parameters, more data and more compute, a relationship described by empirical scaling laws. Model sizes range from tens of millions of parameters for compact encoders to hundreds of billions for frontier language models.
The main cost is that self-attention compares every token with every other, so compute and memory grow quadratically with sequence length. Doubling the context roughly quadruples attention cost, which is why long context windows require specific engineering.
| Technique | Problem addressed | Effect |
|---|---|---|
| Key-value caching | Repeated computation during generation | Faster token-by-token decoding |
| Memory-efficient attention kernels | Memory traffic of attention | Longer sequences on the same hardware |
| Sparse or sliding-window attention | Quadratic cost | Near-linear cost, some loss of global view |
| Grouped-query attention | Large key-value cache | Lower memory per generated token |
| Mixture of experts | Compute per token | Large capacity with fewer active parameters |
| Quantization | Model size and bandwidth | 8-bit or 4-bit weights, small accuracy cost |
Limitations and common misconceptions
A transformer has no built-in sense of order, so it depends entirely on positional information, and models often degrade on sequences longer than those seen in training unless the position scheme is designed to extrapolate.
- Attention weights are not reliable explanations; high weight does not prove a token caused the output.
- A larger context window does not guarantee equal use of all positions; information in the middle can be underused.
- Transformers learn statistical patterns and can produce fluent but incorrect output.
- Training large transformers requires massive datasets and accelerator clusters, which limits who can build them from scratch.
A frequent misconception is that the transformer is a single model. It is an architecture, a reusable blueprint; the behavior of any specific system depends on its training data, objective, size and fine-tuning. Two transformers with identical layer counts can differ enormously in capability, so architecture alone says little about quality, and evaluation on the intended task remains essential.
In plain words
When people read a sentence, they understand each word by looking at the other words around it. In the sentence the trophy did not fit in the suitcase because it was too big, readers link it to the trophy without effort.
A transformer is a computer design that does exactly this, for every word at once. Each word asks which other words matter to it and gathers meaning from them. Doing all of this in parallel is what made today’s large language tools possible.
A real-world example
A customer support team in an insurance company receives about 10,000 emails per week in several languages and wants to route each one to the right department. Instead of training a model from scratch, engineers fine-tune a pretrained encoder-only transformer with roughly 100 million parameters on 20,000 historical emails labeled with their correct queue.
Inputs are truncated to 512 tokens, and fine-tuning takes a few hours on a single graphics card. The model reads the full message with bidirectional attention, so phrases such as not the car but the house claim are interpreted correctly. In evaluation on held-out emails, routing accuracy clearly exceeds the previous keyword rules, and manual re-routing drops substantially in the first month after launch.