Positional encoding is the mechanism that gives a transformer information about the order of tokens in a sequence, either by adding position-dependent vectors to the token embeddings or by modifying how attention compares positions. In neural networks built on self-attention, it is essential because attention by itself is permutation-invariant: without positional information, the sentences “dog bites man” and “man bites dog” would produce the same set of internal representations.
The term covers both fixed schemes, such as the sinusoidal functions of the original transformer, and learned schemes, often called a position embedding. Modern large language models mostly use relative methods that encode the distance between tokens rather than their absolute index.
Why attention needs positional information
Self-attention computes a weighted sum over all tokens, with weights derived from dot products between queries and keys. Because the same computation is applied to every pair regardless of where the tokens sit, shuffling the input shuffles the output in exactly the same way. The layer is therefore blind to word order unless order is injected explicitly.
Recurrent neural networks and convolutional networks do not have this problem, since they process tokens sequentially or through local windows and so encode order in their structure. The transformer gave up that structure to gain parallel computation across the whole sequence, and positional encoding is the price of that design choice.
The table below summarizes how common sequence architectures obtain order information.
| Architecture | How order is represented | Needs explicit encoding |
|---|---|---|
| Recurrent network | Hidden state updated step by step | No |
| Convolutional network | Fixed local kernel offsets | Partly, for global position |
| Transformer | None in attention itself | Yes |
| State space model | Recurrent state over time | No |
How sinusoidal positional encoding works
The original transformer of 2017 used fixed sinusoidal encodings added to the token embeddings before the first layer. For position pos and dimension index i, even dimensions use sin(pos / 10000^(2i/d)) and odd dimensions use the matching cosine, where d is the model dimension.
Each pair of dimensions oscillates at a different frequency, from a wavelength of about 2π up to roughly 10,000 times 2π. Low dimensions change quickly and distinguish neighboring tokens; high dimensions change slowly and separate distant regions. Because a shift in position corresponds to a linear rotation of each sine-cosine pair, the model can in principle learn to attend by relative offset.
Sinusoidal encodings need no trained parameters and produce a value for any position, but in practice models trained on short sequences still degrade on longer inputs, because the attention weights never saw those patterns during training.
Main types of positional encoding
Positional encoding methods differ in where position enters the computation and whether they represent absolute index or relative distance. The table compares the families that appear most often in practice.
| Method | Where it is applied | Absolute or relative | Typical use |
|---|---|---|---|
| Sinusoidal | Added to input embeddings | Absolute | Original transformer, many encoders |
| Learned absolute | Trainable vector per index | Absolute | BERT and early GPT models |
| Relative bias | Learned bias on attention scores | Relative | T5-style models |
| ALiBi | Linear distance penalty on scores | Relative | Models aiming at length extrapolation |
| Rotary position embedding | Rotation of queries and keys | Relative | Most current decoder language models |
Rotary position embedding, usually abbreviated RoPE, rotates query and key vectors by angles proportional to position, so their dot product depends only on the relative distance between tokens. It has become the default in open decoder models and has its own detailed treatment as a separate concept.
Key design parameters and their effects
Several parameters determine how well a positional scheme serves a given model. The most consequential is the maximum trained context length, because behavior beyond it is rarely reliable without adaptation.
| Parameter | Typical values | What it affects |
|---|---|---|
| Trained context length | 512 to over 100,000 tokens | Range of reliable positions |
| Frequency base | 10,000 by default, larger for long context | Resolution of near versus far positions |
| Embedding dimension | 512 to 8,192 or more | Number of distinct frequencies available |
| Scaling or interpolation factor | 1 to 32 in extension methods | How positions are compressed into the trained range |
Context extension techniques such as position interpolation and frequency-base adjustment rescale positions so that a longer sequence maps into the range seen during training. They usually require a short phase of fine-tuning on long documents to recover accuracy.
Limitations and common mistakes
The most frequent problem is poor length extrapolation: a model that performs well at its trained length can produce incoherent output when the input grows longer. A second issue is that position information can fade in deep networks, since absolute encodings are added only once at the input.
| Symptom | Likely cause | Remedy |
|---|---|---|
| Quality collapse past a length | Positions beyond trained range | Interpolation plus long-context fine-tuning |
| Weak use of distant context | Attention biased toward recent tokens | Adjust frequency base or train on long data |
| Crash on long input | Learned table has no entry for the index | Switch to a relative or extendable method |
| Mismatch after loading weights | Different encoding at inference than in training | Use the exact original configuration |
A common misconception is that positional encoding stores meaning about the token; it carries only order or distance. Another is that relative methods remove all length limits. They extend gracefully further, but attention patterns still reflect the lengths seen in training.
The idea in brief
A transformer reads all the words of a sentence at once, like seeing a pile of word cards dropped on a table. Without extra help it cannot tell which card came first, so “dog bites man” and “man bites dog” look the same.
Positional encoding is like numbering the cards before they are dropped. Each word gets a small signal that says where it sits, or how far it is from its neighbors. With that signal, the model can follow word order, grammar and who did what to whom.
A practical scenario
A legal technology team fine-tunes an open language model with rotary position embedding and a trained context of 8,192 tokens to summarize contracts. Many of their contracts run to about 25,000 tokens, and summaries of those documents lose track of clauses near the end and sometimes repeat sections.
The engineers apply position interpolation with a scaling factor of 4, which compresses 32,768 positions into the range the model already knows, and then fine-tune for a short run on long contracts. They verify the change with retrieval tests that place a key clause at different depths in the document.
After the adjustment, the model correctly retrieves late clauses in long contracts, while quality on short documents stays essentially unchanged. The positional scheme, not the model size, was the limiting factor.