Positional encoding

Also known as: position embedding

On this page 7

Positional encoding is the mechanism that gives a transformer information about the order of tokens in a sequence, either by adding position-dependent vectors to the token embeddings or by modifying how attention compares positions. In neural networks built on self-attention, it is essential because attention by itself is permutation-invariant: without positional information, the sentences “dog bites man” and “man bites dog” would produce the same set of internal representations.

The term covers both fixed schemes, such as the sinusoidal functions of the original transformer, and learned schemes, often called a position embedding. Modern large language models mostly use relative methods that encode the distance between tokens rather than their absolute index.

Why attention needs positional information

Self-attention computes a weighted sum over all tokens, with weights derived from dot products between queries and keys. Because the same computation is applied to every pair regardless of where the tokens sit, shuffling the input shuffles the output in exactly the same way. The layer is therefore blind to word order unless order is injected explicitly.

Recurrent neural networks and convolutional networks do not have this problem, since they process tokens sequentially or through local windows and so encode order in their structure. The transformer gave up that structure to gain parallel computation across the whole sequence, and positional encoding is the price of that design choice.

The table below summarizes how common sequence architectures obtain order information.

ArchitectureHow order is representedNeeds explicit encoding
Recurrent networkHidden state updated step by stepNo
Convolutional networkFixed local kernel offsetsPartly, for global position
TransformerNone in attention itselfYes
State space modelRecurrent state over timeNo
Sources of order information by architecture

How sinusoidal positional encoding works

The original transformer of 2017 used fixed sinusoidal encodings added to the token embeddings before the first layer. For position pos and dimension index i, even dimensions use sin(pos / 10000^(2i/d)) and odd dimensions use the matching cosine, where d is the model dimension.

Each pair of dimensions oscillates at a different frequency, from a wavelength of about 2π up to roughly 10,000 times 2π. Low dimensions change quickly and distinguish neighboring tokens; high dimensions change slowly and separate distant regions. Because a shift in position corresponds to a linear rotation of each sine-cosine pair, the model can in principle learn to attend by relative offset.

Sinusoidal encodings need no trained parameters and produce a value for any position, but in practice models trained on short sequences still degrade on longer inputs, because the attention weights never saw those patterns during training.

Main types of positional encoding

Positional encoding methods differ in where position enters the computation and whether they represent absolute index or relative distance. The table compares the families that appear most often in practice.

MethodWhere it is appliedAbsolute or relativeTypical use
SinusoidalAdded to input embeddingsAbsoluteOriginal transformer, many encoders
Learned absoluteTrainable vector per indexAbsoluteBERT and early GPT models
Relative biasLearned bias on attention scoresRelativeT5-style models
ALiBiLinear distance penalty on scoresRelativeModels aiming at length extrapolation
Rotary position embeddingRotation of queries and keysRelativeMost current decoder language models
Families of positional encoding

Rotary position embedding, usually abbreviated RoPE, rotates query and key vectors by angles proportional to position, so their dot product depends only on the relative distance between tokens. It has become the default in open decoder models and has its own detailed treatment as a separate concept.

Key design parameters and their effects

Several parameters determine how well a positional scheme serves a given model. The most consequential is the maximum trained context length, because behavior beyond it is rarely reliable without adaptation.

ParameterTypical valuesWhat it affects
Trained context length512 to over 100,000 tokensRange of reliable positions
Frequency base10,000 by default, larger for long contextResolution of near versus far positions
Embedding dimension512 to 8,192 or moreNumber of distinct frequencies available
Scaling or interpolation factor1 to 32 in extension methodsHow positions are compressed into the trained range
Parameters that shape positional behavior

Context extension techniques such as position interpolation and frequency-base adjustment rescale positions so that a longer sequence maps into the range seen during training. They usually require a short phase of fine-tuning on long documents to recover accuracy.

Limitations and common mistakes

The most frequent problem is poor length extrapolation: a model that performs well at its trained length can produce incoherent output when the input grows longer. A second issue is that position information can fade in deep networks, since absolute encodings are added only once at the input.

SymptomLikely causeRemedy
Quality collapse past a lengthPositions beyond trained rangeInterpolation plus long-context fine-tuning
Weak use of distant contextAttention biased toward recent tokensAdjust frequency base or train on long data
Crash on long inputLearned table has no entry for the indexSwitch to a relative or extendable method
Mismatch after loading weightsDifferent encoding at inference than in trainingUse the exact original configuration
Typical symptoms, causes and remedies

A common misconception is that positional encoding stores meaning about the token; it carries only order or distance. Another is that relative methods remove all length limits. They extend gracefully further, but attention patterns still reflect the lengths seen in training.

The idea in brief

A transformer reads all the words of a sentence at once, like seeing a pile of word cards dropped on a table. Without extra help it cannot tell which card came first, so “dog bites man” and “man bites dog” look the same.

Positional encoding is like numbering the cards before they are dropped. Each word gets a small signal that says where it sits, or how far it is from its neighbors. With that signal, the model can follow word order, grammar and who did what to whom.

A practical scenario

A legal technology team fine-tunes an open language model with rotary position embedding and a trained context of 8,192 tokens to summarize contracts. Many of their contracts run to about 25,000 tokens, and summaries of those documents lose track of clauses near the end and sometimes repeat sections.

The engineers apply position interpolation with a scaling factor of 4, which compresses 32,768 positions into the range the model already knows, and then fine-tune for a short run on long contracts. They verify the change with retrieval tests that place a key clause at different depths in the document.

After the adjustment, the model correctly retrieves late clauses in long contracts, while quality on short documents stays essentially unchanged. The positional scheme, not the model size, was the limiting factor.