Rotary position embedding

Also known as: RoPE

On this page 7

Rotary position embedding (RoPE) is a method for encoding token position in transformer models by rotating query and key vectors by angles proportional to each token’s position. Because the dot product of two rotated vectors depends only on the difference between their rotation angles, attention scores become a function of the relative distance between tokens rather than of their absolute positions.

Neural networks built on self-attention are otherwise permutation-invariant: without positional information they cannot tell the order of their inputs. RoPE, introduced by Su and colleagues in the RoFormer paper in 2021, has become the default positional scheme in most open large language models, and it is a central topic in artificial intelligence work on long-context modeling.

How RoPE rotates queries and keys

RoPE splits each query and key vector of dimension d into d/2 two-dimensional pairs. Each pair is treated as a point in a plane and rotated by the angle m·θ_i, where m is the token position and θ_i is a pair-specific frequency. In the standard formulation, θ_i = base^(-2i/d) with a base of 10,000, so early pairs rotate quickly and later pairs rotate slowly.

The key property follows from rotation algebra. If a query at position m and a key at position n are rotated by mθ and nθ, their dot product equals the dot product of the unrotated vectors with a single rotation by (m−n)θ. Relative position is therefore built into the attention score without any learned bias table or extra parameters.

The table below lists the components of the computation and what each one controls.

ComponentTypical valueEffect
Head dimension d64 to 128Number of rotated pairs, d/2
Base frequency10,000 to 1,000,000Sets the longest wavelength and long-range resolution
Frequency θ_iFrom 1 down to about 1/baseFast pairs capture local order, slow pairs capture distant order
Position m0 to context length minus 1Multiplies every pair angle
Rotated vectorsQueries and keys onlyValues stay unrotated, so outputs are not shifted
Components of the RoPE computation

In practice the rotation is implemented with element-wise multiplications by precomputed cosine and sine tables, not with explicit matrix products. The cost is negligible compared with the attention computation itself, and the rotation is applied inside every attention layer, not once at the input.

RoPE compared with other positional encodings

RoPE is one of several answers to the same problem, and its advantages are easiest to see against the alternatives. The original transformer added fixed sinusoidal encodings to token embeddings, while later models experimented with learned tables and attention biases.

MethodPosition typeWhere appliedExtra parameters
Sinusoidal absoluteAbsoluteAdded to input embeddingsNone
Learned absoluteAbsoluteAdded to input embeddingsOne vector per position
T5 relative biasRelative, bucketedAdded to attention logitsSmall bias table
ALiBiRelative, linear penaltyAdded to attention logitsNone
RoPERelative via rotationMultiplied into queries and keysNone
Positional encoding methods compared

RoPE combines several useful properties: no learned position parameters, compatibility with efficient attention kernels, and a natural decay of attention with distance for many vector pairs. Unlike additive biases, it modifies the representation itself, which lets the model learn how strongly each head relies on position. ALiBi is simpler to extrapolate, but RoPE models have generally shown stronger quality at the trained length.

Extending the context window

A RoPE model trained on 4,096 tokens usually degrades sharply beyond its training length, because the slow pairs encounter rotation angles they never saw during training. Context extension methods change how positions map to angles and are normally combined with short fine-tuning on longer sequences.

MethodMechanismMain trade-off
Position interpolationScales all positions down by the extension factorBlurs local order for short distances
NTK-aware scalingIncreases the base so low frequencies stretch moreWorks without tuning but less precisely
YaRNScales frequency bands differently and adjusts attention temperatureMore hyperparameters to set
Larger base at pretrainingTrains with a base such as 500,000 from the startRequires long-sequence training data
LongRoPESearches per-dimension scaling factorsSearch cost and added complexity
Common RoPE context extension methods

The practical lesson is that a long advertised context window does not guarantee effective long-context use. Retrieval tests over the full window, such as finding a planted fact deep in a document, often reveal degraded accuracy in the middle of long inputs even after successful extension.

Implementation details and variants

Implementations differ in how they pair dimensions. Some libraries rotate adjacent elements (dimension 0 with 1, 2 with 3), while others rotate the first half against the second half. Both are mathematically valid, but weights trained with one layout produce wrong outputs under the other, which is a frequent source of silent errors when models are converted between frameworks.

  • Partial RoPE rotates only a fraction of each head, leaving the rest position-free.
  • Decoupled RoPE in multi-head latent attention keeps a separate rotated component so key compression still works.
  • Multimodal RoPE assigns separate angle sets to time, height and width for video and image tokens.
  • KV cache handling stores keys after rotation, so cached keys must not be rotated a second time.

Limitations and common mistakes

RoPE is robust but not free of pitfalls. The table below links frequent symptoms to their usual causes and remedies.

SymptomLikely causeFix
Gibberish after conversionMismatched pairing layoutMatch interleaved or half-split convention
Collapse beyond training lengthUnseen rotation anglesApply interpolation or YaRN and fine-tune
Repetition in long generationsWrong position offset in KV cacheTrack absolute positions across decoding steps
Numerical drift at long positionsLow-precision angle computationCompute angles in 32-bit floating point
Typical RoPE problems and fixes

A conceptual misconception is that RoPE makes models position-agnostic beyond training length. It encodes relative distance elegantly, but the network still learns its attention patterns only for distances it observed during training.

The short version

A language model reads words as a set of vectors. Without help, it cannot tell which word came first. Rotary position embedding solves this by turning each word’s vector by an angle that depends on its place in the text.

Picture the hands of many clocks moving at different speeds. When two words are compared, only the difference between their clock positions matters. So the model learns how far apart words are, not their exact seat numbers. That is why RoPE handles word order cheaply and reliably.

Where it makes a difference

A legal technology team runs an open-weight language model trained with an 8,192-token context and a RoPE base of 500,000. Its clients want contract reviews of documents up to 60,000 tokens, and the unmodified model produces incoherent answers past about 9,000 tokens.

The engineers apply YaRN scaling with a factor of 8 and fine-tune the model for several hundred steps on long contracts. They then test retrieval by planting clause references at 20 depths across 64,000-token documents. Accuracy rises from near zero beyond the original window to above 90% across most depths, with a modest dip in the middle. The outcome is a usable long-document assistant built without retraining the base model from scratch.