Byte-pair encoding

Also known as: BPE

On this page 7

Byte-pair encoding (BPE) is a subword tokenization algorithm that learns a vocabulary by repeatedly merging the most frequent pair of adjacent symbols in a training corpus, so that common words become single tokens while rare words split into smaller, reusable pieces. In language models, BPE is the dominant way to turn raw text into the integer token IDs that a neural network actually processes.

The method began in 1994 as a simple data compression technique and was adapted for neural machine translation in 2016. Today most decoder-only language models in artificial intelligence, including the GPT family and many open-weight models, use a byte-level variant of BPE, which guarantees that any input string can be encoded without an unknown-token fallback.

How byte-pair encoding builds a vocabulary

BPE training is a greedy, frequency-driven merge loop. It starts from a base alphabet of single characters or bytes, counts every adjacent pair in the corpus, merges the most frequent pair into a new symbol, and repeats until the vocabulary reaches a target size chosen in advance.

  1. Pre-tokenize the corpus, usually by splitting on whitespace and punctuation with a regular expression, so merges never cross word boundaries.
  2. Represent each pre-token as a sequence of base symbols, for example UTF-8 bytes.
  3. Count the frequency of every adjacent symbol pair across the whole corpus.
  4. Merge the highest-frequency pair into one new symbol and record the merge as a rule.
  5. Update the counts and repeat until the vocabulary size is reached.

The output of training is two artifacts: a vocabulary that maps each symbol to an integer ID, and an ordered list of merge rules. The order matters, because encoding new text replays the merges in exactly the sequence in which they were learned.

The following table traces the first merges on a toy corpus containing the words low, lower, newest and widest with their frequencies.

StepMost frequent pairNew symbolEffect
1e + sesShared by newest and widest
2es + testCaptures the superlative suffix
3l + oloShared by low and lower
4lo + wlowWhole word becomes one token
5n + eneBegins the stem of newest
First merge steps on a toy corpus

Encoding and decoding text with learned merges

At inference time, a BPE tokenizer applies the stored merge rules in priority order to each pre-token. A word seen often during training collapses into one token, while an unseen word such as a new product name breaks into several familiar subwords, so the model never encounters a truly unknown input.

Decoding is the exact inverse: the tokenizer maps IDs back to symbols and concatenates them. In byte-level BPE the result is a byte sequence that is then decoded as UTF-8, which makes the round trip lossless for any input, including emoji, code and mixed scripts. Some implementations also mark word starts with a special prefix character so that spaces can be restored exactly.

Two implementation details strongly affect behavior. The pre-tokenization regular expression decides, for instance, whether digits are split individually and whether a leading space is attached to the following word. Special tokens such as end-of-text markers or chat template delimiters are added outside the merge process and must never be produced from ordinary user text.

Key parameters and design choices

The most consequential setting is vocabulary size, which trades shorter sequences against a larger embedding matrix. Other choices, listed below, determine how fairly the tokenizer treats languages, numbers and source code.

ParameterTypical valuesWhat it affects
Vocabulary size32,000 to 256,000 tokensSequence length, embedding memory, rare-word coverage
Base alphabetUnicode characters or 256 bytesWhether unknown tokens can occur
Pre-tokenization patternWhitespace, punctuation, digit rulesWhich merges are allowed
Digit handlingSingle digits or groups of up to threeArithmetic consistency
Training corpus mixLanguage and code proportionsFertility per language
BPE dropout rate0 to 0.1 during trainingRobustness to segmentation noise
Main BPE design parameters

A useful diagnostic is fertility, the average number of tokens per word. English text with a modern tokenizer typically averages about 1.3 tokens per word, while languages that were underrepresented in the training corpus can need two to four times more tokens for the same meaning, which raises cost and consumes the context window faster.

How BPE compares with other subword tokenizers

BPE is one of three widely used subword segmentation families, and the differences lie mainly in how each method selects vocabulary entries. WordPiece and the unigram language model reach similar quality in practice, so the choice is often driven by tooling and model lineage.

MethodSelection criterionSegmentation at inferenceCommon use
BPEMost frequent adjacent pairDeterministic replay of mergesGPT family, Llama, many open-weight models
WordPiecePair that most increases likelihoodGreedy longest match from the leftBERT and other encoder-only models
Unigram language modelPrune tokens that least reduce likelihoodMost probable segmentationT5 and other SentencePiece models
Character or byte levelNo learned mergesOne symbol per character or byteResearch models, robustness studies
Subword tokenization methods compared

SentencePiece is often confused with an algorithm, but it is a library that can train either BPE or unigram models directly on raw text, treating the space as an ordinary symbol. Likewise, tiktoken and similar packages are fast BPE implementations, not separate methods.

Limitations and common failure modes

Because BPE optimizes for compression frequency rather than meaning, its token boundaries rarely align with morphemes, digits or characters. Many surprising model errors trace back to segmentation rather than to reasoning.

SymptomUnderlying causeMitigation
Errors counting letters in a wordCharacters hidden inside multi-letter tokensSpell the word out or use tools
Inconsistent arithmeticNumbers split into irregular chunksSplit digits individually
Higher cost for some languagesCorpus dominated by EnglishBalance the tokenizer training data
Glitch tokens with odd outputsTokens frequent in tokenizer data but rare in model dataTrain tokenizer and model on matched data
Leading-space sensitivitySpace attached to the next wordNormalize prompts and stop sequences
Typical tokenization problems and mitigations

A tokenizer is also fixed once a model is trained. Changing the vocabulary later requires retraining or carefully extending the embedding matrix, so teams should treat tokenizer design as a foundational, hard-to-reverse decision and evaluate it on representative multilingual and code samples before pretraining.

Explained without jargon

A computer model cannot read letters the way people do, so text must first be cut into pieces and turned into numbers. Byte-pair encoding decides where those cuts go. It studies a huge amount of text and notices which letter pairs appear together most often, then glues them into bigger pieces, again and again.

The result works like a set of word fragments. Everyday words become one piece, while a rare or brand-new word is assembled from several familiar fragments, much as a child sounds out an unfamiliar name syllable by syllable. The key point is that nothing is ever unreadable: any text can always be broken into known pieces.

A typical use case

A team building a customer-support assistant for a European retailer compares two candidate open-weight models before deployment. Both answer well, but the team measures token counts on 10,000 real tickets in English, German and Polish. One tokenizer, with a vocabulary of about 128,000 tokens trained on a multilingual corpus, encodes the Polish tickets with roughly 40 percent fewer tokens than the other, which uses a smaller, English-heavy BPE vocabulary.

Because hosting cost and latency scale with token count, the team selects the multilingual model. The decision cuts projected inference spend for Polish traffic by about a third and leaves more of the context window free for retrieved product documentation, without any change to prompts or model quality.