Byte-pair encoding (BPE) is a subword tokenization algorithm that learns a vocabulary by repeatedly merging the most frequent pair of adjacent symbols in a training corpus, so that common words become single tokens while rare words split into smaller, reusable pieces. In language models, BPE is the dominant way to turn raw text into the integer token IDs that a neural network actually processes.
The method began in 1994 as a simple data compression technique and was adapted for neural machine translation in 2016. Today most decoder-only language models in artificial intelligence, including the GPT family and many open-weight models, use a byte-level variant of BPE, which guarantees that any input string can be encoded without an unknown-token fallback.
How byte-pair encoding builds a vocabulary
BPE training is a greedy, frequency-driven merge loop. It starts from a base alphabet of single characters or bytes, counts every adjacent pair in the corpus, merges the most frequent pair into a new symbol, and repeats until the vocabulary reaches a target size chosen in advance.
- Pre-tokenize the corpus, usually by splitting on whitespace and punctuation with a regular expression, so merges never cross word boundaries.
- Represent each pre-token as a sequence of base symbols, for example UTF-8 bytes.
- Count the frequency of every adjacent symbol pair across the whole corpus.
- Merge the highest-frequency pair into one new symbol and record the merge as a rule.
- Update the counts and repeat until the vocabulary size is reached.
The output of training is two artifacts: a vocabulary that maps each symbol to an integer ID, and an ordered list of merge rules. The order matters, because encoding new text replays the merges in exactly the sequence in which they were learned.
The following table traces the first merges on a toy corpus containing the words low, lower, newest and widest with their frequencies.
| Step | Most frequent pair | New symbol | Effect |
|---|---|---|---|
| 1 | e + s | es | Shared by newest and widest |
| 2 | es + t | est | Captures the superlative suffix |
| 3 | l + o | lo | Shared by low and lower |
| 4 | lo + w | low | Whole word becomes one token |
| 5 | n + e | ne | Begins the stem of newest |
Encoding and decoding text with learned merges
At inference time, a BPE tokenizer applies the stored merge rules in priority order to each pre-token. A word seen often during training collapses into one token, while an unseen word such as a new product name breaks into several familiar subwords, so the model never encounters a truly unknown input.
Decoding is the exact inverse: the tokenizer maps IDs back to symbols and concatenates them. In byte-level BPE the result is a byte sequence that is then decoded as UTF-8, which makes the round trip lossless for any input, including emoji, code and mixed scripts. Some implementations also mark word starts with a special prefix character so that spaces can be restored exactly.
Two implementation details strongly affect behavior. The pre-tokenization regular expression decides, for instance, whether digits are split individually and whether a leading space is attached to the following word. Special tokens such as end-of-text markers or chat template delimiters are added outside the merge process and must never be produced from ordinary user text.
Key parameters and design choices
The most consequential setting is vocabulary size, which trades shorter sequences against a larger embedding matrix. Other choices, listed below, determine how fairly the tokenizer treats languages, numbers and source code.
| Parameter | Typical values | What it affects |
|---|---|---|
| Vocabulary size | 32,000 to 256,000 tokens | Sequence length, embedding memory, rare-word coverage |
| Base alphabet | Unicode characters or 256 bytes | Whether unknown tokens can occur |
| Pre-tokenization pattern | Whitespace, punctuation, digit rules | Which merges are allowed |
| Digit handling | Single digits or groups of up to three | Arithmetic consistency |
| Training corpus mix | Language and code proportions | Fertility per language |
| BPE dropout rate | 0 to 0.1 during training | Robustness to segmentation noise |
A useful diagnostic is fertility, the average number of tokens per word. English text with a modern tokenizer typically averages about 1.3 tokens per word, while languages that were underrepresented in the training corpus can need two to four times more tokens for the same meaning, which raises cost and consumes the context window faster.
How BPE compares with other subword tokenizers
BPE is one of three widely used subword segmentation families, and the differences lie mainly in how each method selects vocabulary entries. WordPiece and the unigram language model reach similar quality in practice, so the choice is often driven by tooling and model lineage.
| Method | Selection criterion | Segmentation at inference | Common use |
|---|---|---|---|
| BPE | Most frequent adjacent pair | Deterministic replay of merges | GPT family, Llama, many open-weight models |
| WordPiece | Pair that most increases likelihood | Greedy longest match from the left | BERT and other encoder-only models |
| Unigram language model | Prune tokens that least reduce likelihood | Most probable segmentation | T5 and other SentencePiece models |
| Character or byte level | No learned merges | One symbol per character or byte | Research models, robustness studies |
SentencePiece is often confused with an algorithm, but it is a library that can train either BPE or unigram models directly on raw text, treating the space as an ordinary symbol. Likewise, tiktoken and similar packages are fast BPE implementations, not separate methods.
Limitations and common failure modes
Because BPE optimizes for compression frequency rather than meaning, its token boundaries rarely align with morphemes, digits or characters. Many surprising model errors trace back to segmentation rather than to reasoning.
| Symptom | Underlying cause | Mitigation |
|---|---|---|
| Errors counting letters in a word | Characters hidden inside multi-letter tokens | Spell the word out or use tools |
| Inconsistent arithmetic | Numbers split into irregular chunks | Split digits individually |
| Higher cost for some languages | Corpus dominated by English | Balance the tokenizer training data |
| Glitch tokens with odd outputs | Tokens frequent in tokenizer data but rare in model data | Train tokenizer and model on matched data |
| Leading-space sensitivity | Space attached to the next word | Normalize prompts and stop sequences |
A tokenizer is also fixed once a model is trained. Changing the vocabulary later requires retraining or carefully extending the embedding matrix, so teams should treat tokenizer design as a foundational, hard-to-reverse decision and evaluate it on representative multilingual and code samples before pretraining.
Explained without jargon
A computer model cannot read letters the way people do, so text must first be cut into pieces and turned into numbers. Byte-pair encoding decides where those cuts go. It studies a huge amount of text and notices which letter pairs appear together most often, then glues them into bigger pieces, again and again.
The result works like a set of word fragments. Everyday words become one piece, while a rare or brand-new word is assembled from several familiar fragments, much as a child sounds out an unfamiliar name syllable by syllable. The key point is that nothing is ever unreadable: any text can always be broken into known pieces.
A typical use case
A team building a customer-support assistant for a European retailer compares two candidate open-weight models before deployment. Both answer well, but the team measures token counts on 10,000 real tickets in English, German and Polish. One tokenizer, with a vocabulary of about 128,000 tokens trained on a multilingual corpus, encodes the Polish tickets with roughly 40 percent fewer tokens than the other, which uses a smaller, English-heavy BPE vocabulary.
Because hosting cost and latency scale with token count, the team selects the multilingual model. The decision cuts projected inference spend for Polish traffic by about a third and leaves more of the context window free for retrieved product documentation, without any change to prompts or model quality.