WordPiece

On this page 7

WordPiece is a subword tokenization algorithm that splits text into a fixed vocabulary of whole words and word fragments, choosing which fragments to keep by how much they increase the likelihood of the training corpus. In language models and wider artificial intelligence practice, it is best known as the tokenizer behind BERT and many encoder-only models derived from it, where a word such as “unhappiness” becomes the pieces un, ##happi and ##ness.

The method was first described by Google researchers for Japanese and Korean voice search and later became the default for BERT, DistilBERT and ELECTRA. It sits between character-level and word-level tokenization: the vocabulary stays small enough to train efficiently (typically about 30,000 entries for an English model), yet almost any word, including rare names and new coinages, can be represented as a sequence of known pieces.

How WordPiece builds its vocabulary

WordPiece training starts from a base vocabulary of single characters and repeatedly adds the merged pair of units that brings the largest gain in corpus likelihood under a simple language model. In practice, the common implementation scores each candidate pair by dividing the pair frequency by the product of the frequencies of its two parts, so a merge wins when its parts appear together far more often than chance would predict.

This scoring rule is the key difference from byte-pair encoding, which simply merges the most frequent pair. Two very common characters that often sit side by side do not automatically merge in WordPiece; a pair of rarer units that almost always co-occur may win instead. The result is a vocabulary that favors linguistically coherent fragments such as suffixes and stems.

Before training, text passes through normalization and pre-tokenization. The original BERT pipeline cleans control characters, optionally lowercases and strips accents (the “uncased” models), splits on whitespace and punctuation, and surrounds every CJK character with spaces. WordPiece then operates only inside these pre-split words, which is why it never produces a token that crosses a space.

The training procedure follows a small number of repeated steps, summarized below.

  1. Collect word frequencies from the pre-tokenized corpus.
  2. Initialize the vocabulary with all characters, marking non-initial ones with the ## prefix.
  3. Score every adjacent pair of units by its likelihood gain.
  4. Add the best-scoring merge to the vocabulary and update the counts.
  5. Stop when the vocabulary reaches the target size, such as 30,522 entries.

How WordPiece tokenizes new text

At inference time, WordPiece does not replay the merge history. It applies a greedy longest-match-first search, often called MaxMatch: starting at the beginning of a word, it takes the longest vocabulary entry that matches, then repeats from the next position using only ##-prefixed continuation pieces. The `##` prefix marks a piece that continues a word rather than starting one.

If the search reaches a position where no vocabulary entry matches, the entire word becomes `[UNK]`, not just the failing fragment. This all-or-nothing behavior is a notable weakness, because a single unusual character, such as a rare emoji or symbol, can erase the information carried by the rest of the word. The table below shows typical splits from an uncased English BERT vocabulary.

Input wordWordPiece outputWhat it shows
playingplay, ##ingCommon stem plus suffix
tokenizationtoken, ##izationProductive suffix kept whole
unaffableuna, ##ffa, ##bleRare word split into fragments
TransformerstransformersFrequent word stored whole
word with unknown symbol[UNK]Whole word lost on failure
Example splits from a typical English BERT vocabulary

A naive implementation of MaxMatch runs in quadratic time per word. Google later published Fast WordPiece, which uses an Aho-Corasick-style trie with failure links to tokenize in linear time, a meaningful saving for high-throughput serving.

Key parameters and special tokens

A WordPiece tokenizer is defined by a small set of design choices that strongly affect downstream model behavior. The vocabulary size, casing policy and maximum word length are fixed at training time and cannot be changed later without retraining the model’s embeddings.

ParameterTypical valueEffect
Vocabulary size30,522 English; about 119,500 multilingualLarger means shorter sequences, bigger embedding table
CasingUncased or casedUncased merges forms, loses capitalization signal
Continuation prefix##Separates word starts from inner pieces
Maximum characters per word100Longer words become [UNK]
Unknown token[UNK]Fallback for unmatched words
Common WordPiece settings and their effects

BERT-style vocabularies also reserve special tokens that carry structural meaning: [CLS] begins every input and feeds classification heads, [SEP] separates segments, [MASK] hides tokens for masked language model pretraining, and [PAD] fills batches to equal length. Some vocabularies keep a block of [unused] slots that practitioners can repurpose for domain terms without resizing the embedding matrix.

WordPiece compared with other subword methods

WordPiece is one of three dominant subword families, alongside byte-pair encoding and the Unigram language model. All three share the goal of a compact, open vocabulary, but they differ in the training criterion, the inference procedure and how they handle text the vocabulary cannot cover.

AlgorithmTraining criterionInferenceUnknown handlingTypical models
WordPieceLikelihood-based merge scoreGreedy longest matchWhole word to [UNK]BERT, DistilBERT, ELECTRA
Byte-pair encodingMost frequent pairReplay learned mergesByte fallback, no [UNK]GPT family, many decoder-only models
Unigram language modelPrune pieces by likelihood lossViterbi best segmentationCharacter or byte fallbackT5, ALBERT, XLNet
Comparison of major subword tokenization algorithms

The most practical consequence concerns robustness to unseen characters. Byte-level BPE can encode any UTF-8 string, so it never emits an unknown token, which is one reason modern decoder-only models favor it. WordPiece vocabularies built on characters must anticipate every script and symbol, and multilingual coverage requires a much larger vocabulary. Unigram, usually trained through the SentencePiece library, additionally supports subword regularization, sampling alternative segmentations during training.

Practical use and common pitfalls

Most engineers meet WordPiece when fine-tuning or serving an existing BERT-family model, and the overriding rule is simple: always use the exact tokenizer the model was pretrained with. A mismatched vocabulary or casing setting silently maps text to the wrong embeddings and degrades accuracy without raising an error.

Token-level tasks such as named entity recognition need extra care, because one word may produce several pieces while the labels exist per word. The usual approaches are listed in the table together with other frequent problems.

SymptomLikely causeRemedy
Many [UNK] tokensEmoji, rare scripts or symbols missingNormalize input or choose a broader vocabulary
Misaligned entity labelsWords split into several piecesLabel first piece, mask the others
Inputs truncated earlyDomain jargon split into many piecesAdd domain tokens or use a domain vocabulary
Lost case informationUncased model used for case-sensitive dataSwitch to a cased checkpoint
Poor multilingual qualityLow-resource languages over-fragmentedUse a language-specific or larger vocabulary
Frequent WordPiece problems and their remedies

Over-fragmentation also matters for cost: the context window of most BERT models is 512 tokens, so a vocabulary that splits technical or non-English text into many pieces leaves less room for actual content. Domain-specific models in biomedicine and law often retrain WordPiece on in-domain text for exactly this reason.

A non-technical view

A computer model cannot read words the way people do. It needs a fixed list of building blocks, much like a set of stamps. WordPiece is the method that decides which stamps go into that set. Common words get their own stamp, while rare words are printed by combining smaller stamps for familiar parts, such as “un”, “happi” and “ness”.

The main benefit is flexibility with a modest list: about thirty thousand stamps are enough to spell nearly any English word. The main weakness is that if one part of a word has no stamp at all, the whole word is replaced with a blank marker meaning “unknown”.

A concrete example

A hospital analytics team fine-tunes a general English BERT model to extract medication names from discharge notes. During testing, the engineers notice that a drug name such as “pembrolizumab” is split into six or more WordPiece fragments, and dense notes regularly exceed the 512-token limit, forcing truncation that cuts off the medication list at the end of each note.

The team switches to a biomedical model whose WordPiece vocabulary was trained on clinical and research text. The same drug names now need only two or three pieces, the average note shrinks by roughly a fifth in token count, and fewer notes are truncated. Entity extraction accuracy on the held-out test set improves noticeably, with the largest gains on rare drug names, showing that the tokenizer vocabulary is a real modeling decision rather than a preprocessing detail.