Masked language model

Also known as: MLM · masked language modeling

On this page 7

A masked language model (MLM) is a neural network trained to predict deliberately hidden tokens in a sequence by reading the context on both sides of each gap. Among language models in artificial intelligence, it is the classic pretraining approach behind encoder-only models such as BERT, RoBERTa and DeBERTa, and its objective is usually called masked language modeling.

Unlike an autoregressive model, which writes text one token at a time from left to right, a masked language model is optimized to understand text rather than generate it. Its output is a set of contextual embeddings that downstream systems use for classification, search, entity extraction and ranking.

How masked language modeling trains a model

Masked language modeling trains a network by corrupting part of the input and asking it to restore the original tokens. The text is first split by a tokenizer such as WordPiece or byte-pair encoding, then a random subset of positions is selected, and the loss is computed only on those selected positions.

The original BERT recipe selects 15 percent of tokens. Of the selected positions, 80 percent are replaced with a special [MASK] token, 10 percent with a random token and 10 percent are left unchanged. This 80-10-10 rule reduces the mismatch between pretraining, where [MASK] appears, and fine-tuning, where it never does.

Because every layer uses bidirectional self-attention with no causal mask, each position can attend to every other position. The final hidden state above a masked slot passes through a prediction head that outputs a probability distribution over the vocabulary, and cross-entropy loss compares it with the true token.

The following table summarizes the main components of a standard masked language model pipeline.

ComponentRoleTypical setting
TokenizerSplits text into subword unitsWordPiece or BPE, 30,000-130,000 tokens
Masking policyChooses positions to corrupt15 percent, 80-10-10 replacement
Encoder stackBuilds contextual representations12-24 Transformer layers
Prediction headMaps hidden states to vocabulary logitsLinear layer tied to embeddings
LossScores predictions on masked slotsCross-entropy on selected tokens
Components of a masked language model

Masking strategies and model variants

Researchers have refined which tokens are masked and how the task is framed, because random single-token masking leaves easy shortcuts. Masking a fragment of a word, for example, lets the model guess it from the remaining fragments without learning much about meaning.

The table below compares common variants of the objective and what each one changes.

StrategyWhat changesMain benefit
Static maskingMasks fixed once during preprocessingSimple and reproducible
Dynamic maskingNew mask for every pass over the dataMore varied training signal
Whole-word maskingAll subwords of a word masked togetherRemoves subword shortcuts
Span maskingContiguous spans of tokens maskedBetter phrase-level understanding
Replaced token detectionDiscriminator flags substituted tokensLearns from every position
Higher mask rates30-40 percent of tokens maskedCan help larger models
Common masking strategies

RoBERTa showed that dynamic masking, more data and longer training matter more than BERT’s auxiliary next-sentence prediction task, which it dropped. ELECTRA replaced the generative objective with replaced token detection, which is more sample-efficient because the loss covers all positions instead of only 15 percent. Multilingual variants such as XLM-R apply the same objective across about 100 languages.

Masked versus autoregressive language models

The essential difference is the direction of context: a masked language model sees both sides of a gap, while an autoregressive model sees only preceding tokens and predicts the next one. That choice shapes what each family does well and how it is deployed.

PropertyMasked language modelAutoregressive model
Training objectiveFill in hidden tokensNext-token prediction
Attention patternBidirectionalCausal, left to right
Typical architectureEncoder-onlyDecoder-only
Signal per sequenceAbout 15 percent of positionsEvery position
Text generationPoor without extra machineryNative strength
Typical size100 million to 1.5 billion parametersBillions to hundreds of billions
Best fitClassification, retrieval, taggingChat, writing, code, reasoning
Masked and autoregressive objectives compared

Encoder-decoder models such as T5 sit between the two families: they use a span corruption objective resembling masking, yet decode answers autoregressively. Discrete diffusion language models also revisit iterative unmasking as a route to parallel generation, which shows the idea remains active research rather than a closed chapter.

Where masked language models are used

Masked language models remain the workhorse for text understanding where latency, cost and accuracy on a fixed task matter more than open-ended generation. A model with roughly 110 million parameters can run on a CPU at high throughput, which a large generative model cannot.

The decision table below matches typical tasks to the way an MLM is adapted.

TaskAdaptationReason
Sentiment or topic labelsFine-tune with a classification headHigh accuracy from modest labeled data
Named entity recognitionToken-level tagging headEvery token receives contextual features
Semantic searchContrastive tuning into a bi-encoderFast vector retrieval at scale
Search rerankingCross-encoder on query and passage pairsPrecise relevance scores
Domain text such as legal or clinicalContinued MLM pretraining on domain dataAdapts vocabulary and usage
Choosing how to apply a masked language model

A practical rule is to start from a pretrained checkpoint and fine-tune all layers with a small learning rate, typically 1e-5 to 5e-5, for a few epochs. Domain-adaptive pretraining, which simply continues masked language modeling on in-domain text, often yields measurable gains before task fine-tuning.

Limitations and common misconceptions

The objective has known weaknesses that practitioners should account for when choosing or evaluating a model. The most frequent issues are listed below.

  • Pretrain-finetune mismatch: the [MASK] token never appears in real inputs, which the 80-10-10 rule only partly offsets.
  • Independence assumption: several masked tokens are predicted independently, so the model cannot coordinate two related gaps.
  • Limited context window: classic checkpoints accept 512 tokens, so long documents require chunking; newer encoders extend this to several thousand.
  • Not a chatbot: an MLM is not designed for free-form generation, and sampling text from it produces incoherent output.
  • Raw embeddings are weak for similarity: averaged hidden states rank sentences poorly until the model is contrastively fine-tuned.

A further misconception holds that masked language models are obsolete. In practice they still power a large share of production retrieval, moderation and tagging pipelines, because they deliver strong accuracy at a small fraction of the inference cost of a large generative model.

Stripped to the essentials

A masked language model learns language by playing a fill-in-the-blank game millions of times. Words in a sentence are hidden, and the model must guess them by looking at the words before and after each gap.

Because it always reads the whole sentence at once, it becomes very good at understanding what a text means, such as whether a review is positive or which words name a person or a place. It is not built to write long answers. Think of it as a careful reader rather than a storyteller.

Where it is applied

A mid-sized insurance company receives about 20,000 customer emails a week and wants them routed automatically to claims, billing, complaints or policy changes. The data team starts from a pretrained masked language model with about 110 million parameters and continues masked language modeling for two days on a year of anonymized emails, so the model learns industry vocabulary such as deductible and subrogation.

They then fine-tune it on 8,000 hand-labeled emails with a classification head. On a held-out set, the model assigns the correct department to about 94 percent of messages, compared with roughly 80 percent for the previous keyword rules. It runs on ordinary CPU servers in under 30 milliseconds per email, and manual triage time drops by more than half, with uncertain cases still sent to a human reviewer.