A masked language model (MLM) is a neural network trained to predict deliberately hidden tokens in a sequence by reading the context on both sides of each gap. Among language models in artificial intelligence, it is the classic pretraining approach behind encoder-only models such as BERT, RoBERTa and DeBERTa, and its objective is usually called masked language modeling.
Unlike an autoregressive model, which writes text one token at a time from left to right, a masked language model is optimized to understand text rather than generate it. Its output is a set of contextual embeddings that downstream systems use for classification, search, entity extraction and ranking.
How masked language modeling trains a model
Masked language modeling trains a network by corrupting part of the input and asking it to restore the original tokens. The text is first split by a tokenizer such as WordPiece or byte-pair encoding, then a random subset of positions is selected, and the loss is computed only on those selected positions.
The original BERT recipe selects 15 percent of tokens. Of the selected positions, 80 percent are replaced with a special [MASK] token, 10 percent with a random token and 10 percent are left unchanged. This 80-10-10 rule reduces the mismatch between pretraining, where [MASK] appears, and fine-tuning, where it never does.
Because every layer uses bidirectional self-attention with no causal mask, each position can attend to every other position. The final hidden state above a masked slot passes through a prediction head that outputs a probability distribution over the vocabulary, and cross-entropy loss compares it with the true token.
The following table summarizes the main components of a standard masked language model pipeline.
| Component | Role | Typical setting |
|---|---|---|
| Tokenizer | Splits text into subword units | WordPiece or BPE, 30,000-130,000 tokens |
| Masking policy | Chooses positions to corrupt | 15 percent, 80-10-10 replacement |
| Encoder stack | Builds contextual representations | 12-24 Transformer layers |
| Prediction head | Maps hidden states to vocabulary logits | Linear layer tied to embeddings |
| Loss | Scores predictions on masked slots | Cross-entropy on selected tokens |
Masking strategies and model variants
Researchers have refined which tokens are masked and how the task is framed, because random single-token masking leaves easy shortcuts. Masking a fragment of a word, for example, lets the model guess it from the remaining fragments without learning much about meaning.
The table below compares common variants of the objective and what each one changes.
| Strategy | What changes | Main benefit |
|---|---|---|
| Static masking | Masks fixed once during preprocessing | Simple and reproducible |
| Dynamic masking | New mask for every pass over the data | More varied training signal |
| Whole-word masking | All subwords of a word masked together | Removes subword shortcuts |
| Span masking | Contiguous spans of tokens masked | Better phrase-level understanding |
| Replaced token detection | Discriminator flags substituted tokens | Learns from every position |
| Higher mask rates | 30-40 percent of tokens masked | Can help larger models |
RoBERTa showed that dynamic masking, more data and longer training matter more than BERT’s auxiliary next-sentence prediction task, which it dropped. ELECTRA replaced the generative objective with replaced token detection, which is more sample-efficient because the loss covers all positions instead of only 15 percent. Multilingual variants such as XLM-R apply the same objective across about 100 languages.
Masked versus autoregressive language models
The essential difference is the direction of context: a masked language model sees both sides of a gap, while an autoregressive model sees only preceding tokens and predicts the next one. That choice shapes what each family does well and how it is deployed.
| Property | Masked language model | Autoregressive model |
|---|---|---|
| Training objective | Fill in hidden tokens | Next-token prediction |
| Attention pattern | Bidirectional | Causal, left to right |
| Typical architecture | Encoder-only | Decoder-only |
| Signal per sequence | About 15 percent of positions | Every position |
| Text generation | Poor without extra machinery | Native strength |
| Typical size | 100 million to 1.5 billion parameters | Billions to hundreds of billions |
| Best fit | Classification, retrieval, tagging | Chat, writing, code, reasoning |
Encoder-decoder models such as T5 sit between the two families: they use a span corruption objective resembling masking, yet decode answers autoregressively. Discrete diffusion language models also revisit iterative unmasking as a route to parallel generation, which shows the idea remains active research rather than a closed chapter.
Where masked language models are used
Masked language models remain the workhorse for text understanding where latency, cost and accuracy on a fixed task matter more than open-ended generation. A model with roughly 110 million parameters can run on a CPU at high throughput, which a large generative model cannot.
The decision table below matches typical tasks to the way an MLM is adapted.
| Task | Adaptation | Reason |
|---|---|---|
| Sentiment or topic labels | Fine-tune with a classification head | High accuracy from modest labeled data |
| Named entity recognition | Token-level tagging head | Every token receives contextual features |
| Semantic search | Contrastive tuning into a bi-encoder | Fast vector retrieval at scale |
| Search reranking | Cross-encoder on query and passage pairs | Precise relevance scores |
| Domain text such as legal or clinical | Continued MLM pretraining on domain data | Adapts vocabulary and usage |
A practical rule is to start from a pretrained checkpoint and fine-tune all layers with a small learning rate, typically 1e-5 to 5e-5, for a few epochs. Domain-adaptive pretraining, which simply continues masked language modeling on in-domain text, often yields measurable gains before task fine-tuning.
Limitations and common misconceptions
The objective has known weaknesses that practitioners should account for when choosing or evaluating a model. The most frequent issues are listed below.
- Pretrain-finetune mismatch: the
[MASK]token never appears in real inputs, which the 80-10-10 rule only partly offsets. - Independence assumption: several masked tokens are predicted independently, so the model cannot coordinate two related gaps.
- Limited context window: classic checkpoints accept 512 tokens, so long documents require chunking; newer encoders extend this to several thousand.
- Not a chatbot: an MLM is not designed for free-form generation, and sampling text from it produces incoherent output.
- Raw embeddings are weak for similarity: averaged hidden states rank sentences poorly until the model is contrastively fine-tuned.
A further misconception holds that masked language models are obsolete. In practice they still power a large share of production retrieval, moderation and tagging pipelines, because they deliver strong accuracy at a small fraction of the inference cost of a large generative model.
Stripped to the essentials
A masked language model learns language by playing a fill-in-the-blank game millions of times. Words in a sentence are hidden, and the model must guess them by looking at the words before and after each gap.
Because it always reads the whole sentence at once, it becomes very good at understanding what a text means, such as whether a review is positive or which words name a person or a place. It is not built to write long answers. Think of it as a careful reader rather than a storyteller.
Where it is applied
A mid-sized insurance company receives about 20,000 customer emails a week and wants them routed automatically to claims, billing, complaints or policy changes. The data team starts from a pretrained masked language model with about 110 million parameters and continues masked language modeling for two days on a year of anonymized emails, so the model learns industry vocabulary such as deductible and subrogation.
They then fine-tune it on 8,000 hand-labeled emails with a classification head. On a held-out set, the model assigns the correct department to about 94 percent of messages, compared with roughly 80 percent for the previous keyword rules. It runs on ordinary CPU servers in under 30 milliseconds per email, and manual triage time drops by more than half, with uncertain cases still sent to a human reviewer.