Diffusion language model

On this page 7

A diffusion language model is a text generator that produces a whole sequence by iteratively denoising a corrupted draft, typically starting from a fully masked or noisy sequence and refining many positions in parallel over a series of steps. Among language models in artificial intelligence, it is the main alternative to the autoregressive model, which writes text strictly left to right through next-token prediction.

The approach adapts the diffusion framework that dominates image generation to discrete tokens. Its appeal lies in parallel decoding, bidirectional context and the ability to revise earlier choices, while its costs include more complex training objectives, weaker tooling and quality that still depends heavily on the number of refinement steps.

How a diffusion language model generates text

A diffusion language model learns to reverse a forward corruption process that gradually destroys a text sequence. During training, a clean sentence is corrupted to a randomly chosen noise level, and the network learns to predict the original tokens from the damaged version. At inference time, the model starts from maximum corruption and applies the learned reverse process step by step until a clean sequence remains.

In the now dominant masked diffusion formulation, corruption means replacing tokens with a special mask symbol. At noise level t, each token is masked independently with probability t, so t = 1 is an entirely masked sequence. The denoiser is usually a bidirectional transformer, closer to an encoder-only model than to a decoder-only model, because every position may attend to every other position.

Generation then proceeds as iterative unmasking. At each step, the model predicts a distribution for every masked position, commits a subset of tokens and leaves the rest masked for later steps. Many samplers use confidence-based remasking: the most certain predictions are kept and uncertain ones are masked again, so difficult positions benefit from more surrounding context.

The following table summarizes the stages of one generation pass in a masked diffusion language model.

StageWhat happensMain design choice
InitializationA sequence of mask tokens of fixed length is createdOutput length or block size
PredictionThe denoiser scores every masked position at onceModel size and attention pattern
CommitmentA subset of positions receives final tokensTokens unmasked per step
RemaskingLow-confidence tokens may return to the mask stateConfidence threshold or schedule
TerminationNo masks remain or the step budget is used upNumber of sampling steps
Stages of masked diffusion generation

Main variants of diffusion language models

Diffusion language models differ mainly in where the noise is applied and in how strictly they abandon left-to-right order. Early research added Gaussian noise to continuous word embeddings and rounded the result back to tokens, but rounding errors and weak likelihoods limited quality. Discrete formulations, which corrupt tokens directly, proved easier to scale and now underpin most large systems.

The table compares the principal families and the situations in which each is typically chosen.

FamilyCorruption typeTypical strengthTypical weakness
Continuous embedding diffusionGaussian noise on embeddingsSmooth controllable generationRounding errors, lower fluency
Uniform discrete diffusionTokens replaced by random tokensCan revise any tokenHarder to train at scale
Masked (absorbing) diffusionTokens replaced by a mask symbolSimple objective, strong scalingCommitted tokens rarely revised
Block diffusionMasking inside sequential blocksVariable length, reusable cachePartly sequential again
Families of diffusion language models

Block diffusion, also called semi-autoregressive diffusion, generates text in chunks of perhaps 16 to 256 tokens. Within a block, tokens are denoised in parallel; across blocks, the model proceeds left to right. This hybrid restores variable-length output and allows cached key and value states for finished blocks, at the price of partly reintroducing sequential dependence.

Diffusion compared with autoregressive language models

The essential difference is the unit of decision per forward pass. An autoregressive LLM produces exactly one token per pass and can never change it afterward, while a diffusion model can fix several tokens per pass and sees both earlier and later context. That makes latency depend on step count rather than on output length.

The comparison below covers the properties that most often matter when choosing between the two paradigms.

PropertyDiffusion language modelAutoregressive model
Generation orderAny order, parallel per stepStrictly left to right
Passes for N tokensSet by the step budgetN sequential passes
Context usedBidirectionalOnly preceding tokens
Training objectiveVariational bound on likelihoodExact next-token likelihood
Key-value cachingLimited without block structureMature and efficient
Infilling and editingNativeNeeds special training formats
Diffusion and autoregressive generation side by side

The likelihood gap deserves attention. A masked diffusion model is trained on an evidence lower bound, so its perplexity is an upper estimate, and at equal compute it has historically trailed autoregressive models of the same size. Recent scaling work has narrowed this gap considerably, but parity is task dependent and remains an open research question.

Sampling steps, speed and quality

The number of denoising steps is the central control for the speed and quality trade-off. Committing many tokens in a single step is fast but risks incoherent combinations, because positions decoded simultaneously are sampled independently given the current context. Using one step per token removes that problem but also removes the speed advantage.

The following table shows how step budgets typically behave for a fixed output length.

Step budgetTokens committed per stepTypical effect
Equal to output lengthAbout oneHighest quality, no speedup
Half of output lengthAbout twoSmall quality loss, faster
A quarter or lessFour or moreNoticeable errors in structured text
Adaptive by confidenceVaries per stepGood balance on easy passages
Effect of the step budget on a fixed-length output

Parallel decoding pays off most on predictable spans such as boilerplate code or formulaic phrasing, where many tokens are nearly certain at once. Reasoning-heavy passages tend to need more steps, so adaptive schedules that unmask by confidence often beat fixed schedules. Temperature and top-p or top-k style filtering still apply to each position, but their interaction with remasking differs from autoregressive decoding.

Limitations and practical considerations

Diffusion language models remain less mature in production than autoregressive systems. Serving stacks, quantization methods and speculative decoding were built around left-to-right generation, so real-world throughput gains are often smaller than theoretical step counts suggest. Practitioners should evaluate end-to-end latency on their own hardware rather than rely on step counts.

  • Fixed output length in pure masked diffusion forces the model to pad or truncate, so answers may end abruptly or contain filler end tokens.
  • Bidirectional attention prevents simple key-value caching, which makes each step more expensive than one autoregressive step.
  • Instruction tuning and alignment methods such as preference optimization need adaptation, because exact sequence likelihoods are not available.
  • Hallucination is not solved by parallel refinement; the model can still produce fluent but false statements.

Where they fit well, diffusion models offer real advantages: native infilling for code completion in the middle of a file, global constraints such as fixed formats, and the ability to edit a draft rather than regenerate it from scratch.

The simple explanation

Most chatbots write like a person typing: one word after another, never going back. A diffusion language model works more like filling in a crossword. It starts with a row of blank squares, fills in the words it is most sure about first, and then uses those to work out the harder gaps.

Because it can fill several squares in each round, it can be much faster on easy text. The catch is that filling too many at once can produce words that do not fit together, so there is always a balance between speed and accuracy.

How it is used in practice

A software team adds an assistant to its code editor that completes a function body between an existing signature and an existing return statement. An autoregressive model sees only the code above the cursor, but a diffusion language model is given both the prefix and the suffix and fills the masked middle, so variable names and types agree with the code that follows.

The team tests outputs of about 128 tokens and finds that 32 denoising steps produce code that passes its unit tests about as often as 128 steps, with roughly three times lower latency on its GPU servers. For longer explanatory comments, quality drops at low step counts, so the team keeps an adaptive schedule that adds steps when confidence falls. The result is a completion feature that feels instant for routine code while preserving accuracy where it matters.