Score-based generative model

On this page 7

A score-based generative model is a generative model that learns the score function, the gradient of the log probability density of the data, and then generates new samples by repeatedly moving random noise in the direction of higher data density. In generative AI, this family provides the mathematical foundation behind modern diffusion models used for images, video, audio and molecular design.

Instead of estimating the probability density itself, which requires an intractable normalizing constant, the model estimates only where probability increases. That shift makes training stable and avoids the adversarial game of a GAN as well as the restricted architectures of a normalizing flow.

What the score function represents

The score of a distribution at a point is the vector that points toward the steepest increase of log density. For an image, it indicates how every pixel value should change to make the image slightly more like real data. Because the gradient of a logarithm cancels any constant factor, the score is independent of the normalizing constant, which is the main reason this formulation is tractable.

A neural network called the score network approximates this vector field. In practice it is a U-Net or a transformer that takes a noisy input and a noise level and returns a tensor of the same shape as the input.

QuantityMeaningRole in the model
Data densityProbability of a data pointNever computed directly
ScoreGradient of log densityLearned target of the network
Noise levelStandard deviation of added noiseConditions the network
Score networkNeural approximation of the scoreGuides every sampling step
SamplerNumerical procedure using the scoreTurns noise into samples
Core quantities in score-based modeling

How score matching trains the network

The true score of real data is unknown, so training relies on denoising score matching. Clean training samples are corrupted with Gaussian noise of a known scale, and the network learns to predict the score of the corrupted distribution, which is proportional to the added noise and therefore directly available as a regression target.

A single noise scale is not enough. In low-density regions far from the data, a network trained only on small perturbations receives almost no training signal and its score estimates are unreliable. The solution, introduced in noise-conditional score networks, is to train across a ladder of noise levels from very large to very small, so that heavily noised samples fill the empty space and give the sampler a reliable path toward the data.

Each noise level is weighted in the loss so that all scales contribute comparably. The choice of this weighting and the noise schedule strongly affects final sample quality and remains an active design question.

Sampling with Langevin dynamics and stochastic differential equations

Generation reverses the corruption process. The sampler starts from pure Gaussian noise and alternates small steps along the learned score with small injections of fresh noise, a procedure known as annealed Langevin dynamics, gradually lowering the noise level until a clean sample emerges.

The continuous-time view describes corruption as a forward stochastic differential equation (SDE) and generation as the corresponding reverse-time SDE, which depends only on the score. The same framework yields a deterministic probability flow ODE with identical marginal distributions, enabling exact likelihood computation and faster solvers.

SamplerTypeTypical stepsCharacteristics
Annealed Langevin dynamicsStochasticHundreds to thousandsSimple, slow, original method
Reverse SDE solverStochasticHundredsHigh diversity, corrects errors
Predictor-correctorStochasticHundreds, doubled costCombines SDE step with Langevin refinement
Probability flow ODEDeterministicTens to hundredsReproducible, enables likelihoods
Distilled samplersDeterministicOne to eightFast, slight quality loss
Main sampling approaches for score-based models

Relationship to diffusion and other generative models

Score-based models and denoising diffusion probabilistic models were developed separately but are now understood as two views of one framework. Predicting the added noise, as DDPM does, is mathematically equivalent to predicting a scaled score, and the variance-preserving SDE recovers the DDPM forward process.

Model familyWhat it learnsTraining stabilitySampling speed
Score-based modelGradient of log densityHighSlow without acceleration
DDPMNoise added at each stepHighSlow without acceleration
GANGenerator fooling a discriminatorProne to mode collapseOne forward pass
Variational autoencoderEncoder and decoder with latent priorHighOne forward pass
Normalizing flowInvertible mapping to a simple densityHighOne pass, constrained architecture
Flow matchingVelocity field between noise and dataHighOften fewer steps
Score-based models compared with neighboring approaches

The score perspective explains why diffusion systems achieve broad mode coverage: the training objective rewards covering every region of the data distribution rather than finding a few convincing outputs, which is the typical weakness of adversarial training.

Strengths, limitations and practical considerations

The main strength of the approach is sample quality combined with diversity, plus a flexible mechanism for conditioning. Adding the gradient of a classifier or using classifier-free guidance steers generation toward a prompt without retraining the score network.

IssueCauseTypical remedy
Slow generationMany small sampling stepsODE solvers, distillation, consistency models
Poor fine detailInaccurate score at small noiseBetter weighting, more training
Color or brightness driftDiscretization error over long trajectoriesCorrector steps, tuned schedules
High memory and computeWork in full pixel spaceLatent diffusion in a compressed space
Memorization of training dataOverfitting on small or duplicated datasetsDeduplication, larger data, auditing
Common issues and their usual remedies

A common misconception is that the model stores images and retrieves them. It stores a learned vector field; nevertheless, research has shown that near-copies of training images can appear when data is duplicated, which matters for copyright and privacy reviews.

Stripped to the essentials

Imagine a photo covered in heavy static. A score-based generative model has learned, from millions of examples, which small change makes a noisy picture look a little more real. It does not memorize pictures; it learns a sense of direction.

To create something new, it starts with pure static and takes hundreds of tiny steps in that direction. Each step removes a bit of noise, and a clear, original image slowly appears, much like a figure emerging from fog as one walks toward it.

Where it is applied

A materials research group wants new candidate crystal structures for battery electrodes. Testing each idea in the laboratory takes weeks, so the group trains a score-based model on roughly 100,000 known stable crystals, representing each one by atom types, positions and lattice parameters.

During sampling, the researchers add a guidance term from a separate property predictor that favors high lithium-ion mobility. The model proposes several thousand structures in a day. Physics simulations filter them to a few dozen plausible candidates, and only those reach the laboratory, cutting the experimental search space by orders of magnitude while keeping the final judgment with domain experts.