Variational autoencoder

Also known as: VAE

On this page 7

A variational autoencoder (VAE) is a generative model that learns to compress data into a probabilistic latent space and to reconstruct data from points in that space, so that sampling the latent space produces new, plausible examples. In generative AI, the VAE is one of the foundational model families alongside the GAN, normalizing flows and the diffusion model.

Introduced in 2013, the VAE combines a neural encoder and decoder with variational inference, a statistical method for approximating intractable probability distributions. Today VAEs are rarely used as standalone image generators, but they remain essential as the compression stage of latent diffusion models, where they translate between pixels and the compact latent representation in which diffusion actually runs.

How a variational autoencoder encodes and decodes data

A VAE works by mapping each input not to a single point but to a probability distribution over latent codes, then decoding a sample from that distribution back into data space. The encoder outputs a mean vector and a variance vector, which together describe a Gaussian distribution for that input.

A latent vector is then drawn from this Gaussian using the reparameterization trick: the model samples random noise from a standard normal distribution, multiplies it by the predicted standard deviation and adds the mean. This makes the sampling step differentiable, so the whole network can be trained with ordinary backpropagation. The decoder finally turns the latent vector into a reconstruction of the input.

The table below lists the main components of a VAE and the role each one plays during training and generation.

ComponentFunctionUsed during generation
Encoder networkPredicts mean and variance of the latent distributionNo, only for encoding inputs
Latent spaceLow-dimensional continuous representationYes, source of samples
Reparameterization stepMakes sampling differentiableReplaced by direct sampling
Decoder networkMaps latent vectors back to dataYes, produces the output
Prior distributionUsually a standard normal GaussianYes, defines where to sample
Main components of a variational autoencoder

The VAE training objective

A VAE is trained to maximize the evidence lower bound (ELBO), a tractable lower bound on the log-likelihood of the data. The ELBO has two terms that pull in different directions, and their balance determines the character of the model.

The reconstruction term rewards the decoder for reproducing the input accurately. The KL divergence term penalizes the encoder when its distributions drift away from the prior, which keeps the latent space smooth, continuous and centered. Without the KL term, the model collapses into a plain autoencoder whose latent space has gaps that decode into nonsense.

TermWhat it measuresEffect if too strong
Reconstruction lossFidelity of decoded output to inputIrregular latent space, poor sampling
KL divergenceDistance of encoder distribution from priorBlurry outputs, ignored latents
Beta weightRelative scale of the KL termHigh beta favors disentanglement over detail
Terms of the ELBO and their effects

The beta-VAE variant multiplies the KL term by a coefficient, usually written as beta. Values above 1 encourage disentangled latent factors, where single dimensions control interpretable properties such as rotation or lighting, at the price of reconstruction quality.

VAE variants used in generative AI

Several VAE variants address the weaknesses of the original formulation, especially blurry reconstructions and limited sample quality. The following table summarizes the variants most relevant to current generative systems.

VariantKey changeTypical use
Conditional VAEEncoder and decoder receive a label or conditionClass-controlled generation
Beta-VAEWeighted KL termDisentangled representation learning
VQ-VAEDiscrete codebook instead of continuous GaussiansTokenizing images and audio
Hierarchical VAEMultiple stacked latent layersHigher-fidelity image synthesis
KL-regularized autoencoderVery small KL weight plus perceptual and adversarial lossesLatent space for diffusion models
Common VAE variants

The last row describes the component often called simply the VAE in Stable Diffusion. It compresses a 512 by 512 pixel image into a latent grid of 64 by 64 with 4 channels, an 8 times spatial downsampling. The diffusion process denoises this latent, and the VAE decoder converts the result back into pixels. Newer image models use latents with 16 or more channels to preserve fine detail such as small text.

VAE compared with other generative models

Compared with other generative models, a VAE offers fast sampling, stable training and a meaningful latent space, but lower sample sharpness than adversarial or diffusion approaches. The comparison below highlights the trade-offs.

Model familyTraining stabilitySample sharpnessSampling speed
Variational autoencoderHighModerate, often blurryOne decoder pass
GANLow, prone to mode collapseHighOne generator pass
Diffusion modelHighVery highMany denoising steps
Normalizing flowHighModerate to highOne invertible pass
VAE versus neighboring generative model families

Blurriness arises because the usual pixel-wise reconstruction loss averages over plausible outputs, and the Gaussian decoder assumption treats fine texture as noise. Modern systems therefore add perceptual and adversarial losses to the VAE when sharp detail matters.

Limitations and practical pitfalls

The best-known failure mode is posterior collapse, in which a powerful decoder learns to ignore the latent code and the KL term drops to near zero. It is common when VAEs are paired with autoregressive decoders for text, and it is mitigated with KL annealing, free-bits constraints or weaker decoders.

  • Latent artifacts: a poorly trained VAE in a latent diffusion pipeline causes washed-out colors or distorted faces even when the diffusion model is good.
  • Compression ceiling: whatever the VAE cannot reconstruct, such as tiny text, cannot appear in generated images.
  • Mismatched components: pairing a diffusion model with a different VAE than the one it was trained on produces color shifts and noise.
  • Likelihood is not quality: a good ELBO does not guarantee samples that look convincing to people.

The short version

A variational autoencoder is a model that learns to shrink data into a compact code and rebuild it. It does not store one exact code for each picture. Instead, it learns a fuzzy region of codes that all decode into something similar.

Because those regions are smooth and connected, you can pick a new point and decode it. The result is a new, believable example. Think of a map where nearby spots describe similar faces. Moving across the map changes the face gradually.

Where it makes a difference

A design studio uses a latent diffusion model to generate product mockups that include small printed labels and logos. The team notices that fine lettering always comes out smeared, even after more sampling steps and stronger guidance. A technical artist decodes a few real photos through the VAE alone, without any diffusion, and sees the same smearing.

The test shows that the bottleneck is the VAE, not the diffusion model. The studio switches to a model whose VAE uses a 16-channel latent space and keeps a short reference set of label photos to check reconstruction quality first. Legible text at typical label sizes became achievable, and the team now tests the VAE whenever it evaluates a new image model.