Autoregressive image generation

On this page 7

Autoregressive image generation is a generative AI approach that represents an image as a sequence of discrete tokens and produces it one token at a time, with each new token predicted from all previous tokens and from a conditioning prompt. It applies the same next-token prediction that drives large language models to pictures instead of words.

Within generative AI and artificial intelligence more broadly, it is the main alternative to the diffusion model. Systems such as the original DALL·E, Parti and several recent natively multimodal models use this approach, because a single transformer can then read and write both text and images in one shared token stream.

How autoregressive image generation builds a picture

Autoregressive image generation works in two stages: an image tokenizer converts pixels into discrete codes, and a transformer learns the probability of each code given the codes before it. At inference time the transformer samples codes in order, and the tokenizer’s decoder turns the finished code sequence back into pixels.

The tokenizer is usually a VQ-VAE or VQ-GAN, a vector-quantized autoencoder that compresses an image patch into the index of the nearest entry in a learned codebook. A 256 by 256 image with a downsampling factor of 16 becomes a 16 by 16 grid of 256 tokens, and a codebook of 8,192 to 16,384 entries is common.

The transformer then uses causal attention: when predicting token number 100, it may look at tokens 1 to 99 and at the text prompt, but never at later tokens. This restriction is what makes the model autoregressive and allows exact likelihood training with ordinary cross-entropy loss.

The table below summarizes the pipeline stages and what each one contributes.

StageTypical componentOutput
Image tokenizerVQ-VAE or VQ-GAN encoderGrid of codebook indices
Text encoderLanguage model tokenizer or encoderPrompt tokens
Sequence modelDecoder-only transformerNext image token probabilities
SamplingTop-k, top-p, guidanceOne chosen token per step
Image decoderVQ-GAN decoderReconstructed pixels
Stages of an autoregressive image pipeline

Token orders and model variants

The generation order strongly affects quality and speed. Early systems used a raster scan, left to right and top to bottom, which is simple but forces the model to commit to the top of a picture before it knows the bottom. Newer variants change the order or the unit of prediction to reduce this weakness.

The following table compares the main variants found in research and production.

VariantWhat is predicted per stepKey trade-off
Raster-scan AROne token in reading orderSimple, but slow and order-biased
Masked parallel decodingSeveral masked tokens at onceFewer steps, not strictly causal
Next-scale predictionA whole coarser-to-finer token mapGlobal layout first, faster sampling
Continuous-token ARReal-valued latent vectorsAvoids quantization loss, harder sampling
Unified multimodal ARInterleaved text and image tokensOne model for reading and drawing
Main variants of autoregressive image models

Masked approaches such as MaskGIT are sometimes grouped with this family, although they predict many tokens in parallel and are therefore only partly autoregressive. Next-scale prediction, introduced by visual autoregressive modeling, generates a low-resolution token map first and refines it scale by scale, which mirrors how artists sketch before adding detail.

Autoregressive versus diffusion and GAN approaches

Autoregressive models, diffusion models and GANs solve the same task with different mathematical tools. Diffusion models iteratively remove noise from a whole image, GANs generate an image in one pass guided by a discriminator, and autoregressive models factorize the image into a chain of conditional token probabilities.

MethodGeneration processStrengthWeakness
AutoregressiveSequential token samplingShares architecture with LLMsMany sequential steps
Diffusion modelIterative denoising of the whole imageHigh fidelity and diversityNeeds a separate text encoder
GANSingle forward passVery fast samplingMode collapse, unstable training
Comparison with other generative image methods

The practical consequence is that unified multimodal assistants favor autoregressive designs, because one transformer can follow instructions, reason about an image and then draw it. Diffusion remains common in dedicated text-to-image tools where raw image fidelity matters most, and hybrid systems combine an autoregressive planner with a diffusion decoder.

Sampling controls and practical settings

The output of an autoregressive image model depends heavily on decoding settings, just as with a language model. Because each image token is sampled from a probability distribution, the same prompt and the same random seed reproduce the same image, while changing the seed explores alternatives.

  • Temperature around 0.8 to 1.0 balances variety and coherence; lower values produce flatter, repetitive textures.
  • Top-k or top-p filtering removes unlikely codes that cause visual artifacts.
  • Classifier-free guidance, adapted from diffusion, sharpens prompt adherence at a cost in diversity.
  • KV caching stores attention keys and values so each new token does not recompute the full sequence.

Resolution drives cost directly. Doubling the image side quadruples the token count, so a 1024 by 1024 image at a downsampling factor of 16 needs 4,096 tokens and thousands of sequential steps unless parallel or multi-scale decoding is used.

Limitations and failure modes

The main limitation is sequential latency: tokens are produced one after another, so generation time grows with resolution. Quality is also capped by the tokenizer’s reconstruction fidelity, since fine textures, small text and faces can be lost when patches are forced into a finite codebook.

The table below lists common symptoms, their causes and typical remedies.

SymptomLikely causeTypical fix
Blurry fine detailLossy codebook quantizationLarger codebook or continuous tokens
Inconsistent lower halfEarly raster errors accumulateMulti-scale or masked decoding
Repeated texturesTemperature too lowRaise temperature or use top-p
Ignored prompt detailsWeak conditioning signalClassifier-free guidance
Slow generationThousands of sequential stepsKV caching, speculative decoding
Common problems and fixes

Exposure bias is another known issue: during training the model always sees correct previous tokens, but during sampling it sees its own mistakes, which can compound into visible artifacts later in the sequence.

Explained without jargon

Autoregressive image generation paints a picture piece by piece. The computer first learns a vocabulary of small image pieces, much like a set of mosaic tiles. To create a new image, it picks the first tile, then chooses each next tile based on the tiles already placed and on the description it was given.

This is the same trick chatbots use to write sentences word by word, applied to pictures. The benefit is that one system can understand text and draw images together. The cost is speed, because a large picture needs thousands of small decisions made in order.

A typical use case

An online furniture retailer wants product illustrations for a catalog of about 3,000 items, each in several room styles. The design team uses a multimodal assistant built on autoregressive image generation, so a merchandiser can type a request such as a walnut side table in a bright Scandinavian living room, review the result, and then ask in the same conversation for a darker floor or a different camera angle.

Because the same model reads the text and produces the image tokens, follow-up edits keep the context of the earlier request. The team generates images at 1024 by 1024 pixels with a fixed seed per product for reproducibility, and a human reviewer checks dimensions and proportions. The workflow cuts first-draft illustration time from several days to under an hour per batch, while final hero images are still checked and retouched by designers.