Multimodal pretraining

Also known as: vision-language pretraining

On this page 8

Multimodal pretraining is the large-scale, mostly self-supervised training stage in which a model learns from several kinds of data at once, typically images and text, so that it builds shared representations across modalities before any task-specific tuning. In Multimodal AI, it is the foundation on which vision-language models, cross-modal retrieval systems and multimodal assistants are built.

The stage is also called vision-language pretraining when the modalities are images and text. Its output is not a finished product but a general-purpose backbone: an encoder, a pair of aligned encoders or a generative model that later stages adapt through fine-tuning, visual instruction tuning or zero-shot prompting.

How multimodal pretraining builds shared representations

Multimodal pretraining works by giving the model naturally co-occurring signals, such as an image and its alt text or a video and its audio track, and asking it to predict or match one from the other. Because the pairing comes from the web or from recordings rather than from human annotators, the data scales to hundreds of millions or billions of examples.

A typical pipeline contains three parts. A modality encoder per input type, such as a vision transformer for images, converts raw data into vectors. An alignment or fusion mechanism relates those vectors, either by projecting them into a common embedding space or by letting tokens attend to each other. A pretraining objective turns the co-occurrence into a loss that gradient descent can minimize.

The model never receives explicit labels such as “dog” or “car”. Instead, it learns that the pixels of a dog and the word “dog” tend to appear together, and this weak supervision from co-occurrence produces features that transfer to classification, retrieval and question answering.

Pretraining objectives and what each one teaches

The choice of objective decides what the pretrained model is good at. Contrastive objectives produce strong retrieval and zero-shot classification, while generative objectives produce models that can describe, answer and reason in text. Many modern recipes combine two or three of them.

ObjectiveWhat the model predictsMain strengthRepresentative work
Image-text contrastive (ITC)Which caption belongs to which image in a batchRetrieval, zero-shot classificationCLIP, ALIGN
Sigmoid pairwise lossWhether each single pair matchesEfficient at smaller batch sizesSigLIP
Image-text matching (ITM)A binary match label for a fused pairFine-grained alignmentALBEF, BLIP
Captioning or prefix language modelingText tokens conditioned on the imageDescription and question answeringSimVLM, CoCa
Masked multimodal modelingHidden patches or words from contextDense, fused representationsBEiT-3, FLAVA
Next-token prediction on interleaved dataThe next token in mixed image-text sequencesIn-context learning, dialogFlamingo, native multimodal models
Common objectives in multimodal pretraining

The contrastive loss in CLIP uses a symmetric cross-entropy over a batch similarity matrix, so every other pair in the batch acts as a negative example. This makes batch size a key hyperparameter: CLIP used batches of about 32,000 pairs. The sigmoid loss in SigLIP removes the need for a global softmax and works well with smaller batches.

Architectures used during pretraining

Architecture determines how much the modalities interact during pretraining. Dual-encoder models keep image and text separate until a final dot product, which makes retrieval fast because embeddings can be precomputed. Fusion encoders mix the modalities through cross-attention, which improves reasoning but makes every query more expensive.

ArchitectureInteraction between modalitiesInference costTypical use
Dual encoderOnly at the final similarity scoreLow, embeddings are cachedSearch, zero-shot labels
Fusion encoderDeep cross-attention in many layersHigh, per pairMatching, visual entailment
Encoder-decoderDecoder attends to visual featuresMediumCaptioning, question answering
Frozen LLM with adapterGated cross-attention or a projectorMedium to highMultimodal assistants
Single native transformerShared tokens from the first layerHighAny-to-any generation
Architectural families and their trade-offs

A common modern pattern is staged pretraining. First, a vision encoder is pretrained contrastively. Next, a multimodal projector is trained to map its outputs into the embedding space of a pretrained language model, often with both backbones frozen. Finally, the whole stack is trained further on interleaved data before instruction tuning.

Data for multimodal pretraining

Data quality usually matters as much as model size in multimodal pretraining. Web-scale corpora such as LAION-5B, with about 5.85 billion image-text pairs, offer coverage but contain noisy, mismatched or unsafe pairs. Curated or filtered subsets often produce better models with fewer samples.

Data typeTypical scaleWhat it contributesMain risk
Image-text pairsHundreds of millions to billionsBroad visual vocabularyNoisy or generic captions
Interleaved documentsTens to hundreds of millions of pagesContext, multi-image reasoningWeak image-text links
Video with transcriptsMillions of hoursTemporal and audio groundingSpeech unrelated to visuals
Synthetic captionsVaries with compute budgetDenser, more accurate descriptionsErrors of the captioning model
Curated annotated setsThousands to millionsPrecise grounding and evaluationHigh cost, narrow coverage
Data types and their role

Standard curation steps include CLIP-score filtering to drop pairs whose image and caption disagree, near-duplicate removal, language identification, safety and personal-data filtering, and rebalancing of concept frequencies so that rare concepts are not drowned out. Recaptioning web images with a stronger model has become a widespread way to raise text quality.

Multimodal pretraining is the broad, data-hungry stage; later stages are narrower and use far less data. Confusing them leads to wrong expectations about what a model can do out of the box.

  • Unimodal pretraining trains on one modality, for example text-only language modeling, and supplies backbones that multimodal pretraining may reuse.
  • Visual instruction tuning follows pretraining and teaches the model to follow requests about images with curated conversations.
  • Task fine-tuning adapts a pretrained model to a single benchmark or product task, such as document understanding.
  • Multimodal fusion describes how signals are combined, which is an architectural choice inside pretraining rather than a separate stage.

Limitations and failure modes

Pretrained multimodal models inherit the biases and gaps of web data, including skewed representation of people, languages and regions. Contrastive models are known to behave like a bag of words, struggling with word order, counting, negation and spatial relations such as left versus right.

Generative models pretrained on weakly aligned pairs are prone to multimodal hallucination, describing objects that are not present. Other practical risks are benchmark contamination when evaluation images appear in web crawls, copyright and consent questions around scraped data, and very high compute cost, often thousands of accelerator days for frontier-scale runs.

A frequent misconception is that more pairs always help. Beyond a certain point, noisy data yields diminishing returns, and smaller well-filtered datasets regularly match or beat larger unfiltered ones at equal compute.

The idea in brief

Multimodal pretraining is how an AI model first learns that pictures, words and sounds can describe the same thing. It looks at a huge number of images with their captions, or videos with their soundtracks, and slowly learns which ones belong together.

Think of a child who flips through millions of picture books without a teacher. Nobody points at each animal, yet the child eventually links the drawing of a cat with the word cat. The key point is that the model learns general connections first, and later training teaches it specific jobs such as answering questions about photos.

A practical scenario

An online furniture retailer wants shoppers to search its catalog of about 400,000 product photos by typing phrases such as “oak desk with two drawers”. The team has no labeled training data for this, so it starts from a dual-encoder model produced by contrastive multimodal pretraining on several hundred million web image-text pairs.

Engineers embed every catalog photo once with the image encoder and store the vectors in a vector index. At query time, only the short text is encoded, and the nearest image vectors are returned in a few milliseconds. Testing on 2,000 real queries shows that the zero-shot model already ranks a relevant product in the top ten for most of them.

To close remaining gaps, such as confusing walnut and oak finishes, the team continues contrastive training for a short period on 150,000 of its own photo-description pairs. The result is a working visual search feature built in weeks, because the expensive general learning had already been done during pretraining.