Interleaved image-text data

Also known as: interleaved multimodal data

On this page 7

Interleaved image-text data is training data in which images and text passages alternate in their natural document order, as they do on web pages, in articles, tutorials and books. In multimodal AI, it complements simple image-caption pairs by teaching a model how several images and long stretches of text relate across one coherent document.

Datasets of this kind, also called interleaved multimodal data, became central to artificial intelligence research when models such as Flamingo showed that training on them enables multimodal in-context learning: the ability to follow a few image-text examples in a prompt and generalize to a new image without any fine-tuning.

How interleaved image-text data is structured

An interleaved sample is a single document serialized as a sequence of text segments and image slots. Each image is replaced in the token stream by a placeholder token, such as an image marker, which the training pipeline later fills with visual tokens produced by a vision encoder and a multimodal projector. The model therefore sees text before an image, the image itself and text after it, in the order the author intended.

This ordering carries information that captions alone cannot provide. Text that precedes an image often sets up an expectation, while text that follows it frequently explains, compares or refers back to it. A recipe page, for instance, pairs each step photograph with the instruction above it, and a product review compares two photographs in the paragraph that follows them.

The following table lists the main components of a typical interleaved training sample and the role each one plays.

ComponentTypical formRole in training
Text segmentsParagraphs, headings, list itemsLanguage modeling targets and context
Image slotsPlaceholder token per imageMark where visual tokens are inserted
Visual tokens64 to several hundred per imageCarry encoded image content
Document boundariesStart and end markersPrevent attention across unrelated documents
MetadataSource URL hash, alt text, filtersSupport deduplication and quality filtering
Components of an interleaved training sample

How interleaved corpora are built from web documents

Most large interleaved corpora are extracted from Common Crawl snapshots. The pipeline parses each HTML page, keeps the main content, downloads the referenced images and records their position relative to the surrounding text. Two broad strategies exist: preserving the original DOM order of the page, or re-assigning images to the sentences they match best by CLIP similarity scores.

Filtering is the expensive part. Pipelines remove boilerplate, advertising images, logos and tracking pixels, apply text-quality heuristics, drop documents with too many or too few images, detect near-duplicate images, and screen for unsafe or personal content. Many pipelines also honor opt-out signals from site owners and remove images that appear on known blocklists.

The table below compares several well-known public interleaved datasets. Figures are approximate and depend on the released version.

DatasetApproximate scaleNotable design choice
M3WAbout 43 million web pagesProprietary corpus used to train Flamingo
MMC4About 100 million documents, 570 million imagesImages placed by CLIP-based bipartite matching
OBELICSAbout 140 million documents, 350 million imagesOriginal web page order preserved
MINT-1TAbout 1 trillion text tokens, 3 billion imagesAdds PDF and ArXiv documents to HTML
OmniCorpusBillions of images, over a trillion tokensMultilingual sources beyond English
Representative interleaved image-text datasets

How it compares with image-text pair datasets

Interleaved data differs from an image-text pair dataset such as LAION-5B in granularity and context length. A pair dataset holds one image and one short caption per sample, which suits contrastive training of models like CLIP and SigLIP and strengthens zero-shot image classification. An interleaved document can hold dozens of images and thousands of words, which suits autoregressive multimodal large language models.

The comparison below summarizes the practical differences that matter when designing a pretraining mixture.

PropertyImage-text pairsInterleaved documents
Images per sampleOneSeveral, often up to dozens
Text lengthShort caption, often under 30 wordsFull paragraphs and sections
Alignment signalStrong, directWeaker, contextual
Typical objectiveContrastive or captioning lossNext-token prediction over text
Main capability gainedZero-shot recognitionFew-shot and multi-image reasoning
Main weaknessNo document contextNoisy and loose image-text links
Interleaved data versus paired caption data

Ablation studies on multimodal pretraining have repeatedly found that the two sources are complementary rather than interchangeable. Caption data improves zero-shot captioning and recognition, while interleaved data improves few-shot performance and helps preserve text-only abilities, because documents contain long, fluent passages of ordinary language.

Why interleaved data enables multimodal in-context learning

A model learns in-context behavior from data that already looks like a prompt with examples. Web documents naturally contain patterns such as image, description, image, description, which resemble a few-shot prompt. Training on millions of such sequences teaches the model to condition on earlier image-text examples when processing a later image.

Architectures handle the interleaving differently. Flamingo and its open reproductions use gated cross-attention layers, where each text token attends only to the image that immediately precedes it, while the language model carries information from earlier images through its own self-attention. Models with early fusion or native multimodality instead place visual tokens directly in the shared sequence, so every token can attend to every earlier image.

The training loss is usually computed only on text tokens, because predicting image patches is not the goal for an understanding model. Any-to-any models that also generate images may add a loss on discrete image tokens produced by an image tokenizer, which turns interleaved documents into a source for interleaved generation as well.

Quality problems and common mistakes

Interleaved corpora are noisier than curated caption sets, and most failures trace back to weak links between an image and its neighboring text. The table below lists frequent problems, their causes and established mitigations.

ProblemCauseMitigation
Irrelevant imagesAds, logos and decorative banners keptSize, aspect ratio and position filters
Misplaced imagesLayout lost during HTML parsingPreserve DOM order or verify with CLIP scores
Duplicate contentTemplates repeated across sitesImage hashing and document deduplication
Context overflowToo many images per documentCap images per sample or split documents
Harmful contentUnfiltered web crawlSafety classifiers and blocklists
Degraded text skillsMixture dominated by short captionsKeep a share of interleaved and text-only data
Typical issues in interleaved training data

A common misconception is that interleaved data replaces visual instruction tuning. It does not: pretraining on documents builds broad knowledge and few-shot ability, but conversational behavior, reliable visual grounding and resistance to multimodal hallucination still require curated instruction data and evaluation.

In everyday language

Most early image models learned from photos with short labels, like flash cards. Interleaved image-text data is more like giving a model whole magazine articles, where pictures sit between paragraphs and the words around each picture explain it.

Reading full documents this way teaches a model to connect several pictures with a longer story. That is why it can later look at a few examples in a question and work out what to do with a new picture.

Seen in the real world

A research team training an open vision-language model with about 8 billion parameters builds its pretraining mixture from three sources: roughly 45 percent image-caption pairs, 45 percent interleaved web documents and 10 percent text-only data. The interleaved portion comes from filtered web pages such as cooking guides, travel blogs and product comparisons, capped at 10 images per document.

When the team evaluates the model, a version trained without the interleaved share answers single-image questions well but performs poorly when a prompt contains four example images with answers followed by a new image. After restoring the interleaved documents, few-shot accuracy on visual question answering improves noticeably, and the model can compare two product photos described in the same prompt. The team concludes that document-style data is essential for multi-image reasoning, while captions remain the stronger source for plain recognition.