Interleaved image-text data is training data in which images and text passages alternate in their natural document order, as they do on web pages, in articles, tutorials and books. In multimodal AI, it complements simple image-caption pairs by teaching a model how several images and long stretches of text relate across one coherent document.
Datasets of this kind, also called interleaved multimodal data, became central to artificial intelligence research when models such as Flamingo showed that training on them enables multimodal in-context learning: the ability to follow a few image-text examples in a prompt and generalize to a new image without any fine-tuning.
How interleaved image-text data is structured
An interleaved sample is a single document serialized as a sequence of text segments and image slots. Each image is replaced in the token stream by a placeholder token, such as an image marker, which the training pipeline later fills with visual tokens produced by a vision encoder and a multimodal projector. The model therefore sees text before an image, the image itself and text after it, in the order the author intended.
This ordering carries information that captions alone cannot provide. Text that precedes an image often sets up an expectation, while text that follows it frequently explains, compares or refers back to it. A recipe page, for instance, pairs each step photograph with the instruction above it, and a product review compares two photographs in the paragraph that follows them.
The following table lists the main components of a typical interleaved training sample and the role each one plays.
| Component | Typical form | Role in training |
|---|---|---|
| Text segments | Paragraphs, headings, list items | Language modeling targets and context |
| Image slots | Placeholder token per image | Mark where visual tokens are inserted |
| Visual tokens | 64 to several hundred per image | Carry encoded image content |
| Document boundaries | Start and end markers | Prevent attention across unrelated documents |
| Metadata | Source URL hash, alt text, filters | Support deduplication and quality filtering |
How interleaved corpora are built from web documents
Most large interleaved corpora are extracted from Common Crawl snapshots. The pipeline parses each HTML page, keeps the main content, downloads the referenced images and records their position relative to the surrounding text. Two broad strategies exist: preserving the original DOM order of the page, or re-assigning images to the sentences they match best by CLIP similarity scores.
Filtering is the expensive part. Pipelines remove boilerplate, advertising images, logos and tracking pixels, apply text-quality heuristics, drop documents with too many or too few images, detect near-duplicate images, and screen for unsafe or personal content. Many pipelines also honor opt-out signals from site owners and remove images that appear on known blocklists.
The table below compares several well-known public interleaved datasets. Figures are approximate and depend on the released version.
| Dataset | Approximate scale | Notable design choice |
|---|---|---|
| M3W | About 43 million web pages | Proprietary corpus used to train Flamingo |
| MMC4 | About 100 million documents, 570 million images | Images placed by CLIP-based bipartite matching |
| OBELICS | About 140 million documents, 350 million images | Original web page order preserved |
| MINT-1T | About 1 trillion text tokens, 3 billion images | Adds PDF and ArXiv documents to HTML |
| OmniCorpus | Billions of images, over a trillion tokens | Multilingual sources beyond English |
How it compares with image-text pair datasets
Interleaved data differs from an image-text pair dataset such as LAION-5B in granularity and context length. A pair dataset holds one image and one short caption per sample, which suits contrastive training of models like CLIP and SigLIP and strengthens zero-shot image classification. An interleaved document can hold dozens of images and thousands of words, which suits autoregressive multimodal large language models.
The comparison below summarizes the practical differences that matter when designing a pretraining mixture.
| Property | Image-text pairs | Interleaved documents |
|---|---|---|
| Images per sample | One | Several, often up to dozens |
| Text length | Short caption, often under 30 words | Full paragraphs and sections |
| Alignment signal | Strong, direct | Weaker, contextual |
| Typical objective | Contrastive or captioning loss | Next-token prediction over text |
| Main capability gained | Zero-shot recognition | Few-shot and multi-image reasoning |
| Main weakness | No document context | Noisy and loose image-text links |
Ablation studies on multimodal pretraining have repeatedly found that the two sources are complementary rather than interchangeable. Caption data improves zero-shot captioning and recognition, while interleaved data improves few-shot performance and helps preserve text-only abilities, because documents contain long, fluent passages of ordinary language.
Why interleaved data enables multimodal in-context learning
A model learns in-context behavior from data that already looks like a prompt with examples. Web documents naturally contain patterns such as image, description, image, description, which resemble a few-shot prompt. Training on millions of such sequences teaches the model to condition on earlier image-text examples when processing a later image.
Architectures handle the interleaving differently. Flamingo and its open reproductions use gated cross-attention layers, where each text token attends only to the image that immediately precedes it, while the language model carries information from earlier images through its own self-attention. Models with early fusion or native multimodality instead place visual tokens directly in the shared sequence, so every token can attend to every earlier image.
The training loss is usually computed only on text tokens, because predicting image patches is not the goal for an understanding model. Any-to-any models that also generate images may add a loss on discrete image tokens produced by an image tokenizer, which turns interleaved documents into a source for interleaved generation as well.
Quality problems and common mistakes
Interleaved corpora are noisier than curated caption sets, and most failures trace back to weak links between an image and its neighboring text. The table below lists frequent problems, their causes and established mitigations.
| Problem | Cause | Mitigation |
|---|---|---|
| Irrelevant images | Ads, logos and decorative banners kept | Size, aspect ratio and position filters |
| Misplaced images | Layout lost during HTML parsing | Preserve DOM order or verify with CLIP scores |
| Duplicate content | Templates repeated across sites | Image hashing and document deduplication |
| Context overflow | Too many images per document | Cap images per sample or split documents |
| Harmful content | Unfiltered web crawl | Safety classifiers and blocklists |
| Degraded text skills | Mixture dominated by short captions | Keep a share of interleaved and text-only data |
A common misconception is that interleaved data replaces visual instruction tuning. It does not: pretraining on documents builds broad knowledge and few-shot ability, but conversational behavior, reliable visual grounding and resistance to multimodal hallucination still require curated instruction data and evaluation.
In everyday language
Most early image models learned from photos with short labels, like flash cards. Interleaved image-text data is more like giving a model whole magazine articles, where pictures sit between paragraphs and the words around each picture explain it.
Reading full documents this way teaches a model to connect several pictures with a longer story. That is why it can later look at a few examples in a question and work out what to do with a new picture.
Seen in the real world
A research team training an open vision-language model with about 8 billion parameters builds its pretraining mixture from three sources: roughly 45 percent image-caption pairs, 45 percent interleaved web documents and 10 percent text-only data. The interleaved portion comes from filtered web pages such as cooking guides, travel blogs and product comparisons, capped at 10 images per document.
When the team evaluates the model, a version trained without the interleaved share answers single-image questions well but performs poorly when a prompt contains four example images with answers followed by a new image. After restoring the interleaved documents, few-shot accuracy on visual question answering improves noticeably, and the model can compare two product photos described in the same prompt. The team concludes that document-style data is essential for multi-image reasoning, while captions remain the stronger source for plain recognition.