Multimodal chain-of-thought

Also known as: visual chain-of-thought

On this page 7

Multimodal chain-of-thought is a reasoning technique in which a model produces explicit intermediate steps that draw on several input modalities, typically images and text, before it commits to a final answer. Within Multimodal AI, it extends the text-only chain-of-thought idea to problems where the evidence sits in a diagram, a photograph, a chart, a screen or a video frame, and it is also called visual chain-of-thought when the non-text input is visual.

The method matters because many multimodal errors are perception errors disguised as reasoning errors. When a model must first state what it sees, then connect those observations to the question, and only then answer, both the quality of the answer and the auditability of the reasoning usually improve. The technique is now a standard component of multimodal large language models, benchmark evaluation and multimodal agents in artificial intelligence.

How multimodal chain-of-thought works

Multimodal chain-of-thought works by forcing a model to externalize the path from perception to conclusion as a sequence of written or structured steps. A typical chain begins with observation steps that describe relevant regions of an image, continues with inference steps that relate those observations to the question and to background knowledge, and ends with a final answer conditioned on the whole chain.

In a modern vision-language model, the image passes through a vision encoder and a multimodal projector, which turn it into visual tokens placed in the same context as the text. The language model then generates the rationale token by token, attending to visual tokens at every step. Because each generated sentence becomes part of the context, earlier observations condition later reasoning, which is the core mechanism shared with text-only chain-of-thought.

The following table lists the main stages that a well-formed multimodal chain contains and what each stage contributes.

StageWhat the model doesTypical failure if skipped
PerceptionDescribes objects, text, values or layout in the inputAnswer built on a misread detail
GroundingLinks each observation to a region, frame or timestampClaims that cannot be verified
InferenceCombines observations with rules or world knowledgeCorrect facts, wrong conclusion
VerificationRe-checks key observations against the inputErrors propagate unnoticed
AnswerStates the result in the required formatUnparseable or ambiguous output
Stages of a multimodal reasoning chain

Main variants of multimodal chain-of-thought

Multimodal chain-of-thought exists in several variants that differ in where the reasoning happens and what form the intermediate steps take. Early research systems used a two-stage framework: one pass generated a rationale from the image and question, and a second pass used that rationale to infer the answer. This separation showed that even models below one billion parameters could benefit when the rationale was grounded in visual features rather than produced from text alone.

Later approaches moved the whole process into a single multimodal large language model and diversified the format of the steps, as the table below summarizes.

VariantIntermediate stepsTypical use
Zero-shot promptingFree-text rationale triggered by an instructionQuick gains on general models
Two-stage rationaleSeparate rationale model and answer modelScience questions, smaller models
Grounded chainText steps with bounding boxes or pointsVisual grounding, counting, GUI tasks
Interleaved chainSteps that insert crops, zooms or sketchesHigh-resolution images, geometry
Tool-augmented chainCalls to OCR, detectors or codeDocuments, charts, measurements
Trained reasoning modelLong internal chains learned with reinforcement learningHard math, science and puzzles
Common variants and their intermediate representations

The most recent direction, often described as thinking with images, lets the model manipulate the image during reasoning, for example by cropping a region, rotating a scan or drawing auxiliary lines, and then reasoning over the modified view. This reduces dependence on a single fixed-resolution encoding.

Multimodal chain-of-thought is a reasoning strategy, not a model architecture, so it can run on top of many architectures and complements neighboring techniques rather than replacing them. The comparison below clarifies the boundaries with concepts that are often mentioned in the same breath.

ConceptPrimary goalRelation to multimodal chain-of-thought
Text-only chain-of-thoughtStepwise reasoning over textThe direct ancestor, without visual evidence
Multimodal reasoningAny inference across modalitiesThe broader capability that chains make explicit
Visual question answeringAnswering questions about imagesA task that chains often improve
Visual groundingLocating referred regionsA building block used inside grounded chains
Multimodal RAGRetrieving external images and textSupplies evidence that a chain then reasons over
Comparison with neighboring concepts

A frequent misconception is that a fluent rationale proves the model actually looked at the image. Rationales can be post-hoc: a model may write a plausible explanation that does not reflect the computation that produced the answer, which is a known faithfulness problem in all chain-of-thought methods.

Applying multimodal chain-of-thought effectively

Multimodal chain-of-thought is most useful when a task requires several dependent observations or a computation over visual data. For simple recognition, such as naming a single object, the extra tokens add latency and cost without improving accuracy, and long chains can even introduce new errors.

Practitioners usually decide whether and how to use it according to the task profile shown in the following decision table.

SituationRecommended approachReason
Single-object recognitionDirect answer, no chainReasoning adds cost but little accuracy
Chart or table readingChain that first extracts valuesSeparates reading errors from arithmetic errors
Geometry or physics diagramInterleaved chain with annotationsSpatial relations need explicit tracking
Screen or GUI automationGrounded chain with coordinatesActions must target exact elements
Long video questionChain anchored to timestampsTemporal evidence must be located first
Regulated decision supportChain plus human reviewRationale aids audit but is not proof
When to use multimodal chain-of-thought

Effective prompts ask the model to describe before it decides, to quote exact values it reads, and to keep the final answer in a fixed format. For trained reasoning models, the main control is usually a reasoning budget or effort setting rather than an explicit instruction to think step by step.

Limitations and common failure modes

The main limitation of multimodal chain-of-thought is that reasoning cannot repair poor perception. If the vision encoder works at a low resolution or misreads small text, a careful chain will reason correctly from false premises. Chains also increase multimodal hallucination risk in one specific way: a model may invent visual details to complete a convincing narrative.

  • Error propagation: one wrong observation early in the chain contaminates every later step.
  • Unfaithful rationales: the written steps may not match the internal computation.
  • Higher latency and cost: chains often multiply output tokens several times.
  • Overthinking simple inputs: long chains can talk the model out of a correct first impression.
  • Evaluation difficulty: grading the chain, not only the answer, requires extra annotation or judge models.

Mitigations include grounding every claim in coordinates or crops, adding a verification pass that re-inspects key regions, using external tools for exact reading, and reporting accuracy separately for perception and reasoning subtasks.

Understanding it simply

Multimodal chain-of-thought means asking an AI system to show its work while looking at a picture. Instead of glancing at an image and blurting out an answer, the system first says what it sees, then explains how those details connect to the question, and only then gives its answer.

It resembles a student solving a geometry problem from a diagram. A good student writes down the given angles, notes which lines are parallel and then calculates. The written steps make mistakes easier to catch. The key point is that the steps help, but they are only as good as the looking: if the student misreads an angle, neat reasoning still leads to a wrong result.

Applied in practice

An insurance claims team uses a multimodal model to pre-screen photographs of damaged vehicles together with the written accident description. With direct answering, the model often labeled a claim as consistent even when the dent was on the wrong side of the car, because it matched the general impression of a collision.

The team switched to a grounded chain-of-thought prompt. The model must first list each visible damage area with its position on the vehicle, then quote the relevant sentence of the description, then state whether the two match, and finally give a label of consistent, inconsistent or unclear. Each damage claim includes a bounding box that reviewers can check in one click.

In an internal sample of several hundred claims, the share of mismatches that reviewers had to catch manually fell noticeably, and reviewers reported that the listed observations made disagreements quick to resolve. The cost was roughly three times as many output tokens per claim, which the team accepted because human review time dropped more than compute cost rose.