Visual dialog

Also known as: visual dialogue

On this page 8

Visual dialog (also written visual dialogue) is the task in which an AI system holds a multi-turn conversation about an image, answering a sequence of natural-language questions while using both the picture and the dialog history to interpret each new question. Within Multimodal AI, it extends single-shot visual question answering into an exchange where later questions depend on earlier answers.

The defining difficulty is that questions such as “Is he wearing a hat?” or “What color is it?” cannot be answered from the image alone. The model must resolve references across turns, keep track of what has already been established and then ground the answer in the correct image region. This combination of conversational memory and visual grounding makes visual dialog a demanding test of whether a system genuinely connects language and vision.

How a visual dialog system processes each turn

A visual dialog system answers each question by combining three inputs: the image, the current question and the history of previous question-answer pairs, often together with an initial caption. Classic research models encode each input separately and then fuse them; current multimodal large language models place visual tokens and the full conversation in a single context window and let attention do the fusion.

The task was formalized by the VisDial benchmark introduced in 2017. Its data were collected by pairing two crowd workers: a questioner who saw only a caption and an answerer who saw the image. Because the questioner could not see the picture, the questions are open, exploratory and heavily dependent on earlier turns, which is exactly the behavior the task is meant to capture.

The following table summarizes the components found in most visual dialog architectures, whether classic or modern.

ComponentRoleTypical implementation
Image encoderTurns pixels into visual featuresRegion detector features or a vision transformer
Question encoderRepresents the current questionRecurrent network or transformer text encoder
History encoderSummarizes earlier turns and the captionHierarchical encoder, memory network or full-context attention
Fusion moduleCombines visual, question and history signalsCo-attention, cross-attention or a shared transformer
Answer decoderProduces or selects the answerDiscriminative ranker or generative text decoder
Core components of a visual dialog model

Two decoding styles exist. A discriminative decoder ranks a fixed list of candidate answers, which suits benchmark evaluation, while a generative decoder writes a free-form answer token by token, which suits real assistants but is harder to score objectively.

Why dialog history matters in visual dialog

Dialog history matters because many follow-up questions are underspecified without context. The central skill is visual coreference resolution: linking pronouns and short phrases such as “it”, “the other one” or “the man on the left” to the entity discussed earlier and then to the right region of the image.

The table below lists the conversational phenomena that distinguish visual dialog from isolated question answering.

PhenomenonExample turnWhat the model must do
Pronoun coreferenceIs it moving?Link the pronoun to an entity named earlier
EllipsisAnd the other one?Restore the omitted predicate from the previous question
Answer dependenceWhat is he holding, then?Use a prior answer as a premise
Topic shiftIs there any sky visible?Recognize that history no longer applies
ConsistencyAre there two dogs?Avoid contradicting earlier answers
Conversational phenomena in visual dialog

Research has shown that history is a double-edged signal. It is essential for coreference, yet models can also over-rely on it, copying earlier answers or exploiting conversational patterns instead of examining the image. Well-designed systems therefore learn when to consult history and when to look only at the picture.

How visual dialog performance is measured

Visual dialog is usually measured with retrieval-style ranking metrics, because the standard benchmark gives each question 100 candidate answers and asks the model to rank them. Each image carries ten question-answer rounds, drawn mainly from COCO photographs, with roughly 120,000 images in the training split of version 1.0.

MetricWhat it measuresBetter value
Mean reciprocal rank (MRR)Position of the single human answer in the rankingHigher
Recall@kWhether the human answer appears in the top 1, 5 or 10Higher
Mean rankAverage position of the human answerLower
NDCGRanking quality against dense relevance scores for all candidatesHigher
Standard evaluation metrics

NDCG was added because many candidates are acceptable paraphrases, such as “yes” and “yes, it is”. A notable finding is the trade-off between MRR and NDCG: fine-tuning on the dense relevance annotations raises NDCG while often lowering MRR, so a single headline number rarely describes model quality fully. Generative assistants are increasingly evaluated with human judgment or model-based grading instead.

Visual dialog sits between visual question answering, which handles one question at a time, and open-ended multimodal chat, which handles arbitrary requests about images. The comparison below clarifies where the boundaries lie.

TaskTurnsMain challengeTypical output
Visual question answeringSingleGrounding one question in the imageShort answer
Image captioningNoneDescribing salient contentOne sentence
Visual groundingSingleLocating the referred regionBounding box or mask
Visual dialogMultipleCoreference and consistent grounding over turnsAnswer per turn
Goal-oriented visual dialogMultipleAsking questions to identify a target objectQuestions and a final guess
Visual dialog and neighboring tasks

Goal-oriented variants, such as guessing games in which one agent must identify a hidden object, add strategic question asking to the answering skill. In contrast, the original VisDial setup evaluates only the answerer, which keeps measurement simpler but leaves the questioner role unexamined.

From specialized models to multimodal assistants

Early visual dialog models used late fusion, hierarchical recurrent encoders and memory networks that stored each previous turn as a separate slot. Later work brought vision-language pretraining on large image-text corpora, followed by transformer models that processed image regions, history and question jointly and set new benchmark records.

Today, most practical visual dialog runs inside multimodal large language models, where a vision encoder and a multimodal projector feed visual tokens into a language model that already handles long conversations. Visual instruction tuning on multi-turn image conversations gives these assistants dialog ability almost as a side effect, although benchmark-specific ranking performance is no longer the main design goal.

Limitations and failure modes of visual dialog

The main weaknesses of visual dialog systems are language priors, history shortcuts and hallucination, all of which let a model sound fluent while ignoring what the image shows. The table maps common failures to causes and mitigations.

FailureTypical causeMitigation
Answering without lookingStrong language priors in training dataBalanced data and image-ablation tests
Copying earlier answersOver-reliance on dialog historyHistory dropout and consistency checks
Wrong referentWeak coreference resolutionExplicit grounding of mentions to regions
Invented objectsMultimodal hallucinationGrounded training and refusal on uncertainty
Self-contradictionNo memory of committed factsState tracking and contradiction penalties
Common failure modes and mitigations

Benchmarks add their own limits. Many VisDial answers are short yes or no responses, which inflates scores for models that guess well, and crowd-sourced answers sometimes disagree with the image. High benchmark scores do not guarantee reliable behavior in long, real conversations about unfamiliar images.

A non-technical view

Visual dialog is like talking about a photo with someone who can see it while you cannot. You ask a question, they answer, and your next question builds on what they just said. If you ask “Is there a dog?” and then “What color is it?”, they know that “it” means the dog.

An AI system doing visual dialog must do the same thing: look at the picture, remember the conversation so far and answer each question in a way that fits both. The key point is that every answer must match the image and the earlier answers at the same time.

A concrete example

A blind user photographs a shelf in a pharmacy with a smartphone assistant built on a multimodal language model. The first question is “What is on the middle shelf?” and the assistant answers that there are three boxes of pain relievers. The user then asks “Which one is the cheapest?” and later “Does that one contain ibuprofen?”

Each follow-up depends on the previous turn: “that one” refers to the box identified as cheapest, so the assistant must keep the referent fixed while reading a different part of the label. When the price tag is blurred, a well-designed system says it cannot read the price instead of guessing. The outcome is a short, reliable exchange of four turns that replaces a request for help from staff.