Visual dialog (also written visual dialogue) is the task in which an AI system holds a multi-turn conversation about an image, answering a sequence of natural-language questions while using both the picture and the dialog history to interpret each new question. Within Multimodal AI, it extends single-shot visual question answering into an exchange where later questions depend on earlier answers.
The defining difficulty is that questions such as “Is he wearing a hat?” or “What color is it?” cannot be answered from the image alone. The model must resolve references across turns, keep track of what has already been established and then ground the answer in the correct image region. This combination of conversational memory and visual grounding makes visual dialog a demanding test of whether a system genuinely connects language and vision.
How a visual dialog system processes each turn
A visual dialog system answers each question by combining three inputs: the image, the current question and the history of previous question-answer pairs, often together with an initial caption. Classic research models encode each input separately and then fuse them; current multimodal large language models place visual tokens and the full conversation in a single context window and let attention do the fusion.
The task was formalized by the VisDial benchmark introduced in 2017. Its data were collected by pairing two crowd workers: a questioner who saw only a caption and an answerer who saw the image. Because the questioner could not see the picture, the questions are open, exploratory and heavily dependent on earlier turns, which is exactly the behavior the task is meant to capture.
The following table summarizes the components found in most visual dialog architectures, whether classic or modern.
| Component | Role | Typical implementation |
|---|---|---|
| Image encoder | Turns pixels into visual features | Region detector features or a vision transformer |
| Question encoder | Represents the current question | Recurrent network or transformer text encoder |
| History encoder | Summarizes earlier turns and the caption | Hierarchical encoder, memory network or full-context attention |
| Fusion module | Combines visual, question and history signals | Co-attention, cross-attention or a shared transformer |
| Answer decoder | Produces or selects the answer | Discriminative ranker or generative text decoder |
Two decoding styles exist. A discriminative decoder ranks a fixed list of candidate answers, which suits benchmark evaluation, while a generative decoder writes a free-form answer token by token, which suits real assistants but is harder to score objectively.
Why dialog history matters in visual dialog
Dialog history matters because many follow-up questions are underspecified without context. The central skill is visual coreference resolution: linking pronouns and short phrases such as “it”, “the other one” or “the man on the left” to the entity discussed earlier and then to the right region of the image.
The table below lists the conversational phenomena that distinguish visual dialog from isolated question answering.
| Phenomenon | Example turn | What the model must do |
|---|---|---|
| Pronoun coreference | Is it moving? | Link the pronoun to an entity named earlier |
| Ellipsis | And the other one? | Restore the omitted predicate from the previous question |
| Answer dependence | What is he holding, then? | Use a prior answer as a premise |
| Topic shift | Is there any sky visible? | Recognize that history no longer applies |
| Consistency | Are there two dogs? | Avoid contradicting earlier answers |
Research has shown that history is a double-edged signal. It is essential for coreference, yet models can also over-rely on it, copying earlier answers or exploiting conversational patterns instead of examining the image. Well-designed systems therefore learn when to consult history and when to look only at the picture.
How visual dialog performance is measured
Visual dialog is usually measured with retrieval-style ranking metrics, because the standard benchmark gives each question 100 candidate answers and asks the model to rank them. Each image carries ten question-answer rounds, drawn mainly from COCO photographs, with roughly 120,000 images in the training split of version 1.0.
| Metric | What it measures | Better value |
|---|---|---|
| Mean reciprocal rank (MRR) | Position of the single human answer in the ranking | Higher |
| Recall@k | Whether the human answer appears in the top 1, 5 or 10 | Higher |
| Mean rank | Average position of the human answer | Lower |
| NDCG | Ranking quality against dense relevance scores for all candidates | Higher |
NDCG was added because many candidates are acceptable paraphrases, such as “yes” and “yes, it is”. A notable finding is the trade-off between MRR and NDCG: fine-tuning on the dense relevance annotations raises NDCG while often lowering MRR, so a single headline number rarely describes model quality fully. Generative assistants are increasingly evaluated with human judgment or model-based grading instead.
Visual dialog compared with related multimodal tasks
Visual dialog sits between visual question answering, which handles one question at a time, and open-ended multimodal chat, which handles arbitrary requests about images. The comparison below clarifies where the boundaries lie.
| Task | Turns | Main challenge | Typical output |
|---|---|---|---|
| Visual question answering | Single | Grounding one question in the image | Short answer |
| Image captioning | None | Describing salient content | One sentence |
| Visual grounding | Single | Locating the referred region | Bounding box or mask |
| Visual dialog | Multiple | Coreference and consistent grounding over turns | Answer per turn |
| Goal-oriented visual dialog | Multiple | Asking questions to identify a target object | Questions and a final guess |
Goal-oriented variants, such as guessing games in which one agent must identify a hidden object, add strategic question asking to the answering skill. In contrast, the original VisDial setup evaluates only the answerer, which keeps measurement simpler but leaves the questioner role unexamined.
From specialized models to multimodal assistants
Early visual dialog models used late fusion, hierarchical recurrent encoders and memory networks that stored each previous turn as a separate slot. Later work brought vision-language pretraining on large image-text corpora, followed by transformer models that processed image regions, history and question jointly and set new benchmark records.
Today, most practical visual dialog runs inside multimodal large language models, where a vision encoder and a multimodal projector feed visual tokens into a language model that already handles long conversations. Visual instruction tuning on multi-turn image conversations gives these assistants dialog ability almost as a side effect, although benchmark-specific ranking performance is no longer the main design goal.
Limitations and failure modes of visual dialog
The main weaknesses of visual dialog systems are language priors, history shortcuts and hallucination, all of which let a model sound fluent while ignoring what the image shows. The table maps common failures to causes and mitigations.
| Failure | Typical cause | Mitigation |
|---|---|---|
| Answering without looking | Strong language priors in training data | Balanced data and image-ablation tests |
| Copying earlier answers | Over-reliance on dialog history | History dropout and consistency checks |
| Wrong referent | Weak coreference resolution | Explicit grounding of mentions to regions |
| Invented objects | Multimodal hallucination | Grounded training and refusal on uncertainty |
| Self-contradiction | No memory of committed facts | State tracking and contradiction penalties |
Benchmarks add their own limits. Many VisDial answers are short yes or no responses, which inflates scores for models that guess well, and crowd-sourced answers sometimes disagree with the image. High benchmark scores do not guarantee reliable behavior in long, real conversations about unfamiliar images.
A non-technical view
Visual dialog is like talking about a photo with someone who can see it while you cannot. You ask a question, they answer, and your next question builds on what they just said. If you ask “Is there a dog?” and then “What color is it?”, they know that “it” means the dog.
An AI system doing visual dialog must do the same thing: look at the picture, remember the conversation so far and answer each question in a way that fits both. The key point is that every answer must match the image and the earlier answers at the same time.
A concrete example
A blind user photographs a shelf in a pharmacy with a smartphone assistant built on a multimodal language model. The first question is “What is on the middle shelf?” and the assistant answers that there are three boxes of pain relievers. The user then asks “Which one is the cheapest?” and later “Does that one contain ibuprofen?”
Each follow-up depends on the previous turn: “that one” refers to the box identified as cheapest, so the assistant must keep the referent fixed while reading a different part of the label. When the price tag is blurred, a well-designed system says it cannot read the price instead of guessing. The outcome is a short, reliable exchange of four turns that replaces a request for help from staff.