Visual question answering

Also known as: VQA

On this page 7

Visual question answering (VQA) is the task of producing a correct natural-language answer to a question about an image, where the model must combine what it sees with what the question asks. In Multimodal AI, VQA is one of the oldest and most widely used benchmarks for testing whether a system genuinely connects vision and language rather than handling each modality in isolation.

A VQA system receives two inputs, such as a street photograph and the question “How many bicycles are parked by the wall?”, and must output a short answer such as “three”. The task looks simple, yet it bundles object recognition, counting, spatial reasoning, text reading and commonsense knowledge into a single query, which is why it remains a demanding test of visual understanding.

How visual question answering systems work

Every VQA system follows the same basic pipeline: it encodes the image, encodes the question, fuses the two representations and decodes an answer. The architecture of each stage has changed considerably since the task was formalized around 2015, but the logical steps remain the same.

Early systems used a convolutional neural network for the image and a recurrent network for the question, then combined the two vectors by element-wise multiplication or concatenation. The answer was chosen by a classifier over a fixed vocabulary, typically the 1,000 to 3,000 most frequent answers in the training set. Later designs added attention mechanisms that let the question select relevant image regions, often using object detector features rather than a uniform grid.

Modern systems are built on vision-language models. A vision encoder, usually a Vision Transformer, turns the image into a sequence of visual tokens; a multimodal projector maps those tokens into the embedding space of a large language model; and the language model generates the answer as free text. This generative formulation removes the fixed answer vocabulary and allows explanations, but it also makes evaluation harder.

The table below summarizes how the main architectural generations differ.

GenerationImage representationFusion methodAnswer output
Joint embedding modelsGlobal CNN feature vectorConcatenation or multiplicationClassifier over fixed answers
Attention-based modelsGrid or detected object regionsQuestion-guided attentionClassifier over fixed answers
Pretrained transformersRegion features with positionsCross-attention layersClassifier or short decoder
Vision-language modelsVisual tokens from a ViT encoderTokens fed into a language modelFree-form generated text
Architectural generations of VQA systems

Question types and the skills they test

VQA questions are usually grouped by the reasoning skill they require, because a single accuracy score hides large differences between categories. Yes/no questions are easiest, while counting and text-reading questions remain the most error-prone for many models.

Question typeExample questionSkill requiredTypical difficulty
Yes/noIs the door open?Attribute verificationLow
Object identityWhat is the man holding?RecognitionLow to medium
CountingHow many cups are on the table?Enumeration of instancesHigh
Spatial relationWhat is left of the lamp?Spatial reasoningMedium to high
Scene textWhat does the sign say?Optical character readingHigh
Knowledge-basedWhich sport uses this equipment?External world knowledgeMedium to high
Common VQA question categories

Specialized subfields have grown from these categories. Document and chart questions test reading of layouts and numbers, and medical VQA asks about radiology or pathology images. When the input is a clip rather than a still frame, the task becomes video question answering, which adds temporal reasoning.

Benchmarks and evaluation metrics

VQA progress is measured on public benchmark datasets that pair images with human-written questions and reference answers. Each benchmark targets a different weakness, and strong results on one rarely guarantee strong results on another.

BenchmarkFocusAnswer format
VQA v2General questions with balanced image pairsShort open answers
GQACompositional questions from scene graphsShort open answers
OK-VQAQuestions needing outside knowledgeShort open answers
TextVQAReading text inside natural imagesShort open answers
VizWizPhotos taken by blind usersOpen answers or unanswerable
Widely used VQA benchmarks

The standard VQA accuracy metric compares a prediction with ten human answers and counts it as fully correct if at least three annotators gave the same answer. This soft consensus scoring tolerates legitimate disagreement, such as “red” versus “dark red”. Generative models often produce longer phrasing, so evaluators increasingly normalize answers or use a language model as a judge, which introduces its own bias and must be documented.

VQA sits among several vision-language tasks that share architectures but differ in input, output and evaluation. Distinguishing them helps when choosing a dataset or interpreting a reported score.

TaskInputOutputMain difference from VQA
Image captioningImage onlyDescriptive sentenceNo question steers the output
Visual groundingImage and phraseBounding box or regionLocalizes rather than answers
Visual entailmentImage and statementEntail, neutral or contradictJudges a claim, not a query
Visual dialogImage and dialog historyAnswer in a conversationAdds multi-turn context
Image-text matchingImage and captionMatch scoreRanks pairs rather than answering
VQA and neighboring vision-language tasks

In current practice, a single multimodal large language model handles all of these tasks through instructions, so the boundaries are now defined more by benchmarks and prompts than by separate architectures.

Limitations, biases and failure modes

The best-documented weakness of VQA systems is language prior bias: models learn to answer from the question alone. In early datasets, answering “yes” to every “Is there a…” question or “tennis” to every sport question produced deceptively high scores. VQA v2 was built with complementary image pairs to reduce this shortcut, but the tendency persists in newer models.

  • Multimodal hallucination: the model asserts objects or attributes that are not in the image, especially in long generated answers.
  • Counting errors: performance drops sharply beyond about five to ten instances, partly because visual tokens compress small repeated objects.
  • Low-resolution loss: fixed input sizes such as 224 or 336 pixels erase small text and fine detail unless tiling or higher resolutions are used.
  • Unanswerable questions: many models answer confidently even when the image does not contain the information, which matters greatly in assistive use.
  • Benchmark contamination: public test images and questions can leak into pretraining data and inflate reported accuracy.

Mitigations include visual grounding of answers to image regions, higher-resolution encoders, explicit training on unanswerable examples and evaluation on held-out or newly collected data. For high-stakes domains such as medicine, human review of answers remains necessary.

Understanding it simply

Visual question answering means showing a computer a picture and asking it a question about that picture in ordinary words. The computer has to look, understand the question and reply. A person does the same when glancing at a photo and saying “there are two dogs on the sofa”.

The question can be about anything. It may ask about colors, numbers, positions or written words. A good system really looks at the image instead of guessing from the wording. It should also admit when the picture does not hold the answer.

Applied in practice

A nonprofit building an assistive smartphone app for blind and low-vision users adds a VQA feature. A user photographs a can in the kitchen and asks, “What is in this can and when does it expire?” The app sends the image and question to a vision-language model running at a higher input resolution so that small printed text remains legible.

During testing, the team finds that the model invents expiry dates in about one blurry photo out of ten. They add training examples where the correct reply is “the date is not readable, please rotate the can”, and they prompt the model to state when information is not visible. After the change, confident wrong answers fall noticeably, and users report that they trust the app more because it asks for a better photo instead of guessing.