Video question answering

Also known as: VideoQA

On this page 8

Video question answering (VideoQA) is the task in which a model receives a video clip and a natural-language question about it and must return a correct answer, either by choosing from candidate options or by generating free-form text. In multimodal AI, it is one of the most demanding benchmarks of joint understanding, because the answer often depends on motion, order of events and causality rather than on any single frame.

VideoQA extends visual question answering from still images to time. A system must decide which moments matter, track objects and people across frames, combine visual evidence with speech or subtitles when they exist, and reason about what happened, why it happened and what is likely to happen next.

How video question answering systems process a clip

A VideoQA system turns a clip into a compact sequence of visual representations and conditions a language model or classifier on that sequence together with the question. The pipeline almost always has four stages, and the quality of each one limits the final answer.

  1. Frame sampling: the system selects frames at a fixed rate (often 1 to 2 frames per second) or uniformly across the clip, typically 8 to 64 frames for short videos.
  2. Visual encoding: a vision encoder such as a ViT-based image model or a dedicated video encoder converts frames or spatiotemporal patches into embeddings.
  3. Temporal aggregation: pooling, temporal attention or a video tokenizer compresses many frame embeddings into a manageable number of visual tokens.
  4. Answer generation: a multimodal projector maps visual tokens into the language model space, and the model decodes an answer or scores candidate answers.

The central engineering problem is the token budget. A single high-resolution frame can yield several hundred visual tokens, so a two-minute clip sampled naively would exceed the context window of most models. Practical systems therefore trade spatial detail against temporal coverage, and that trade-off explains many of their errors.

Question types and reasoning levels in VideoQA

VideoQA questions differ sharply in how much temporal reasoning they require, and benchmark scores are only meaningful when the question mix is known. The table below groups the most common question types by the evidence they need.

Question typeExample questionEvidence required
DescriptiveWhat color is the car?One or a few frames
CountingHow many times does the ball bounce?Dense frames across an interval
Temporal orderWhat does the man do after opening the door?Correct sequence of events
CausalWhy did the glass fall?Link between an action and its effect
PredictiveWhat will the cyclist most likely do next?Motion trends and common sense
Long-rangeWhere did the woman leave her keys earlier?Memory across minutes or hours
Common question types in video question answering

A recurring finding in research is the single-frame shortcut: many questions in older datasets can be answered from one well-chosen frame, so a model with no real temporal understanding still scores well. Newer benchmarks deliberately include temporal and causal questions whose answers change if the frames are shuffled.

Model architectures used for VideoQA

Modern VideoQA is dominated by video-language models, which pair a visual encoder with a large language model, but earlier specialized designs remain useful baselines. The architectures differ mainly in where and how vision and language are fused.

ArchitectureHow it fuses video and textMain strengthMain weakness
Memory and attention networksQuestion-guided attention over frame featuresSmall and fastWeak open-ended generation
Pretrained video-text transformersJoint encoder trained on video-caption pairsStrong retrieval and multiple choiceLimited reasoning depth
Projector-based video LLMsVisual tokens fed into a language modelFree-form answers and instructionsHigh token cost per clip
Natively multimodal modelsVideo, audio and text trained together from the startLong context and audio useExpensive to train and serve
Agentic pipelinesLanguage model calls captioning, search and grounding toolsHandles hour-long videoLatency and error compounding
Architecture families and their trade-offs

Two techniques matter across all families. Temporal grounding locates the interval relevant to the question before answering, which saves tokens and improves accuracy on long videos. Speech and subtitle integration adds the audio track, since many questions about instructional or narrative video can only be answered from what is said.

Benchmarks and evaluation metrics

VideoQA is evaluated mostly with multiple-choice accuracy, because free-form answers are hard to score reliably. Open-ended answers are compared with references through exact match, word-overlap metrics or, increasingly, a language model acting as a judge, which is convenient but introduces its own bias.

Well-known datasets include MSRVTT-QA and MSVD-QA for short descriptive questions, ActivityNet-QA for longer activity clips, NExT-QA for causal and temporal reasoning, TVQA for questions that need dialogue, and long-video suites such as EgoSchema and Video-MME. Results on these sets are not directly comparable, because clip length, answer format and question style differ.

SettingTypical clip lengthWhat a high score shows
Short-clip open-ended10 to 30 secondsRecognition of objects and actions
Causal multiple choice30 seconds to 2 minutesEvent ordering and explanation
Dialogue-dependentAbout 1 minuteUse of subtitles alongside vision
Long-form video3 minutes to over 1 hourMemory and evidence retrieval
Evaluation settings and what they measure

A sound evaluation also reports blind baselines, in which a model answers from the question alone or from a single frame. When the blind score is close to the full score, the benchmark is measuring language priors, not video understanding.

VideoQA sits between several neighboring tasks and shares components with all of them, but its defining feature is that the question sets the target. The comparison below clarifies the boundaries.

TaskInputOutputKey difference from VideoQA
Visual question answeringImage and questionAnswerNo time dimension
Video captioningVideoDescriptive textNo question steering the output
Temporal groundingVideo and text queryStart and end timeLocates evidence instead of answering
Cross-modal retrievalText or video queryRanked list of itemsSearches a collection, not one clip
VideoQA compared with neighboring tasks

In practice these tasks are combined: a long-video assistant may retrieve candidate clips, ground the relevant interval and then perform VideoQA on the selected segment.

Limitations, failure modes and design choices

The most frequent failure is missing evidence: the decisive event happens between sampled frames, so the model answers confidently from incomplete input. Increasing the sampling rate helps but raises cost roughly in proportion to the number of frames.

  • Multimodal hallucination: models describe plausible actions that never occur, especially in counting and ordering questions.
  • Weak fine-grained motion: direction, speed and small hand movements are often misread at low frame rates.
  • Language bias: answers follow common-sense priors from text instead of what the video actually shows.
  • Long-video drift: accuracy falls as clips grow longer unless retrieval or grounding narrows the input.
  • Privacy and consent: surveillance and workplace footage raises legal obligations that apply before any model is deployed.

For production use, the decisive choices are the frame budget, whether to include audio transcripts, and whether to add a grounding step. Teams should validate on their own footage, because public benchmark scores transfer poorly to domains such as medical, industrial or sports video.

A plain-language summary

Video question answering means asking a computer a question about a video and getting a sensible answer back. You might ask what someone picked up, how many people entered a room or why a vase fell over.

The hard part is time. A photo shows one moment, but a video tells a story, so the system has to notice what changed, in what order and for what reason. Think of it as a viewer who must watch, remember and explain, rather than just describe a single picture. Current systems are good at what is visible and weaker at counting, fast motion and very long videos.

An example from practice

A video archive team at a regional broadcaster holds about 40,000 hours of news footage and needs to answer editorial requests such as which reports show flooding in a given town and what the reporter says about evacuation. The team builds a pipeline that transcribes speech, samples frames at one frame per second, indexes clips with multimodal embeddings and then runs a video-language model on the top candidate segments.

Editors type a question, receive short answers with time codes and check the cited segments before reuse. In an internal test on 300 real requests, the system located relevant footage for most descriptive questions, while counting and fast-motion questions still needed human review. The outcome is faster research with a human verifier in the loop, which suits archive work where accuracy matters more than speed.