Video question answering (VideoQA) is the task in which a model receives a video clip and a natural-language question about it and must return a correct answer, either by choosing from candidate options or by generating free-form text. In multimodal AI, it is one of the most demanding benchmarks of joint understanding, because the answer often depends on motion, order of events and causality rather than on any single frame.
VideoQA extends visual question answering from still images to time. A system must decide which moments matter, track objects and people across frames, combine visual evidence with speech or subtitles when they exist, and reason about what happened, why it happened and what is likely to happen next.
How video question answering systems process a clip
A VideoQA system turns a clip into a compact sequence of visual representations and conditions a language model or classifier on that sequence together with the question. The pipeline almost always has four stages, and the quality of each one limits the final answer.
- Frame sampling: the system selects frames at a fixed rate (often 1 to 2 frames per second) or uniformly across the clip, typically 8 to 64 frames for short videos.
- Visual encoding: a vision encoder such as a ViT-based image model or a dedicated video encoder converts frames or spatiotemporal patches into embeddings.
- Temporal aggregation: pooling, temporal attention or a video tokenizer compresses many frame embeddings into a manageable number of visual tokens.
- Answer generation: a multimodal projector maps visual tokens into the language model space, and the model decodes an answer or scores candidate answers.
The central engineering problem is the token budget. A single high-resolution frame can yield several hundred visual tokens, so a two-minute clip sampled naively would exceed the context window of most models. Practical systems therefore trade spatial detail against temporal coverage, and that trade-off explains many of their errors.
Question types and reasoning levels in VideoQA
VideoQA questions differ sharply in how much temporal reasoning they require, and benchmark scores are only meaningful when the question mix is known. The table below groups the most common question types by the evidence they need.
| Question type | Example question | Evidence required |
|---|---|---|
| Descriptive | What color is the car? | One or a few frames |
| Counting | How many times does the ball bounce? | Dense frames across an interval |
| Temporal order | What does the man do after opening the door? | Correct sequence of events |
| Causal | Why did the glass fall? | Link between an action and its effect |
| Predictive | What will the cyclist most likely do next? | Motion trends and common sense |
| Long-range | Where did the woman leave her keys earlier? | Memory across minutes or hours |
A recurring finding in research is the single-frame shortcut: many questions in older datasets can be answered from one well-chosen frame, so a model with no real temporal understanding still scores well. Newer benchmarks deliberately include temporal and causal questions whose answers change if the frames are shuffled.
Model architectures used for VideoQA
Modern VideoQA is dominated by video-language models, which pair a visual encoder with a large language model, but earlier specialized designs remain useful baselines. The architectures differ mainly in where and how vision and language are fused.
| Architecture | How it fuses video and text | Main strength | Main weakness |
|---|---|---|---|
| Memory and attention networks | Question-guided attention over frame features | Small and fast | Weak open-ended generation |
| Pretrained video-text transformers | Joint encoder trained on video-caption pairs | Strong retrieval and multiple choice | Limited reasoning depth |
| Projector-based video LLMs | Visual tokens fed into a language model | Free-form answers and instructions | High token cost per clip |
| Natively multimodal models | Video, audio and text trained together from the start | Long context and audio use | Expensive to train and serve |
| Agentic pipelines | Language model calls captioning, search and grounding tools | Handles hour-long video | Latency and error compounding |
Two techniques matter across all families. Temporal grounding locates the interval relevant to the question before answering, which saves tokens and improves accuracy on long videos. Speech and subtitle integration adds the audio track, since many questions about instructional or narrative video can only be answered from what is said.
Benchmarks and evaluation metrics
VideoQA is evaluated mostly with multiple-choice accuracy, because free-form answers are hard to score reliably. Open-ended answers are compared with references through exact match, word-overlap metrics or, increasingly, a language model acting as a judge, which is convenient but introduces its own bias.
Well-known datasets include MSRVTT-QA and MSVD-QA for short descriptive questions, ActivityNet-QA for longer activity clips, NExT-QA for causal and temporal reasoning, TVQA for questions that need dialogue, and long-video suites such as EgoSchema and Video-MME. Results on these sets are not directly comparable, because clip length, answer format and question style differ.
| Setting | Typical clip length | What a high score shows |
|---|---|---|
| Short-clip open-ended | 10 to 30 seconds | Recognition of objects and actions |
| Causal multiple choice | 30 seconds to 2 minutes | Event ordering and explanation |
| Dialogue-dependent | About 1 minute | Use of subtitles alongside vision |
| Long-form video | 3 minutes to over 1 hour | Memory and evidence retrieval |
A sound evaluation also reports blind baselines, in which a model answers from the question alone or from a single frame. When the blind score is close to the full score, the benchmark is measuring language priors, not video understanding.
How VideoQA differs from related multimodal tasks
VideoQA sits between several neighboring tasks and shares components with all of them, but its defining feature is that the question sets the target. The comparison below clarifies the boundaries.
| Task | Input | Output | Key difference from VideoQA |
|---|---|---|---|
| Visual question answering | Image and question | Answer | No time dimension |
| Video captioning | Video | Descriptive text | No question steering the output |
| Temporal grounding | Video and text query | Start and end time | Locates evidence instead of answering |
| Cross-modal retrieval | Text or video query | Ranked list of items | Searches a collection, not one clip |
In practice these tasks are combined: a long-video assistant may retrieve candidate clips, ground the relevant interval and then perform VideoQA on the selected segment.
Limitations, failure modes and design choices
The most frequent failure is missing evidence: the decisive event happens between sampled frames, so the model answers confidently from incomplete input. Increasing the sampling rate helps but raises cost roughly in proportion to the number of frames.
- Multimodal hallucination: models describe plausible actions that never occur, especially in counting and ordering questions.
- Weak fine-grained motion: direction, speed and small hand movements are often misread at low frame rates.
- Language bias: answers follow common-sense priors from text instead of what the video actually shows.
- Long-video drift: accuracy falls as clips grow longer unless retrieval or grounding narrows the input.
- Privacy and consent: surveillance and workplace footage raises legal obligations that apply before any model is deployed.
For production use, the decisive choices are the frame budget, whether to include audio transcripts, and whether to add a grounding step. Teams should validate on their own footage, because public benchmark scores transfer poorly to domains such as medical, industrial or sports video.
A plain-language summary
Video question answering means asking a computer a question about a video and getting a sensible answer back. You might ask what someone picked up, how many people entered a room or why a vase fell over.
The hard part is time. A photo shows one moment, but a video tells a story, so the system has to notice what changed, in what order and for what reason. Think of it as a viewer who must watch, remember and explain, rather than just describe a single picture. Current systems are good at what is visible and weaker at counting, fast motion and very long videos.
An example from practice
A video archive team at a regional broadcaster holds about 40,000 hours of news footage and needs to answer editorial requests such as which reports show flooding in a given town and what the reporter says about evacuation. The team builds a pipeline that transcribes speech, samples frames at one frame per second, indexes clips with multimodal embeddings and then runs a video-language model on the top candidate segments.
Editors type a question, receive short answers with time codes and check the cited segments before reuse. In an internal test on 300 real requests, the system located relevant footage for most descriptive questions, while counting and fast-motion questions still needed human review. The outcome is faster research with a human verifier in the loop, which suits archive work where accuracy matters more than speed.