Video captioning

Also known as: dense video captioning

On this page 7

Video captioning is the task of automatically generating natural-language descriptions of video content, summarizing the objects, actions, events and their order in time. In multimodal AI, it is a core benchmark of whether a model can turn a sequence of frames, and often the accompanying audio, into accurate text.

The task has two main forms. Single-sentence captioning describes a short clip as a whole, while dense video captioning first localizes several events in a long video and then writes a caption with start and end timestamps for each. Unlike subtitles, which transcribe speech, a video caption describes what visibly happens on screen.

How video captioning models turn frames into text

A video captioning system follows an encoder-decoder pipeline: a visual encoder converts sampled frames into feature vectors, a temporal module relates those features across time, and a language decoder produces the caption token by token. The central difficulty is temporal modeling, because the meaning of an action such as opening a door exists only across several frames.

Early systems used convolutional networks with recurrent decoders such as LSTMs. Current systems are mostly video-language models: a pretrained vision encoder produces visual tokens, a multimodal projector maps them into the embedding space of a large language model, and the language model writes the description. Because a minute of video can yield thousands of visual tokens, models rely on frame sampling, token pooling and compression to fit the context window.

The following table lists the main components and the design choices that matter most at each stage.

ComponentRoleCommon design choices
Frame samplerSelects frames to processUniform sampling, 1-8 frames per second, scene-change detection
Visual encoderExtracts per-frame or per-clip featuresViT image encoders, 3D convolutions, video transformers
Temporal moduleRelates features across timeTemporal attention, pooling, token merging
ProjectorAligns visual features with text spaceLinear layer, MLP, query-based resampler
Language decoderGenerates the captionTransformer decoder or full large language model
Event proposerFinds event boundaries in dense captioningProposal networks, timestamp tokens
Components of a typical video captioning pipeline

Types of video captioning tasks

Video captioning covers several task variants that differ in output granularity and in how much temporal localization they require. Choosing the correct variant determines the training data, the architecture and the evaluation metric.

VariantOutputTypical input length
Clip captioningOne sentence per clipRoughly 5-30 seconds
Dense video captioningSeveral timestamped captionsMinutes, sometimes longer
Paragraph captioningCoherent multi-sentence storyOne to several minutes
Detailed captioningLong description of scene, motion and styleShort clips used as training data
Audio-visual captioningDescription that uses sound and speechAny length with an audio track
Main variants of the task

Detailed captioning has become especially important because text-to-video generators are trained on clips paired with rich descriptions. Generation teams therefore caption millions of clips automatically, so caption quality directly limits how well the generator follows prompts.

Video captioning sits among several neighboring tasks in multimodal AI, and the distinctions concern who asks the question and what form the output takes. Captioning produces open-ended descriptive text without a user query.

TaskInputOutputKey difference
Video captioningVideo onlyFree-form descriptionNo question is asked
Video question answeringVideo and a questionTargeted answerOutput is driven by the query
Temporal groundingVideo and a text queryStart and end timeLocates text in the video
Audio captioningAudio onlyDescription of soundsNo visual signal
Speech subtitlingSpeech trackTranscript of spoken wordsTranscribes, does not describe
Comparison with neighboring tasks

In practice, a strong video understanding model performs several of these tasks with the same weights; captioning is the variant whose output is easiest to inspect and most expensive to evaluate reliably.

Training data and evaluation metrics

Video captioning models learn from video-text pairs. Human-annotated benchmarks such as MSR-VTT, MSVD, ActivityNet Captions and YouCook2 are small but precise, while web-scale corpora built from automatic speech transcripts are large but weakly aligned, because narration often describes something other than what is on screen.

Evaluation traditionally relies on n-gram overlap metrics that compare generated captions with human references. These metrics reward wording that matches the references, so they penalize correct paraphrases and fail to detect fluent descriptions of events that never happened.

MetricWhat it measuresMain weakness
BLEUN-gram precision against referencesIgnores meaning and recall
METEORMatches with stems and synonymsStill surface-level
CIDErConsensus-weighted n-gram similarityDepends on many references
SODAStory-level quality of dense captionsComplex and less widely reported
Model-based judgingFactual correctness judged by another modelInherits the judge’s own biases
Common evaluation metrics

For long, detailed captions, teams increasingly combine model-based scoring with human spot checks, because no single automatic metric captures factual accuracy, completeness and temporal order together.

Limitations and common failure modes

The most serious weakness is multimodal hallucination: models describe objects or actions that are plausible for the scene but absent from the footage. A kitchen video may receive a caption that mentions a knife simply because knives usually appear in kitchens.

  • Static bias: models often describe a single representative frame and miss motion, direction and cause.
  • Temporal order errors: events are reported in the wrong sequence when frames are sampled sparsely.
  • Missed short events: actions shorter than the sampling interval may never be seen at all.
  • Counting and identity errors: the number of people or whether the same person reappears is frequently wrong.
  • Long-video cost: compute and memory grow with duration, which forces aggressive compression.

A practical rule is to treat automatic captions as drafts that need verification whenever they are used for accessibility, legal records or safety decisions, and to raise the frame sampling rate for footage with fast action.

Stripped to the essentials

Video captioning means a computer watches a video and writes down what happens in ordinary sentences. It does not copy what people say. It describes what can be seen, such as a dog running across a park.

The software looks at a selection of frames. It works out how things change from one moment to the next. Then it puts that into words. Advanced versions split a long video into separate events and give each one a time stamp. The key point: these descriptions are fast and useful. They can still invent details that are not there, so a person should check them when accuracy matters.

Where it is applied

A regional public broadcaster wants to make its archive of 40,000 hours of news footage searchable. Editors previously relied on sparse manual notes, so finding a clip of a specific flood or bridge opening could take hours. The team runs a dense video captioning model over the archive at two frames per second, producing timestamped descriptions such as a crane lifting a steel beam over a river between 01:12 and 01:47.

The captions are indexed alongside speech transcripts, and archivists review a random 2 percent sample to measure error rates. The review finds that most captions are usable, while counting errors and occasional invented objects remain. The outcome is that search time drops from hours to minutes, and archivists correct the captions only for clips that are actually reused on air.