Video captioning is the task of automatically generating natural-language descriptions of video content, summarizing the objects, actions, events and their order in time. In multimodal AI, it is a core benchmark of whether a model can turn a sequence of frames, and often the accompanying audio, into accurate text.
The task has two main forms. Single-sentence captioning describes a short clip as a whole, while dense video captioning first localizes several events in a long video and then writes a caption with start and end timestamps for each. Unlike subtitles, which transcribe speech, a video caption describes what visibly happens on screen.
How video captioning models turn frames into text
A video captioning system follows an encoder-decoder pipeline: a visual encoder converts sampled frames into feature vectors, a temporal module relates those features across time, and a language decoder produces the caption token by token. The central difficulty is temporal modeling, because the meaning of an action such as opening a door exists only across several frames.
Early systems used convolutional networks with recurrent decoders such as LSTMs. Current systems are mostly video-language models: a pretrained vision encoder produces visual tokens, a multimodal projector maps them into the embedding space of a large language model, and the language model writes the description. Because a minute of video can yield thousands of visual tokens, models rely on frame sampling, token pooling and compression to fit the context window.
The following table lists the main components and the design choices that matter most at each stage.
| Component | Role | Common design choices |
|---|---|---|
| Frame sampler | Selects frames to process | Uniform sampling, 1-8 frames per second, scene-change detection |
| Visual encoder | Extracts per-frame or per-clip features | ViT image encoders, 3D convolutions, video transformers |
| Temporal module | Relates features across time | Temporal attention, pooling, token merging |
| Projector | Aligns visual features with text space | Linear layer, MLP, query-based resampler |
| Language decoder | Generates the caption | Transformer decoder or full large language model |
| Event proposer | Finds event boundaries in dense captioning | Proposal networks, timestamp tokens |
Types of video captioning tasks
Video captioning covers several task variants that differ in output granularity and in how much temporal localization they require. Choosing the correct variant determines the training data, the architecture and the evaluation metric.
| Variant | Output | Typical input length |
|---|---|---|
| Clip captioning | One sentence per clip | Roughly 5-30 seconds |
| Dense video captioning | Several timestamped captions | Minutes, sometimes longer |
| Paragraph captioning | Coherent multi-sentence story | One to several minutes |
| Detailed captioning | Long description of scene, motion and style | Short clips used as training data |
| Audio-visual captioning | Description that uses sound and speech | Any length with an audio track |
Detailed captioning has become especially important because text-to-video generators are trained on clips paired with rich descriptions. Generation teams therefore caption millions of clips automatically, so caption quality directly limits how well the generator follows prompts.
How video captioning differs from related tasks
Video captioning sits among several neighboring tasks in multimodal AI, and the distinctions concern who asks the question and what form the output takes. Captioning produces open-ended descriptive text without a user query.
| Task | Input | Output | Key difference |
|---|---|---|---|
| Video captioning | Video only | Free-form description | No question is asked |
| Video question answering | Video and a question | Targeted answer | Output is driven by the query |
| Temporal grounding | Video and a text query | Start and end time | Locates text in the video |
| Audio captioning | Audio only | Description of sounds | No visual signal |
| Speech subtitling | Speech track | Transcript of spoken words | Transcribes, does not describe |
In practice, a strong video understanding model performs several of these tasks with the same weights; captioning is the variant whose output is easiest to inspect and most expensive to evaluate reliably.
Training data and evaluation metrics
Video captioning models learn from video-text pairs. Human-annotated benchmarks such as MSR-VTT, MSVD, ActivityNet Captions and YouCook2 are small but precise, while web-scale corpora built from automatic speech transcripts are large but weakly aligned, because narration often describes something other than what is on screen.
Evaluation traditionally relies on n-gram overlap metrics that compare generated captions with human references. These metrics reward wording that matches the references, so they penalize correct paraphrases and fail to detect fluent descriptions of events that never happened.
| Metric | What it measures | Main weakness |
|---|---|---|
| BLEU | N-gram precision against references | Ignores meaning and recall |
| METEOR | Matches with stems and synonyms | Still surface-level |
| CIDEr | Consensus-weighted n-gram similarity | Depends on many references |
| SODA | Story-level quality of dense captions | Complex and less widely reported |
| Model-based judging | Factual correctness judged by another model | Inherits the judge’s own biases |
For long, detailed captions, teams increasingly combine model-based scoring with human spot checks, because no single automatic metric captures factual accuracy, completeness and temporal order together.
Limitations and common failure modes
The most serious weakness is multimodal hallucination: models describe objects or actions that are plausible for the scene but absent from the footage. A kitchen video may receive a caption that mentions a knife simply because knives usually appear in kitchens.
- Static bias: models often describe a single representative frame and miss motion, direction and cause.
- Temporal order errors: events are reported in the wrong sequence when frames are sampled sparsely.
- Missed short events: actions shorter than the sampling interval may never be seen at all.
- Counting and identity errors: the number of people or whether the same person reappears is frequently wrong.
- Long-video cost: compute and memory grow with duration, which forces aggressive compression.
A practical rule is to treat automatic captions as drafts that need verification whenever they are used for accessibility, legal records or safety decisions, and to raise the frame sampling rate for footage with fast action.
Stripped to the essentials
Video captioning means a computer watches a video and writes down what happens in ordinary sentences. It does not copy what people say. It describes what can be seen, such as a dog running across a park.
The software looks at a selection of frames. It works out how things change from one moment to the next. Then it puts that into words. Advanced versions split a long video into separate events and give each one a time stamp. The key point: these descriptions are fast and useful. They can still invent details that are not there, so a person should check them when accuracy matters.
Where it is applied
A regional public broadcaster wants to make its archive of 40,000 hours of news footage searchable. Editors previously relied on sparse manual notes, so finding a clip of a specific flood or bridge opening could take hours. The team runs a dense video captioning model over the archive at two frames per second, producing timestamped descriptions such as a crane lifting a steel beam over a river between 01:12 and 01:47.
The captions are indexed alongside speech transcripts, and archivists review a random 2 percent sample to measure error rates. The review finds that most captions are usable, while counting errors and occasional invented objects remain. The outcome is that search time drops from hours to minutes, and archivists correct the captions only for clips that are actually reused on air.