Temporal grounding is the task of locating the start and end timestamps of the video segment that matches a natural-language query, such as “the moment the chef adds salt to the pan”. In multimodal AI it is the time-axis counterpart of spatial visual grounding: instead of pointing to a region in an image, the model points to an interval on the timeline. The task is also called video temporal grounding or moment retrieval, and it is a core capability of modern video-language models.
The output is not a label or a caption but a pair of times, for example 14.2 s to 19.8 s, often with a confidence score. This makes temporal grounding the bridge between understanding what happens in a video and telling a user exactly where it happens, which matters for search, editing, surveillance review and video question answering.
How temporal grounding models locate a moment
A temporal grounding model aligns a text representation of the query with a sequence of video features and then predicts the boundaries of the best-matching span. The video is usually sampled at 1-4 frames per second, each frame or short clip is encoded by a vision encoder, and the query is encoded by a text encoder or a language model.
The two streams meet in a cross-modal fusion step, typically cross-attention, where every time step learns how relevant it is to the query. A prediction head then either scores candidate segments, regresses the start and end offsets directly, or, in generative systems, writes the timestamps as text tokens.
The main architectural families differ in how they produce those boundaries, as the following table shows.
| Approach | How boundaries are produced | Typical strength | Typical weakness |
|---|---|---|---|
| Proposal-based | Score many pre-cut candidate windows | Robust ranking | Slow, fixed window sizes |
| Proposal-free regression | Predict start and end per frame | Fast, flexible lengths | Imprecise on long videos |
| DETR-style query decoding | Learned queries output spans directly | Joint highlight detection | Needs much training data |
| Generative video LLM | Writes timestamps as text | Open-ended queries | Weak numeric precision |
Generative approaches depend heavily on how time is represented. Models that receive explicit time tokens or timestamp embeddings ground moments far better than models that see only an unlabeled stream of visual tokens, because they otherwise have no reliable notion of when a frame occurred.
Benchmarks and evaluation metrics
Temporal grounding is evaluated by comparing the predicted interval with a human-annotated interval using temporal intersection over union (tIoU). The tIoU is the overlap duration divided by the duration of the union of both intervals, so a value of 1.0 means a perfect match.
The headline metric is Recall@1 at a tIoU threshold, written R1@0.5 or R1@0.7: the share of queries for which the top prediction overlaps the ground truth by at least that amount. Mean IoU (mIoU) and mean average precision are also reported, especially when several moments can match one query.
| Dataset | Video domain | Typical video length | Notable property |
|---|---|---|---|
| Charades-STA | Indoor daily activities | About 30 seconds | Short, scripted actions |
| ActivityNet Captions | Varied web videos | About 2 minutes | Dense, overlapping events |
| TACoS | Cooking scenes | Several minutes | Fine-grained kitchen steps |
| QVHighlights | Vlogs and news | About 150 seconds | Moments plus highlight scores |
| Ego4D NLQ | Egocentric footage | 8-minute clips | Very short target windows |
Scores are not comparable across datasets: R1@0.5 can be high on Charades-STA yet low on Ego4D NLQ, where the answer often lasts only a few seconds inside a long clip. Researchers have also shown that some benchmarks contain annotation biases, so a model can score well by predicting typical moment positions without watching the video carefully.
Temporal grounding compared with related tasks
Temporal grounding is one of several video-language tasks, and the differences lie mainly in what the model must output. Treating them as interchangeable leads to wrong benchmark choices and wrong product expectations.
| Task | Input | Output | Key difference |
|---|---|---|---|
| Temporal grounding | One video and a query | Start and end times | Localizes inside a video |
| Visual grounding | One image and a phrase | Bounding box or mask | Spatial, not temporal |
| Cross-modal retrieval | Query and a video collection | Ranked list of videos | Finds which video, not when |
| Video question answering | Video and a question | Text answer | Answer may need no timestamps |
| Video captioning | Video | Descriptive text | Generates language, no query |
| Temporal action localization | Video and a fixed class list | Labeled intervals | Closed vocabulary, no free text |
A combined task, video corpus moment retrieval, first retrieves the right video from a large collection and then grounds the moment inside it. Spatio-temporal grounding goes further and outputs a tube of bounding boxes across the selected frames.
Building and deploying a grounding system
In practice the right design depends on video length, latency budget and query style. Short clips with simple action queries suit compact specialist models, while hours of footage and open-ended questions require hierarchical search or a large video-language model.
| Situation | Recommended setup | Reason |
|---|---|---|
| Short clips, fixed domain | Specialist DETR-style model | High precision, low cost |
| Hour-long recordings | Coarse-to-fine search | Limits frames sent to the model |
| Open, conversational queries | Video LLM with time tokens | Handles complex language |
| Large video library | Retrieval then grounding | Avoids scanning every video |
| Real-time monitoring | Streaming feature cache | Keeps latency bounded |
Long videos are the hardest case. A common pattern is to index segment embeddings once, retrieve a few candidate windows with an embedding search, and then run a more expensive model only on those windows to refine the boundaries.
Limitations and common failure modes
Temporal grounding remains less reliable than spatial grounding, mainly because time is sampled sparsely and human annotators disagree about where an event begins and ends. Typical problems include the following:
- Boundary ambiguity: annotators often differ by one or more seconds, which caps achievable tIoU.
- Sparse frame sampling misses brief actions that fall between sampled frames.
- Positional bias: models learn that moments usually sit near the start or middle of a clip.
- Temporal hallucination: generative models output plausible but invented timestamps.
- Repeated events: queries such as “the person opens the door” may match several intervals.
Robust systems therefore return several ranked candidates with confidence scores and let a user or downstream logic confirm the result instead of trusting a single predicted span.
Stripped to the essentials
Temporal grounding means answering “when” inside a video. You describe a moment in ordinary words, and the system tells you the exact seconds where it starts and ends.
Think of it as a search box for the timeline instead of for a library. A normal video search finds the right film; temporal grounding jumps straight to the right scene in that film. The hard part is that events do not have sharp edges, so even people disagree slightly about where a moment begins.
Where it is applied
A sports broadcaster archives every match as a 95-minute recording. An editor preparing a highlight reel types “the goalkeeper saves a penalty in the second half”. The system samples the footage at two frames per second, retrieves a handful of candidate windows from a precomputed embedding index, and then refines them with a video-language model that outputs timestamped intervals with confidence scores.
The top result points to a span from 71:04 to 71:16, and the editor confirms it in seconds instead of scrubbing through the second half manually. Across a weekend of matches, the editor uses the grounded results as a first draft and cuts review time substantially, while still checking each clip because a few queries return a near-miss such as a regular save instead of the penalty.