Visual grounding

Also known as: referring expression comprehension

On this page 8

Visual grounding is the task of linking a natural language expression to the specific region of an image it describes, usually by predicting a bounding box or a segmentation mask. In multimodal AI it is the capability that turns a vague statement such as “the red mug left of the laptop” into pixel coordinates that software can act on, and it is closely related to referring expression comprehension, the name used in much of the research literature.

Grounding matters because a model that can describe an image is not necessarily a model that knows where things are. Within artificial intelligence, grounded outputs make vision-language systems verifiable, reduce multimodal hallucination and provide the spatial anchor that robots, GUI agents and document tools need before they can click, grasp or extract anything.

How visual grounding maps words to regions

A grounding model receives an image and a text query and returns one or more regions with confidence scores. Internally, a vision encoder produces spatial features, a text encoder embeds the query, and a fusion module aligns words with image locations, typically through cross-attention between visual and textual tokens.

The query can be a single noun phrase, a full referring expression with attributes and relations, or a sentence containing several phrases that each need a box. The hard part is rarely detecting objects; it is resolving relations and attributes such as “second from the left”, “the smaller dog” or “the person holding the umbrella”, which require comparison among candidates.

Three architectural families have dominated the field, and the table below shows how each one produces its answer.

ApproachHow it predicts a regionTypical strengthTypical weakness
Two-stage rankingA detector proposes boxes, then a matcher scores each against the textPrecise boxes from a mature detectorFails when the right object is never proposed
One-stage regressionFused features regress box coordinates directlyFaster and end to endWeaker on crowded scenes with many similar objects
Detection-style groundingGrounding is framed as phrase-conditioned object detectionOpen-vocabulary, many phrases at onceNeeds large grounded pretraining data
Generative coordinatesA multimodal language model writes box coordinates as text tokensHandles complex reasoning and dialogCoordinate precision is coarser and can drift
Main architectural approaches to visual grounding

Task variants within visual grounding

Visual grounding is an umbrella term for several closely related tasks that differ in the form of the query and of the output. Benchmarks and model papers often use these names precisely, so confusing them leads to misleading comparisons of reported accuracy.

TaskInput queryOutputRepresentative benchmarks
Referring expression comprehensionOne expression for one objectSingle bounding boxRefCOCO, RefCOCO+, RefCOCOg
Phrase groundingCaption with several noun phrasesOne or more boxes per phraseFlickr30k Entities
Referring expression segmentationOne expression for one objectPixel-level maskRefCOCO-based mask splits
Generalized referring expressionsExpression for zero, one or many objectsAny number of boxes, including nonegRefCOCO
Open-vocabulary grounded detectionList of category names or phrasesAll matching instancesLVIS, ODinW
Common grounding tasks and their outputs

The classic datasets have known quirks. RefCOCO expressions are short and often positional, RefCOCO+ forbids absolute location words to force appearance-based reasoning, and RefCOCOg contains longer, more descriptive sentences. A model that scores well on one split may still struggle on another, so results should be read split by split rather than as a single number.

Measuring grounding accuracy

Grounding quality is usually measured with intersection over union (IoU), the overlap between the predicted and the reference region divided by their combined area. The standard metric for referring expression comprehension is accuracy at IoU 0.5: a prediction counts as correct when it overlaps the annotated box by at least half.

MetricWhat it measuresWhere it is used
Acc@0.5Share of queries with IoU of at least 0.5Referring expression comprehension
Recall@kWhether a correct box appears among the top kPhrase grounding
Mean IoU and cumulative IoUAverage or summed mask overlapReferring segmentation
No-target accuracyCorrect refusal when nothing matchesGeneralized referring tasks
Metrics used to evaluate grounding

The 0.5 threshold is forgiving for large objects and strict for tiny ones, so headline accuracy can hide poor localization of small targets. Production teams therefore often add stricter IoU thresholds such as 0.75 and evaluate small, medium and large objects separately.

Grounding sits between pure recognition and full reasoning, and several neighboring capabilities are easy to confuse with it. The comparison below clarifies what each capability returns and why grounding is the one that answers “where”.

CapabilityCore questionOutputRelation to grounding
Visual groundingWhere is the thing described?Box or maskThe task itself
Object detectionWhere are all objects of fixed classes?Boxes with class labelsGrounding adds free-form language
Visual question answeringWhat is the answer about this image?Text answerOften relies on implicit grounding
Image-text matchingDoes this caption fit this image?Similarity scoreGlobal, not region-level
Temporal groundingWhen does the described event happen?Time interval in a videoSame idea along the time axis
GUI groundingWhich screen element is meant?Click point or element boxGrounding applied to interfaces
Grounding versus neighboring multimodal tasks

Contrastive models such as CLIP align whole images with text but provide no native localization, which is why grounding models add region-level supervision. Modern multimodal large language models increasingly treat grounding as a built-in output format, emitting coordinates inside their text answers so that every claim can be tied to a visible region.

Building and deploying grounding systems

The right design depends on latency, precision and query complexity. A dedicated grounded detector is usually the best choice for fast, high-volume localization of simple phrases, while a multimodal language model is better when the query involves multi-step reasoning or conversation.

  • Training data combines detection labels, referring expression annotations and grounded captions in which phrases are linked to boxes.
  • Input resolution strongly affects small-object grounding; tiling or high-resolution encoders help with documents and screenshots.
  • Coordinate format must be consistent, for example normalized values from 0 to 1 or integer bins from 0 to 1000.
  • Post-processing such as non-maximum suppression and score thresholds decides how many regions are returned.

Limitations and common failure modes

Grounding models share recognizable weaknesses, and most of them come from shortcuts learned from biased datasets. Knowing the symptoms makes evaluation and debugging considerably faster.

SymptomLikely causeMitigation
Picks the most salient objectModel ignores relational wordsTrain on harder negatives with similar objects
Always returns a boxNo training examples without a targetAdd no-target data and a refusal threshold
Left and right confusedHorizontal-flip augmentation without text updateFlip spatial words together with the image
Loose boxes from language modelsCoarse coordinate tokensRefine with a dedicated detector or mask head
Misses small text or iconsLow input resolutionUse crops, tiling or a higher-resolution encoder
Typical grounding failures and remedies

A further risk is apparent grounding without real understanding: a model can return a plausible box for a hallucinated object. Systems that act on predictions, such as robots or agents, should verify the region before acting, for example by re-checking the crop against the query.

Explained without jargon

Visual grounding is the skill of pointing at the right thing in a picture when someone describes it in words. If you tell a friend “pass me the blue folder under the lamp”, they look, find it and point. A grounding model does the same, but it answers by drawing a box around the object.

This matters because describing a photo and knowing where things are in it are different abilities. A model that can point can be checked, and a model that can point can also hand that location to a robot arm or a mouse click.

A typical use case

A warehouse operator runs a picking robot that receives spoken or typed instructions such as “take the dented box on the top shelf, second from the right”. A camera captures the shelf, and a grounding model returns a bounding box for the described carton together with a confidence score. The robot only proceeds when the score exceeds a set threshold and the box lies within its reachable workspace.

During testing, the team finds that the first model often picks the largest box instead of the dented one. They add training images with several similar cartons, include instructions where no box matches, and require the system to ask for clarification when confidence is low. After these changes, wrong picks drop noticeably and operators can correct the remaining cases with a short follow-up phrase instead of manual intervention.