GUI grounding

Also known as: screen understanding

On this page 7

GUI grounding is the task of locating the graphical user interface element that a natural-language instruction refers to, returning either screen coordinates or an element identifier that an agent can click, type into or inspect. In Multimodal AI, it is the perception step that connects language understanding with action on real software, and it is sometimes described more loosely as screen understanding.

A model that performs GUI grounding receives a screenshot, optionally with accessibility metadata, and a phrase such as “open the export settings”. It must decide which button, menu item or text field matches that intent and where it sits on the screen. Without reliable grounding, a computer-use agent can plan perfectly and still fail because it clicks the wrong pixel.

How GUI grounding maps language to screen positions

GUI grounding works by encoding the screenshot with a vision encoder, encoding the instruction as text, and asking a vision-language model to output the target location. Most current systems emit the answer as text tokens, for example a normalized point such as (0.42, 0.17) or a bounding box with four coordinates, which a controller then converts into a mouse event.

The process has three conceptual stages. First, the model must recognize interface elements, including icons without labels, small checkboxes and text rendered in unusual fonts. Second, it must resolve the referring expression, which may be literal (“the Save button”), functional (“the control that saves the file”) or relational (“the field below Email”). Third, it must localize precisely enough that the predicted point falls inside the element bounds.

The following table shows the main input sources a grounding system can use and what each contributes.

Input signalWhat it providesMain weakness
Raw screenshotUniversal visual view of any applicationSmall text and icons lose detail after downscaling
Accessibility treeElement roles, names and exact boundsOften incomplete, missing or wrong in custom widgets
HTML or DOMStructured web page elements and attributesVery long, and not available for desktop apps
Set-of-marks overlayNumbered boxes drawn on candidate elementsDepends on a separate detector to propose the boxes
Interaction historyPrevious actions and screens in the taskAdds context length and can propagate earlier errors
Input signals used for GUI grounding

Output formats and grounding approaches

GUI grounding systems differ mainly in what they predict and which signals they trust. Pure vision approaches treat the screen as an image and predict coordinates directly, which makes them platform independent across Windows, macOS, Android, iOS and the web. Structured approaches select an element from a list extracted from the accessibility tree or the DOM, which is precise but only as good as that metadata.

The table below compares the common approaches and the conditions under which each tends to work best.

ApproachModel outputBest suited to
Direct coordinate predictionPoint or box as text tokensCross-platform agents working from pixels only
Element selectionIndex of an element in a candidate listWeb pages and apps with clean accessibility data
Set-of-marks promptingNumber of a labeled overlay boxGeneral models without grounding-specific training
Two-stage zoom-inCoarse region, then refined pointHigh-resolution professional software screens
Planner plus grounderText plan from one model, coordinates from anotherAgents that separate reasoning from localization
Common GUI grounding approaches

Research models such as SeeClick, CogAgent, UGround, OS-Atlas, ShowUI and UI-TARS were trained specifically for this capability, usually on large synthetic datasets built by rendering web pages and app screens and pairing each element with automatically generated descriptions. General multimodal large language models have improved at grounding, but dedicated training data remains the strongest predictor of accuracy.

GUI grounding is a specialized form of visual grounding, the broader task of linking a phrase to a region of a natural image. Interfaces differ from photographs because they are dense with small text, highly structured, and full of elements whose meaning comes from convention rather than appearance, such as a gear icon that means settings.

TaskTypical inputTypical output
GUI groundingScreenshot and instructionClick point or element box
Visual groundingNatural image and referring phraseObject bounding box
Document understandingScanned page or PDFExtracted fields, text or answers
Visual question answeringImage and questionText answer, no location required
Multimodal agentGoal, screen and historySequence of actions over many steps
How GUI grounding differs from neighboring tasks

The distinction matters in practice: a model can describe a screen correctly in a visual question answering setting and still miss the target by a few pixels, which is a complete failure for an agent. GUI grounding is therefore evaluated by click accuracy, meaning whether the predicted point lands inside the ground-truth element.

Evaluating GUI grounding accuracy

GUI grounding is measured on benchmarks of annotated screenshots, where each instruction has one correct element. ScreenSpot covers mobile, desktop and web screens with separate scores for text and icon targets, and ScreenSpot-Pro uses high-resolution screenshots of professional applications such as design, engineering and scientific software, where targets often occupy a tiny fraction of the screen.

End-to-end agent benchmarks such as OSWorld, Mind2Web and AndroidWorld measure task completion rather than single clicks, so grounding is only one factor in their scores. Reported accuracy varies widely with screen resolution, target size and whether the target is text or an icon, so results from different benchmarks are not directly comparable.

  • Icon targets are consistently harder than text targets because the model cannot match words in the instruction.
  • Professional software lowers accuracy sharply because toolbars are crowded and elements are small.
  • Image downscaling before encoding is a frequent hidden cause of missed clicks.

Limitations, risks and common failure modes

GUI grounding fails in recognizable patterns, and most of them trace back to resolution, ambiguity or distribution shift. Knowing these patterns helps teams decide when to add accessibility data, cropping or human confirmation.

SymptomLikely causeMitigation
Click lands next to a small iconDownscaled screenshotCrop and zoom into the candidate region
Wrong one of several identical buttonsAmbiguous referenceAdd spatial or contextual wording to the instruction
Fails on an unfamiliar applicationDistribution shift from training dataFine-tune on screens from that software
Clicks a button behind a pop-upStale screenshot or overlay ignoredCapture a fresh screen before every action
Follows text embedded in a web pagePrompt injection through on-screen contentRestrict actions and require confirmation for risky steps
Typical failure modes and mitigations

Security is a serious concern. Because a grounding model reads everything on the screen, malicious text in a web page or email can steer an agent toward unintended elements. Production systems should treat on-screen text as untrusted data, limit the actions an agent may take, and keep human approval for irreversible operations such as payments or deletions.

Explained without jargon

GUI grounding is how an AI figures out where to click when someone tells it what to do on a computer. A person might say “press the blue Send button”, and the AI has to look at a picture of the screen and find that exact button.

Think of giving directions to a friend who is looking at your screen over a video call. They need to understand what you mean and then point at the right spot. If they point even slightly off, nothing useful happens. Good grounding means pointing precisely, every time.

A typical use case

An accounts team at a mid-sized distributor uses an AI agent to download monthly supplier statements from an old desktop accounting program that has no API and a poor accessibility tree. Each statement requires opening a supplier, choosing a date range of 30 days and pressing an unlabeled export icon in a crowded toolbar.

The first version of the agent sends a downscaled 1920 by 1080 screenshot to a general model and misses the export icon on roughly one attempt in five, often hitting a neighboring print icon. The team switches to a two-stage zoom-in grounder: it first predicts the toolbar region, then re-examines a full-resolution crop and returns the final point.

With the refined grounding, missed clicks become rare enough that the agent processes about 120 statements per run with a person only reviewing the exported files. The team also blocks any action outside the export workflow, so an incorrect click cannot change accounting data.