GUI grounding is the task of locating the graphical user interface element that a natural-language instruction refers to, returning either screen coordinates or an element identifier that an agent can click, type into or inspect. In Multimodal AI, it is the perception step that connects language understanding with action on real software, and it is sometimes described more loosely as screen understanding.
A model that performs GUI grounding receives a screenshot, optionally with accessibility metadata, and a phrase such as “open the export settings”. It must decide which button, menu item or text field matches that intent and where it sits on the screen. Without reliable grounding, a computer-use agent can plan perfectly and still fail because it clicks the wrong pixel.
How GUI grounding maps language to screen positions
GUI grounding works by encoding the screenshot with a vision encoder, encoding the instruction as text, and asking a vision-language model to output the target location. Most current systems emit the answer as text tokens, for example a normalized point such as (0.42, 0.17) or a bounding box with four coordinates, which a controller then converts into a mouse event.
The process has three conceptual stages. First, the model must recognize interface elements, including icons without labels, small checkboxes and text rendered in unusual fonts. Second, it must resolve the referring expression, which may be literal (“the Save button”), functional (“the control that saves the file”) or relational (“the field below Email”). Third, it must localize precisely enough that the predicted point falls inside the element bounds.
The following table shows the main input sources a grounding system can use and what each contributes.
| Input signal | What it provides | Main weakness |
|---|---|---|
| Raw screenshot | Universal visual view of any application | Small text and icons lose detail after downscaling |
| Accessibility tree | Element roles, names and exact bounds | Often incomplete, missing or wrong in custom widgets |
| HTML or DOM | Structured web page elements and attributes | Very long, and not available for desktop apps |
| Set-of-marks overlay | Numbered boxes drawn on candidate elements | Depends on a separate detector to propose the boxes |
| Interaction history | Previous actions and screens in the task | Adds context length and can propagate earlier errors |
Output formats and grounding approaches
GUI grounding systems differ mainly in what they predict and which signals they trust. Pure vision approaches treat the screen as an image and predict coordinates directly, which makes them platform independent across Windows, macOS, Android, iOS and the web. Structured approaches select an element from a list extracted from the accessibility tree or the DOM, which is precise but only as good as that metadata.
The table below compares the common approaches and the conditions under which each tends to work best.
| Approach | Model output | Best suited to |
|---|---|---|
| Direct coordinate prediction | Point or box as text tokens | Cross-platform agents working from pixels only |
| Element selection | Index of an element in a candidate list | Web pages and apps with clean accessibility data |
| Set-of-marks prompting | Number of a labeled overlay box | General models without grounding-specific training |
| Two-stage zoom-in | Coarse region, then refined point | High-resolution professional software screens |
| Planner plus grounder | Text plan from one model, coordinates from another | Agents that separate reasoning from localization |
Research models such as SeeClick, CogAgent, UGround, OS-Atlas, ShowUI and UI-TARS were trained specifically for this capability, usually on large synthetic datasets built by rendering web pages and app screens and pairing each element with automatically generated descriptions. General multimodal large language models have improved at grounding, but dedicated training data remains the strongest predictor of accuracy.
GUI grounding compared with related multimodal tasks
GUI grounding is a specialized form of visual grounding, the broader task of linking a phrase to a region of a natural image. Interfaces differ from photographs because they are dense with small text, highly structured, and full of elements whose meaning comes from convention rather than appearance, such as a gear icon that means settings.
| Task | Typical input | Typical output |
|---|---|---|
| GUI grounding | Screenshot and instruction | Click point or element box |
| Visual grounding | Natural image and referring phrase | Object bounding box |
| Document understanding | Scanned page or PDF | Extracted fields, text or answers |
| Visual question answering | Image and question | Text answer, no location required |
| Multimodal agent | Goal, screen and history | Sequence of actions over many steps |
The distinction matters in practice: a model can describe a screen correctly in a visual question answering setting and still miss the target by a few pixels, which is a complete failure for an agent. GUI grounding is therefore evaluated by click accuracy, meaning whether the predicted point lands inside the ground-truth element.
Evaluating GUI grounding accuracy
GUI grounding is measured on benchmarks of annotated screenshots, where each instruction has one correct element. ScreenSpot covers mobile, desktop and web screens with separate scores for text and icon targets, and ScreenSpot-Pro uses high-resolution screenshots of professional applications such as design, engineering and scientific software, where targets often occupy a tiny fraction of the screen.
End-to-end agent benchmarks such as OSWorld, Mind2Web and AndroidWorld measure task completion rather than single clicks, so grounding is only one factor in their scores. Reported accuracy varies widely with screen resolution, target size and whether the target is text or an icon, so results from different benchmarks are not directly comparable.
- Icon targets are consistently harder than text targets because the model cannot match words in the instruction.
- Professional software lowers accuracy sharply because toolbars are crowded and elements are small.
- Image downscaling before encoding is a frequent hidden cause of missed clicks.
Limitations, risks and common failure modes
GUI grounding fails in recognizable patterns, and most of them trace back to resolution, ambiguity or distribution shift. Knowing these patterns helps teams decide when to add accessibility data, cropping or human confirmation.
| Symptom | Likely cause | Mitigation |
|---|---|---|
| Click lands next to a small icon | Downscaled screenshot | Crop and zoom into the candidate region |
| Wrong one of several identical buttons | Ambiguous reference | Add spatial or contextual wording to the instruction |
| Fails on an unfamiliar application | Distribution shift from training data | Fine-tune on screens from that software |
| Clicks a button behind a pop-up | Stale screenshot or overlay ignored | Capture a fresh screen before every action |
| Follows text embedded in a web page | Prompt injection through on-screen content | Restrict actions and require confirmation for risky steps |
Security is a serious concern. Because a grounding model reads everything on the screen, malicious text in a web page or email can steer an agent toward unintended elements. Production systems should treat on-screen text as untrusted data, limit the actions an agent may take, and keep human approval for irreversible operations such as payments or deletions.
Explained without jargon
GUI grounding is how an AI figures out where to click when someone tells it what to do on a computer. A person might say “press the blue Send button”, and the AI has to look at a picture of the screen and find that exact button.
Think of giving directions to a friend who is looking at your screen over a video call. They need to understand what you mean and then point at the right spot. If they point even slightly off, nothing useful happens. Good grounding means pointing precisely, every time.
A typical use case
An accounts team at a mid-sized distributor uses an AI agent to download monthly supplier statements from an old desktop accounting program that has no API and a poor accessibility tree. Each statement requires opening a supplier, choosing a date range of 30 days and pressing an unlabeled export icon in a crowded toolbar.
The first version of the agent sends a downscaled 1920 by 1080 screenshot to a general model and misses the export icon on roughly one attempt in five, often hitting a neighboring print icon. The team switches to a two-stage zoom-in grounder: it first predicts the toolbar region, then re-examines a full-resolution crop and returns the final point.
With the refined grounding, missed clicks become rare enough that the agent processes about 120 statements per run with a person only reviewing the exported files. The team also blocks any action outside the export workflow, so an incorrect click cannot change accounting data.