Browser agent

Also known as: web agent · browser-use agent

On this page 7

A browser agent is an AI agent that controls a web browser to complete tasks by observing pages and issuing actions such as clicking, typing, scrolling and navigating. Also called a web agent or browser-use agent, it belongs to the AI agents branch of artificial intelligence and combines a language model for reasoning with a browser automation layer that turns decisions into real interactions.

Unlike a scraper that follows fixed rules, a browser agent decides its next step at run time based on what the page shows and on the goal it was given. This makes it useful for websites without an API, but it also introduces new security and reliability risks that do not exist in conventional automation.

How a browser agent perceives and acts on web pages

A browser agent works in an observe, reason, act loop. It captures the current state of the page, asks a model to choose the next action toward the goal, executes that action in the browser, and then observes the result before deciding again.

Page state can be represented in several ways, and the choice strongly affects accuracy, cost and robustness. The table compares the main observation formats.

Observation formatWhat the model receivesStrengthsWeaknesses
Raw HTML or DOMMarkup with element attributesPrecise selectors, full textVery long, noisy, costly in tokens
Accessibility treeRoles, names and states of elementsCompact, semantic, stable referencesMisses visual layout and canvas content
ScreenshotRendered image of the viewportSees layout, icons and imagesNeeds coordinates, harder to act precisely
Set-of-marks hybridScreenshot with numbered element labelsCombines vision with clear targetsExtra preprocessing, labels can overlap
Ways a browser agent observes a page

Actions are usually exposed to the model as a small tool schema: navigate to a URL, click an element, type text, select an option, scroll, go back, and finish with an answer. A controller layer built on frameworks such as Playwright, Puppeteer or the Chrome DevTools Protocol translates each tool call into browser commands.

A browser agent is narrower than a computer-use agent, which controls a whole desktop through screenshots and mouse or keyboard events, and broader than a script that replays recorded steps. The comparison clarifies where each approach fits.

ApproachScope of controlAdapts to page changesTypical use
Browser agentOne browser, its tabs and pagesYes, reasons at run timeOpen-ended web tasks
Computer-use agentEntire operating system desktopYes, from screenshotsTasks spanning several applications
Scripted browser automationBrowser through fixed selectorsNo, breaks on layout changesRegression tests, stable workflows
Robotic process automationRecorded user interface stepsLimitedRepetitive back-office processes
API integrationStructured service endpointsNot applicableReliable data exchange when an API exists
Browser agents and neighboring approaches

The decisive guideline is simple: use an API when one exists, use scripted automation for stable and repeated flows, and use a browser agent when the task varies, the interface changes, or no structured interface is available.

Typical tasks and design choices

Browser agents are applied to research, form filling, data collection and end-to-end testing. They search several sites, compare information, fill multi-page forms from supplied data, and report the outcome, often as part of a larger agentic workflow.

  • Headless versus headed mode: headless browsers run faster on servers, while a visible browser helps debugging and human supervision.
  • Session and identity handling: the agent may run in a fresh profile or in a signed-in profile, which changes both capability and risk.
  • Step budgets: limits of roughly 20 to 50 actions per task prevent loops and runaway costs.
  • Human checkpoints: a human-in-the-loop confirmation before purchases, submissions or messages keeps irreversible actions under control.

Evaluation relies on task success rate on realistic benchmarks, such as suites of simulated shops, forums and content management sites. Reported success rates vary widely with the model, the observation format and the benchmark, and they remain well below human performance on long, multi-site tasks.

Security risks of browser agents

The central security risk of a browser agent is indirect prompt injection: text on a web page, in an email or in a hidden element can contain instructions that the model mistakes for its user’s commands. Because the agent can click and type, a successful injection can lead to data exfiltration or unwanted actions.

Defenses combine model behavior with system design. The table pairs the main threats with practical mitigations.

ThreatExampleMitigation
Indirect prompt injectionHidden text tells the agent to send data elsewhereTreat page content as data, never as instructions
Credential exposureAgent types passwords into a phishing formKeep secrets out of the model, use a password manager
Irreversible actionsUnintended purchase or deleted recordRequire human confirmation for high-impact steps
Excessive permissionsAgent reaches internal admin pagesDomain allowlists and least-privilege profiles
Silent failuresTask reported done when a step failedLogged trajectories and verification of final state
Threats and mitigations for browser agents

Running agents in an isolated agent sandbox with a dedicated browser profile, network restrictions and full action logs is the most effective way to limit the impact of a compromised session.

Reliability limits and common failure modes

Browser agents fail most often on dynamic interfaces: pop-ups, cookie banners, infinite scroll, date pickers and elements that load late. A misread page state can send the agent down a wrong branch that compounds over many steps.

Other limits come from the web itself. CAPTCHAs and bot detection are designed to stop automated clients, and responsible agents hand these to a human rather than attempting to bypass them. Latency is also significant, because each step involves a model call of one to several seconds, so a 30-step task can take minutes.

Reliability improves with clear goals, compact observations, explicit success criteria and agent observability that records each action with its reasoning, so failed trajectories can be replayed and analyzed.

In everyday language

A browser agent is like an assistant who uses the web for you. You describe what you want, and it opens websites, reads them, clicks buttons and fills in boxes, step by step, until the job is done.

It is flexible because it looks at each page and decides what to do next, but it can still be fooled by misleading text or confused by messy pages. That is why important steps should be approved by a person.

Seen in the real world

A procurement team at a mid-sized manufacturer must compare prices for 40 replacement parts across five supplier portals, none of which offers an API. Done by hand, the check takes an analyst most of a day each month.

The team configures a browser agent in a sandboxed browser with a domain allowlist covering only the five portals and read-only supplier accounts. For each part, the agent searches every portal, records price, stock and delivery time, and writes the results into a spreadsheet, with a limit of 30 actions per part.

The run finishes in about two hours without supervision. The agent completes 37 of the 40 lookups; the remaining three involved a new login challenge and an unusual product configurator, which it flagged for review instead of guessing. The analyst verifies the flagged items and spends under an hour on the whole task.