In-browser inference is the execution of a trained machine learning model inside the web browser on the user’s own device, so that predictions are computed locally by JavaScript, WebAssembly or WebGPU code rather than by a remote server. Within Edge AI, it is the most portable form of local execution: the model ships like any other web asset and runs on any device with a modern browser.
The approach is also called client-side AI or browser-based AI. Its appeal rests on three properties: data never leaves the device, there is no per-request server cost, and the application can keep working offline after the first load. Its price is a tight budget of memory, download size and compute that shapes every design decision.
How in-browser inference executes a model
In-browser inference works by downloading the model weights and a runtime into the page and then executing the network graph through one of the browser’s compute interfaces. The runtime parses the model file, allocates tensors in browser memory and dispatches each operator to the fastest available execution backend.
A typical pipeline has four stages. The page fetches a converted model file, often in ONNX, TensorFlow.js or GGUF format, and stores it in the Cache Storage API or IndexedDB so later visits skip the download. The runtime then compiles shaders or WebAssembly kernels, preprocesses input such as camera frames or text tokens, runs the forward pass and postprocesses the output. Heavy work is normally moved to a Web Worker so the user interface thread stays responsive.
The following table summarizes the compute interfaces that runtimes use and what each one offers.
| Backend | Hardware used | Strength | Main limitation |
|---|---|---|---|
| WebAssembly (Wasm) | CPU, with SIMD and threads | Runs almost everywhere | Slow for large matrix work |
| WebGL | GPU through graphics shaders | Wide legacy support | Awkward for general compute |
| WebGPU | GPU through compute shaders | Fast matrix multiplication | Uneven support on older devices |
| WebNN | CPU, GPU or NPU via the operating system | Access to neural accelerators | Still maturing across browsers |
Runtimes and model formats used in the browser
Several web inference runtimes package these backends behind a single API. They differ mainly in the model formats they accept and in how well they handle large transformer models, so the choice of runtime often follows from the training framework and the target model size.
| Runtime | Typical model format | Backends | Typical workloads |
|---|---|---|---|
| ONNX Runtime Web | ONNX | Wasm, WebGL, WebGPU, WebNN | Vision, audio, small transformers |
| TensorFlow.js | TF.js graph or layers model | Wasm, WebGL, WebGPU | Pose, object detection, classic CNNs |
| Transformers.js | ONNX exported from Hugging Face | Wasm, WebGPU | Embeddings, speech, text models |
| MediaPipe Tasks | TFLite task bundles | Wasm, WebGL | Face, hand and gesture tracking |
| WebLLM and similar | Compiled or quantized LLM weights | WebGPU | Chat with a local LLM |
Getting a model into one of these formats is a model conversion step that can change numerical behavior. Unsupported operators are a frequent obstacle: a layer that the browser runtime cannot execute on the GPU falls back to the CPU, and a single fallback inside a hot loop can erase most of the speed gain.
Size, memory and speed constraints
The defining constraint of in-browser inference is that every byte of the model must be downloaded before the first prediction. A model that is acceptable as a native app install is often unacceptable as a page asset, because users abandon pages that load slowly and mobile networks may be metered.
For this reason, browser models are compressed aggressively. INT8 quantization cuts weights to about a quarter of their FP32 size, and 4-bit quantization makes small language models of one to three billion parameters feasible on desktop GPUs. Pruning and compact architectures such as MobileNet serve the same goal. The ranges below are indicative and depend heavily on the device and the backend.
| Model class | Typical download | Practical on | Notes |
|---|---|---|---|
| Small vision or audio model | 1-20 MB | Phones and laptops | Real-time at camera frame rates |
| Sentence embedding model | 20-130 MB | Most laptops, many phones | Good fit for local semantic search |
| Speech recognition model | 40-250 MB | Laptops, high-end phones | Latency depends on audio length |
| Small LLM at 4 bits | 0.8-2.5 GB | Desktops with WebGPU | Long first load, needs a capable GPU |
Memory is a second ceiling. Browsers impose per-tab memory limits, Wasm 32-bit memory is capped at 4 GB, and WebGPU buffers are bounded by device limits. Speed varies by an order of magnitude between a desktop GPU and a budget phone, so robust applications detect capabilities at runtime and choose a smaller model or a server path when needed.
How it compares with other ways to run models
In-browser inference is one point on a spectrum that runs from fully remote to fully local execution. Compared with a native edge inference runtime, it trades raw performance and hardware access for zero installation and instant distribution. Compared with cloud inference, it trades model capacity for privacy and cost.
| Approach | Where the model runs | Privacy | Model capacity | Distribution |
|---|---|---|---|---|
| In-browser inference | Browser on the user device | High | Small to medium | Instant through a URL |
| Native on-device inference | Installed app or OS service | High | Medium | App store or installer |
| Cloud inference | Provider data center | Depends on provider | Very large | API call |
| Hybrid inference | Browser first, server as fallback | Mixed | Flexible | URL plus backend |
Many production systems therefore adopt hybrid inference: a small local model handles common or sensitive requests, and harder requests are routed to the server. This mirrors the broader cloud offloading pattern used across Edge AI.
Practical uses and design choices
In-browser inference fits best where latency, privacy or cost per request matter more than maximum model quality. Typical examples include background blur in video calls, live captioning, document scanning, grammar suggestions, on-page semantic search and accessibility features such as image descriptions.
- Start with the smallest model that meets the quality target, then quantize it and measure accuracy again.
- Cache the model file after the first download and version it so updates do not reload unchanged weights.
- Run inference in a Web Worker and stream progress to the interface during long loads.
- Detect WebGPU support at runtime and keep a Wasm fallback for devices without it.
- Provide a server path for devices that cannot meet memory or speed thresholds.
Limitations and common mistakes
The main limitations are hardware fragmentation, large first downloads and the fact that model weights are exposed to anyone who opens the developer tools. A shipped model is effectively public, so proprietary weights need licensing decisions rather than technical protection.
Privacy is also easy to overstate. Local inference keeps inputs on the device only if the page does not send results or telemetry elsewhere, so the claim must be verified at the application level, not assumed from the architecture. Thermal throttling on phones can reduce sustained throughput by half or more during long sessions, which benchmarks run for a few seconds rarely reveal.
In everyday language
In-browser inference means the AI runs inside the web page you have open, on your own computer or phone. Instead of sending your photo, voice or text to a distant server and waiting for an answer, the page downloads a small AI model once and does the work itself.
It is a little like downloading a recipe and cooking at home instead of ordering delivery. Your ingredients never leave your kitchen, and nobody charges you per meal. The catch is that your kitchen is smaller than a restaurant, so the recipe has to be simple enough for the equipment you have.
Seen in the real world
A small legal-services team builds a web tool that lets clients search their own contracts by meaning rather than exact keywords. Because the contracts are confidential, the team decides that no document text may reach a server. They choose a sentence embedding model of about 25 MB after INT8 quantization, run it with a WebGPU backend and a Wasm fallback, and cache it in the browser after the first visit.
On a typical office laptop, the tool embeds a 40-page contract in roughly ten to fifteen seconds and then answers each search almost instantly. The firm avoids hosting a GPU server, clients see that the page works with the network disconnected, and the outcome is a private semantic search with no per-query cost.