In-browser inference

Also known as: client-side AI · browser-based AI

On this page 8

In-browser inference is the execution of a trained machine learning model inside the web browser on the user’s own device, so that predictions are computed locally by JavaScript, WebAssembly or WebGPU code rather than by a remote server. Within Edge AI, it is the most portable form of local execution: the model ships like any other web asset and runs on any device with a modern browser.

The approach is also called client-side AI or browser-based AI. Its appeal rests on three properties: data never leaves the device, there is no per-request server cost, and the application can keep working offline after the first load. Its price is a tight budget of memory, download size and compute that shapes every design decision.

How in-browser inference executes a model

In-browser inference works by downloading the model weights and a runtime into the page and then executing the network graph through one of the browser’s compute interfaces. The runtime parses the model file, allocates tensors in browser memory and dispatches each operator to the fastest available execution backend.

A typical pipeline has four stages. The page fetches a converted model file, often in ONNX, TensorFlow.js or GGUF format, and stores it in the Cache Storage API or IndexedDB so later visits skip the download. The runtime then compiles shaders or WebAssembly kernels, preprocesses input such as camera frames or text tokens, runs the forward pass and postprocesses the output. Heavy work is normally moved to a Web Worker so the user interface thread stays responsive.

The following table summarizes the compute interfaces that runtimes use and what each one offers.

BackendHardware usedStrengthMain limitation
WebAssembly (Wasm)CPU, with SIMD and threadsRuns almost everywhereSlow for large matrix work
WebGLGPU through graphics shadersWide legacy supportAwkward for general compute
WebGPUGPU through compute shadersFast matrix multiplicationUneven support on older devices
WebNNCPU, GPU or NPU via the operating systemAccess to neural acceleratorsStill maturing across browsers
Browser compute backends for model execution

Runtimes and model formats used in the browser

Several web inference runtimes package these backends behind a single API. They differ mainly in the model formats they accept and in how well they handle large transformer models, so the choice of runtime often follows from the training framework and the target model size.

RuntimeTypical model formatBackendsTypical workloads
ONNX Runtime WebONNXWasm, WebGL, WebGPU, WebNNVision, audio, small transformers
TensorFlow.jsTF.js graph or layers modelWasm, WebGL, WebGPUPose, object detection, classic CNNs
Transformers.jsONNX exported from Hugging FaceWasm, WebGPUEmbeddings, speech, text models
MediaPipe TasksTFLite task bundlesWasm, WebGLFace, hand and gesture tracking
WebLLM and similarCompiled or quantized LLM weightsWebGPUChat with a local LLM
Common runtimes for browser-based AI

Getting a model into one of these formats is a model conversion step that can change numerical behavior. Unsupported operators are a frequent obstacle: a layer that the browser runtime cannot execute on the GPU falls back to the CPU, and a single fallback inside a hot loop can erase most of the speed gain.

Size, memory and speed constraints

The defining constraint of in-browser inference is that every byte of the model must be downloaded before the first prediction. A model that is acceptable as a native app install is often unacceptable as a page asset, because users abandon pages that load slowly and mobile networks may be metered.

For this reason, browser models are compressed aggressively. INT8 quantization cuts weights to about a quarter of their FP32 size, and 4-bit quantization makes small language models of one to three billion parameters feasible on desktop GPUs. Pruning and compact architectures such as MobileNet serve the same goal. The ranges below are indicative and depend heavily on the device and the backend.

Model classTypical downloadPractical onNotes
Small vision or audio model1-20 MBPhones and laptopsReal-time at camera frame rates
Sentence embedding model20-130 MBMost laptops, many phonesGood fit for local semantic search
Speech recognition model40-250 MBLaptops, high-end phonesLatency depends on audio length
Small LLM at 4 bits0.8-2.5 GBDesktops with WebGPULong first load, needs a capable GPU
Indicative model sizes and their practicality in the browser

Memory is a second ceiling. Browsers impose per-tab memory limits, Wasm 32-bit memory is capped at 4 GB, and WebGPU buffers are bounded by device limits. Speed varies by an order of magnitude between a desktop GPU and a budget phone, so robust applications detect capabilities at runtime and choose a smaller model or a server path when needed.

How it compares with other ways to run models

In-browser inference is one point on a spectrum that runs from fully remote to fully local execution. Compared with a native edge inference runtime, it trades raw performance and hardware access for zero installation and instant distribution. Compared with cloud inference, it trades model capacity for privacy and cost.

ApproachWhere the model runsPrivacyModel capacityDistribution
In-browser inferenceBrowser on the user deviceHighSmall to mediumInstant through a URL
Native on-device inferenceInstalled app or OS serviceHighMediumApp store or installer
Cloud inferenceProvider data centerDepends on providerVery largeAPI call
Hybrid inferenceBrowser first, server as fallbackMixedFlexibleURL plus backend
In-browser inference against related deployment options

Many production systems therefore adopt hybrid inference: a small local model handles common or sensitive requests, and harder requests are routed to the server. This mirrors the broader cloud offloading pattern used across Edge AI.

Practical uses and design choices

In-browser inference fits best where latency, privacy or cost per request matter more than maximum model quality. Typical examples include background blur in video calls, live captioning, document scanning, grammar suggestions, on-page semantic search and accessibility features such as image descriptions.

  • Start with the smallest model that meets the quality target, then quantize it and measure accuracy again.
  • Cache the model file after the first download and version it so updates do not reload unchanged weights.
  • Run inference in a Web Worker and stream progress to the interface during long loads.
  • Detect WebGPU support at runtime and keep a Wasm fallback for devices without it.
  • Provide a server path for devices that cannot meet memory or speed thresholds.

Limitations and common mistakes

The main limitations are hardware fragmentation, large first downloads and the fact that model weights are exposed to anyone who opens the developer tools. A shipped model is effectively public, so proprietary weights need licensing decisions rather than technical protection.

Privacy is also easy to overstate. Local inference keeps inputs on the device only if the page does not send results or telemetry elsewhere, so the claim must be verified at the application level, not assumed from the architecture. Thermal throttling on phones can reduce sustained throughput by half or more during long sessions, which benchmarks run for a few seconds rarely reveal.

In everyday language

In-browser inference means the AI runs inside the web page you have open, on your own computer or phone. Instead of sending your photo, voice or text to a distant server and waiting for an answer, the page downloads a small AI model once and does the work itself.

It is a little like downloading a recipe and cooking at home instead of ordering delivery. Your ingredients never leave your kitchen, and nobody charges you per meal. The catch is that your kitchen is smaller than a restaurant, so the recipe has to be simple enough for the equipment you have.

Seen in the real world

A small legal-services team builds a web tool that lets clients search their own contracts by meaning rather than exact keywords. Because the contracts are confidential, the team decides that no document text may reach a server. They choose a sentence embedding model of about 25 MB after INT8 quantization, run it with a WebGPU backend and a Wasm fallback, and cache it in the browser after the first visit.

On a typical office laptop, the tool embeds a 40-page contract in roughly ten to fifteen seconds and then answers each search almost instantly. The firm avoids hosting a GPU server, clients see that the page works with the network disconnected, and the outcome is a private semantic search with no per-query cost.