Model offloading is the technique of placing part of a neural network’s weights, activations or optimizer state in slower but larger memory, such as system RAM or an SSD, and moving that data to the accelerator only when it is needed, or computing those layers on the CPU instead. In Edge AI, it is the main way to run a model that is larger than the available GPU or NPU memory on a laptop, workstation or embedded board without sending data to the cloud.
The technique is also called CPU offloading or layer offloading, depending on which resource takes over the work. It trades memory capacity for speed: the model fits, but every byte that crosses a slow bus costs time, so the art lies in deciding what to keep resident and what to move.
How model offloading splits a network across memory tiers
Model offloading works by treating the device as a memory hierarchy and assigning each part of the model to a tier according to how often it is used. The fastest tier, accelerator memory (VRAM or dedicated NPU memory), holds the most frequently accessed tensors; system RAM and storage hold the rest.
There are two basic execution patterns. In compute-where-stored offloading, layers that do not fit on the GPU stay in RAM and the CPU executes them directly, which is how runtimes such as llama.cpp split a language model with a setting for the number of GPU layers. In fetch-on-demand offloading, weights are streamed to the accelerator just before use and evicted afterward, so the GPU performs all computation but waits for transfers.
The following table summarizes the memory tiers involved and their typical characteristics on current consumer hardware.
| Tier | Typical capacity | Typical bandwidth | Role in offloading |
|---|---|---|---|
| GPU or NPU memory | 4-24 GB on consumer devices | 200-1,000 GB/s | Holds hot layers and the KV cache |
| System RAM | 16-128 GB | 50-120 GB/s (dual-channel DDR5) | Holds offloaded layers for CPU compute or streaming |
| PCIe link | Not a storage tier | About 16-32 GB/s per direction | Carries weights between RAM and a discrete GPU |
| NVMe SSD | 0.5-4 TB | 3-7 GB/s sequential read | Last resort for weights or memory-mapped files |
Pipelining reduces the penalty: a well-designed offloading runtime prefetches the next layer while the current one computes, so transfer and computation overlap. Overlap only hides latency when the compute time per layer is at least as long as the transfer time, which is rarely true for single-user token generation.
Variants of offloading and what each moves
Offloading variants differ in which tensors leave the accelerator and whether the goal is inference or training. Choosing the variant determines both the memory saved and the slowdown incurred.
| Variant | What is moved | Typical use | Main cost |
|---|---|---|---|
| Layer offloading | Whole transformer blocks or layers | Local LLM inference on limited VRAM | Slower layers run on CPU |
| Weight streaming | Weights copied to GPU per forward pass | Large models on one GPU | PCIe bandwidth per token |
| KV cache offloading | Attention keys and values | Long contexts, many sessions | Latency when the cache is recalled |
| Optimizer state offloading | Adam moments and master weights | Fine-tuning on one device | CPU update time per step |
| Expert offloading | Inactive mixture-of-experts weights | Sparse MoE models | Stalls when a cold expert is chosen |
Training-oriented approaches, such as the ZeRO-Offload and ZeRO-Infinity strategies in DeepSpeed, move optimizer state and gradients to the CPU or NVMe and perform the parameter update there. Inference-oriented approaches, such as automatic device maps in Hugging Face Accelerate, place layers across GPU, CPU and disk in order of priority.
Model offloading compared with related Edge AI techniques
Model offloading is a local memory-placement technique, which distinguishes it from methods that shrink the model or move work off the device. It is often combined with them rather than used instead of them.
| Approach | Core idea | Data leaves the device | Effect on accuracy |
|---|---|---|---|
| Model offloading | Spread one model across local memory tiers | No | None |
| Cloud offloading | Send inputs to a remote server for inference | Yes | None, or better with a larger model |
| Split computing | Run early layers locally, later layers remotely | Yes, as intermediate features | None |
| 4-bit quantization | Store weights with fewer bits | No | Small loss, model dependent |
| Structured pruning | Remove channels or heads | No | Loss unless the model is retrained |
The naming overlap with cloud offloading causes confusion: in the cloud sense, “offloading” means delegating computation to another machine, while model offloading keeps everything on one machine and only moves data between its own memories. In practice, a quantized model with partial offloading is the most common configuration for running a local LLM on a consumer laptop.
Configuring offloading for acceptable speed
Good configuration starts from the fact that autoregressive decoding is memory-bandwidth bound: each generated token requires reading every active weight once. Throughput therefore falls roughly in proportion to the share of weights served from the slowest tier.
As a rough illustration, a model whose 4-bit weights occupy about 8 GB can read them from VRAM at several hundred GB/s, but streaming the same weights over a PCIe 4.0 link at about 25 GB/s effective caps generation near three tokens per second, before any compute time. CPU execution from fast DDR5 can be faster than streaming, which is why layer splitting often beats weight streaming for single-user chat.
| Situation | Recommended choice | Reason |
|---|---|---|
| Model slightly exceeds VRAM | Offload a few final layers to CPU | Small slowdown, no quality change |
| Model far exceeds VRAM | Quantize first, then offload | Fewer bytes cross the slow bus |
| Long context fills memory | Offload or quantize the KV cache | Cache grows with sequence length |
| Batch inference at high volume | Weight streaming with large batches | Each transfer is reused across many inputs |
| Unified memory device | Avoid offloading if the model fits | CPU and GPU share one memory pool |
Unified memory architectures, found in many laptop and phone systems-on-chip, change the calculation because CPU, GPU and NPU share one physical memory. There, offloading mostly concerns storage-to-RAM paging, and the practical limit is total memory capacity rather than a separate VRAM budget.
Limitations, risks and common mistakes
The central limitation is latency: offloading makes a model run, but it rarely makes it run well. Practitioners should treat it as a capacity tool, not a performance tool, and measure tokens per second before committing to a design.
- Thrashing: when offloaded weights are evicted and reloaded every step, throughput can collapse by an order of magnitude.
- Pinned memory limits: fast transfers need page-locked RAM, and allocating too much of it can starve the operating system.
- SSD wear and heat: repeated weight streaming from storage is slow and adds sustained load on small fanless devices.
- Energy cost: moving data across buses consumes more energy than computing on it, which matters for battery-powered edge devices.
- Misread benchmarks: prompt processing (prefill) speed is compute bound and can look fine while generation speed remains poor.
A frequent mistake is choosing layers to offload by position rather than by cost. Offloading decisions should follow measured size and access frequency, and the embedding and output layers deserve particular attention because their size depends on vocabulary rather than model depth.
In plain words
Think of a small desk next to a large bookshelf. The desk is the fast memory of the graphics chip, and the bookshelf is the computer’s ordinary memory. If a project needs more books than fit on the desk, you keep some on the shelf and fetch them when you need them.
Model offloading does the same with an AI model that is too big for the fast chip. The model still works and gives the same answers, but every trip to the shelf takes time. It makes big models possible on small machines, at the price of speed.
A real-world example
A legal team wants to summarize confidential contracts without sending them to an external service. Their workstation has a GPU with 12 GB of memory and 64 GB of system RAM, and the chosen open-weight language model needs about 20 GB at 4-bit precision.
The engineer loads 26 of the 40 layers onto the GPU and leaves the remaining 14 layers in RAM, where the CPU computes them. Prompt processing stays fast, while generation settles at roughly 6 to 8 tokens per second instead of the 30 or more that a fully resident model would reach.
For overnight batch summaries this speed is acceptable. The documents never leave the building, and the team avoids buying a larger GPU until usage justifies it.