Model offloading

Also known as: CPU offloading · layer offloading

On this page 7

Model offloading is the technique of placing part of a neural network’s weights, activations or optimizer state in slower but larger memory, such as system RAM or an SSD, and moving that data to the accelerator only when it is needed, or computing those layers on the CPU instead. In Edge AI, it is the main way to run a model that is larger than the available GPU or NPU memory on a laptop, workstation or embedded board without sending data to the cloud.

The technique is also called CPU offloading or layer offloading, depending on which resource takes over the work. It trades memory capacity for speed: the model fits, but every byte that crosses a slow bus costs time, so the art lies in deciding what to keep resident and what to move.

How model offloading splits a network across memory tiers

Model offloading works by treating the device as a memory hierarchy and assigning each part of the model to a tier according to how often it is used. The fastest tier, accelerator memory (VRAM or dedicated NPU memory), holds the most frequently accessed tensors; system RAM and storage hold the rest.

There are two basic execution patterns. In compute-where-stored offloading, layers that do not fit on the GPU stay in RAM and the CPU executes them directly, which is how runtimes such as llama.cpp split a language model with a setting for the number of GPU layers. In fetch-on-demand offloading, weights are streamed to the accelerator just before use and evicted afterward, so the GPU performs all computation but waits for transfers.

The following table summarizes the memory tiers involved and their typical characteristics on current consumer hardware.

TierTypical capacityTypical bandwidthRole in offloading
GPU or NPU memory4-24 GB on consumer devices200-1,000 GB/sHolds hot layers and the KV cache
System RAM16-128 GB50-120 GB/s (dual-channel DDR5)Holds offloaded layers for CPU compute or streaming
PCIe linkNot a storage tierAbout 16-32 GB/s per directionCarries weights between RAM and a discrete GPU
NVMe SSD0.5-4 TB3-7 GB/s sequential readLast resort for weights or memory-mapped files
Typical memory tiers used in model offloading

Pipelining reduces the penalty: a well-designed offloading runtime prefetches the next layer while the current one computes, so transfer and computation overlap. Overlap only hides latency when the compute time per layer is at least as long as the transfer time, which is rarely true for single-user token generation.

Variants of offloading and what each moves

Offloading variants differ in which tensors leave the accelerator and whether the goal is inference or training. Choosing the variant determines both the memory saved and the slowdown incurred.

VariantWhat is movedTypical useMain cost
Layer offloadingWhole transformer blocks or layersLocal LLM inference on limited VRAMSlower layers run on CPU
Weight streamingWeights copied to GPU per forward passLarge models on one GPUPCIe bandwidth per token
KV cache offloadingAttention keys and valuesLong contexts, many sessionsLatency when the cache is recalled
Optimizer state offloadingAdam moments and master weightsFine-tuning on one deviceCPU update time per step
Expert offloadingInactive mixture-of-experts weightsSparse MoE modelsStalls when a cold expert is chosen
Main forms of model offloading

Training-oriented approaches, such as the ZeRO-Offload and ZeRO-Infinity strategies in DeepSpeed, move optimizer state and gradients to the CPU or NVMe and perform the parameter update there. Inference-oriented approaches, such as automatic device maps in Hugging Face Accelerate, place layers across GPU, CPU and disk in order of priority.

Model offloading is a local memory-placement technique, which distinguishes it from methods that shrink the model or move work off the device. It is often combined with them rather than used instead of them.

ApproachCore ideaData leaves the deviceEffect on accuracy
Model offloadingSpread one model across local memory tiersNoNone
Cloud offloadingSend inputs to a remote server for inferenceYesNone, or better with a larger model
Split computingRun early layers locally, later layers remotelyYes, as intermediate featuresNone
4-bit quantizationStore weights with fewer bitsNoSmall loss, model dependent
Structured pruningRemove channels or headsNoLoss unless the model is retrained
How offloading differs from neighboring approaches

The naming overlap with cloud offloading causes confusion: in the cloud sense, “offloading” means delegating computation to another machine, while model offloading keeps everything on one machine and only moves data between its own memories. In practice, a quantized model with partial offloading is the most common configuration for running a local LLM on a consumer laptop.

Configuring offloading for acceptable speed

Good configuration starts from the fact that autoregressive decoding is memory-bandwidth bound: each generated token requires reading every active weight once. Throughput therefore falls roughly in proportion to the share of weights served from the slowest tier.

As a rough illustration, a model whose 4-bit weights occupy about 8 GB can read them from VRAM at several hundred GB/s, but streaming the same weights over a PCIe 4.0 link at about 25 GB/s effective caps generation near three tokens per second, before any compute time. CPU execution from fast DDR5 can be faster than streaming, which is why layer splitting often beats weight streaming for single-user chat.

SituationRecommended choiceReason
Model slightly exceeds VRAMOffload a few final layers to CPUSmall slowdown, no quality change
Model far exceeds VRAMQuantize first, then offloadFewer bytes cross the slow bus
Long context fills memoryOffload or quantize the KV cacheCache grows with sequence length
Batch inference at high volumeWeight streaming with large batchesEach transfer is reused across many inputs
Unified memory deviceAvoid offloading if the model fitsCPU and GPU share one memory pool
Decision guide for offloading configuration

Unified memory architectures, found in many laptop and phone systems-on-chip, change the calculation because CPU, GPU and NPU share one physical memory. There, offloading mostly concerns storage-to-RAM paging, and the practical limit is total memory capacity rather than a separate VRAM budget.

Limitations, risks and common mistakes

The central limitation is latency: offloading makes a model run, but it rarely makes it run well. Practitioners should treat it as a capacity tool, not a performance tool, and measure tokens per second before committing to a design.

  • Thrashing: when offloaded weights are evicted and reloaded every step, throughput can collapse by an order of magnitude.
  • Pinned memory limits: fast transfers need page-locked RAM, and allocating too much of it can starve the operating system.
  • SSD wear and heat: repeated weight streaming from storage is slow and adds sustained load on small fanless devices.
  • Energy cost: moving data across buses consumes more energy than computing on it, which matters for battery-powered edge devices.
  • Misread benchmarks: prompt processing (prefill) speed is compute bound and can look fine while generation speed remains poor.

A frequent mistake is choosing layers to offload by position rather than by cost. Offloading decisions should follow measured size and access frequency, and the embedding and output layers deserve particular attention because their size depends on vocabulary rather than model depth.

In plain words

Think of a small desk next to a large bookshelf. The desk is the fast memory of the graphics chip, and the bookshelf is the computer’s ordinary memory. If a project needs more books than fit on the desk, you keep some on the shelf and fetch them when you need them.

Model offloading does the same with an AI model that is too big for the fast chip. The model still works and gives the same answers, but every trip to the shelf takes time. It makes big models possible on small machines, at the price of speed.

A real-world example

A legal team wants to summarize confidential contracts without sending them to an external service. Their workstation has a GPU with 12 GB of memory and 64 GB of system RAM, and the chosen open-weight language model needs about 20 GB at 4-bit precision.

The engineer loads 26 of the 40 layers onto the GPU and leaves the remaining 14 layers in RAM, where the CPU computes them. Prompt processing stays fast, while generation settles at roughly 6 to 8 tokens per second instead of the 30 or more that a fully resident model would reach.

For overnight batch summaries this speed is acceptable. The documents never leave the building, and the team avoids buying a larger GPU until usage justifies it.