A local LLM is a large language model that runs entirely on hardware the user controls, such as a laptop, workstation, phone or on-premises server, so that prompts, documents and generated text never leave the device. In Edge AI, running LLMs locally is the most demanding form of on-device inference, because a model with billions of parameters must fit into limited memory, power and compute budgets.
The approach rests on two developments: open-weight models whose parameters can be downloaded, and aggressive compression, above all quantization, that shrinks those models to a few gigabytes. The result trades some quality and speed for privacy, offline availability, predictable cost and full control over the model and its behavior.
How a local LLM generates text on a device
A local LLM produces text through the same autoregressive decoding loop as a cloud model, but every step runs on local silicon. An inference runtime loads the weight file into memory, tokenizes the prompt, processes it in one parallel prefill phase, and then generates one token at a time in the decode phase, appending each token to the context.
The two phases stress hardware differently. Prefill is compute-bound and benefits from GPU or NPU throughput, while decoding is memory-bandwidth-bound: for every new token, the runtime must read essentially all active weights from memory. A useful rule of thumb is that decode speed is roughly memory bandwidth divided by model size in bytes, which explains why a 4 GB model on a machine with 100 GB/s of bandwidth tops out near 25 tokens per second.
Alongside the weights, the runtime keeps a key-value cache (KV cache) that stores attention states for every token in the context. Its size grows linearly with context length, so a long conversation or a large document can consume several additional gigabytes, which is often the hidden reason a model that loads successfully later runs out of memory.
The table below lists the main components of a typical local deployment and what each one contributes.
| Component | Role | Typical examples |
|---|---|---|
| Model weights | Learned parameters, usually quantized | GGUF, safetensors, MLX files |
| Inference runtime | Loads weights, schedules compute, manages KV cache | llama.cpp, MLC LLM, ONNX Runtime |
| Hardware delegate | Offloads matrix math to an accelerator | CUDA, Metal, Vulkan, NPU drivers |
| Tokenizer | Converts text to token IDs and back | BPE or SentencePiece vocabulary |
| Serving layer | Exposes a chat interface or local API | Desktop app, OpenAI-compatible endpoint |
Memory budgets and model sizes
Memory is the decisive constraint for running LLMs locally. Weight memory is approximately the parameter count multiplied by the bytes per parameter, so a 7-billion-parameter model needs about 14 GB at 16-bit precision but only about 4 GB at 4-bit precision, plus overhead for the KV cache and runtime buffers.
The following table gives approximate memory needs for common size classes at different precisions. Real figures vary by architecture, quantization format and context length.
| Model size | 16-bit weights | 8-bit weights | 4-bit weights | Typical device |
|---|---|---|---|---|
| 1-3 billion parameters | 2-6 GB | 1-3 GB | 0.6-1.8 GB | Phones, small laptops |
| 7-9 billion parameters | 14-18 GB | 7-9 GB | 4-5.5 GB | Laptops with 16 GB RAM |
| 13-14 billion parameters | 26-28 GB | 13-14 GB | 7.5-8.5 GB | Consumer GPUs, 32 GB systems |
| 30-34 billion parameters | 60-68 GB | 30-34 GB | 17-20 GB | 24 GB GPUs, unified-memory desktops |
| 70 billion parameters | about 140 GB | about 70 GB | 35-40 GB | Multi-GPU or high-memory workstations |
Unified memory architectures, in which CPU and GPU share one pool, are popular for local inference because they allow large models without a discrete GPU, although their bandwidth is still lower than that of high-end graphics memory. On phones, the operating system usually limits a single app to a fraction of total RAM, which keeps practical on-device models in the 1-4 billion parameter range.
Compression techniques that make local inference possible
Weight-only quantization is the workhorse of local deployment. It stores weights in 4 to 8 bits while computing activations at higher precision, which cuts memory and bandwidth by two to four times with modest quality loss. Methods such as GPTQ and AWQ choose quantized values that minimize output error, and formats like GGUF mix bit widths across layers.
Other techniques complement quantization: knowledge distillation trains a smaller model to imitate a larger one, structured pruning removes whole attention heads or channels, and speculative decoding uses a tiny draft model to propose tokens that the main model verifies in parallel. The table below compares their effect.
| Technique | Main benefit | Main cost or risk |
|---|---|---|
| 4-bit quantization | About 4x smaller than 16-bit | Small accuracy loss, larger on reasoning tasks |
| 8-bit quantization | About 2x smaller, near-lossless | Less savings than 4-bit |
| Distillation | Smaller model with retained skills | Requires a training pipeline and data |
| Structured pruning | Fewer operations on standard hardware | Needs fine-tuning to recover quality |
| Speculative decoding | Faster decoding without quality change | Extra memory for the draft model |
Local LLM compared with cloud and hybrid inference
A local LLM differs from a cloud-hosted model mainly in where data and cost reside. Cloud APIs offer the largest frontier models and elastic capacity, while local execution offers data sovereignty and zero per-token fees. Hybrid inference combines both, routing simple or sensitive requests to the device and hard requests to the cloud.
| Criterion | Local LLM | Cloud API | Hybrid inference |
|---|---|---|---|
| Data privacy | Data stays on device | Data sent to provider | Depends on routing policy |
| Model quality ceiling | Limited by local memory | Largest available models | High for routed requests |
| Latency | No network round trip | Network plus queue delay | Variable |
| Cost model | Up-front hardware, energy | Per-token or subscription | Mixed |
| Offline use | Fully supported | Not supported | Partial |
Limitations and common mistakes
Running LLMs locally carries real trade-offs. Smaller quantized models hallucinate more, follow complex instructions less reliably and have weaker multilingual and coding abilities than frontier systems. Sustained inference on laptops and phones causes thermal throttling and battery drain, so benchmark speeds measured over ten seconds rarely hold over ten minutes.
- Ignoring the KV cache: sizing hardware for weights alone leads to out-of-memory errors with long contexts.
- Over-quantizing: dropping below 3-4 bits often causes a sharp quality collapse, especially for small models.
- Assuming local means secure: downloaded weights and runtimes are supply-chain risks and should come from verified sources.
- Overlooking license terms: open-weight models carry licenses that may restrict commercial use or require attribution.
In simple terms
A local LLM is a chatbot brain that lives on your own computer or phone instead of on a company’s servers. When you type a question, the answer is worked out right there on your device. That means your words never travel over the internet.
To fit on ordinary hardware, these models are squeezed. It is a bit like compressing a large photo so it takes less space. The squeezed version is slightly less capable than the giant cloud models. Still, it works without a connection. It costs nothing per question and keeps private material private.
A practical example
A small law firm wants help summarizing confidential contracts but cannot send client documents to an outside service. Its IT consultant installs an open-weight model with 8 billion parameters, quantized to 4 bits, on a desktop with 32 GB of unified memory, and exposes it through a local chat interface on the office network.
The model occupies about 5 GB, leaving room for a 16,000-token context that holds a typical contract. It generates around 20 tokens per second, so a one-page summary arrives in under a minute. Lawyers still verify every summary, yet the firm cuts first-draft review time noticeably while no client document ever leaves the building.