Local LLM

Also known as: running LLMs locally

On this page 7

A local LLM is a large language model that runs entirely on hardware the user controls, such as a laptop, workstation, phone or on-premises server, so that prompts, documents and generated text never leave the device. In Edge AI, running LLMs locally is the most demanding form of on-device inference, because a model with billions of parameters must fit into limited memory, power and compute budgets.

The approach rests on two developments: open-weight models whose parameters can be downloaded, and aggressive compression, above all quantization, that shrinks those models to a few gigabytes. The result trades some quality and speed for privacy, offline availability, predictable cost and full control over the model and its behavior.

How a local LLM generates text on a device

A local LLM produces text through the same autoregressive decoding loop as a cloud model, but every step runs on local silicon. An inference runtime loads the weight file into memory, tokenizes the prompt, processes it in one parallel prefill phase, and then generates one token at a time in the decode phase, appending each token to the context.

The two phases stress hardware differently. Prefill is compute-bound and benefits from GPU or NPU throughput, while decoding is memory-bandwidth-bound: for every new token, the runtime must read essentially all active weights from memory. A useful rule of thumb is that decode speed is roughly memory bandwidth divided by model size in bytes, which explains why a 4 GB model on a machine with 100 GB/s of bandwidth tops out near 25 tokens per second.

Alongside the weights, the runtime keeps a key-value cache (KV cache) that stores attention states for every token in the context. Its size grows linearly with context length, so a long conversation or a large document can consume several additional gigabytes, which is often the hidden reason a model that loads successfully later runs out of memory.

The table below lists the main components of a typical local deployment and what each one contributes.

ComponentRoleTypical examples
Model weightsLearned parameters, usually quantizedGGUF, safetensors, MLX files
Inference runtimeLoads weights, schedules compute, manages KV cachellama.cpp, MLC LLM, ONNX Runtime
Hardware delegateOffloads matrix math to an acceleratorCUDA, Metal, Vulkan, NPU drivers
TokenizerConverts text to token IDs and backBPE or SentencePiece vocabulary
Serving layerExposes a chat interface or local APIDesktop app, OpenAI-compatible endpoint
Components of a local LLM stack

Memory budgets and model sizes

Memory is the decisive constraint for running LLMs locally. Weight memory is approximately the parameter count multiplied by the bytes per parameter, so a 7-billion-parameter model needs about 14 GB at 16-bit precision but only about 4 GB at 4-bit precision, plus overhead for the KV cache and runtime buffers.

The following table gives approximate memory needs for common size classes at different precisions. Real figures vary by architecture, quantization format and context length.

Model size16-bit weights8-bit weights4-bit weightsTypical device
1-3 billion parameters2-6 GB1-3 GB0.6-1.8 GBPhones, small laptops
7-9 billion parameters14-18 GB7-9 GB4-5.5 GBLaptops with 16 GB RAM
13-14 billion parameters26-28 GB13-14 GB7.5-8.5 GBConsumer GPUs, 32 GB systems
30-34 billion parameters60-68 GB30-34 GB17-20 GB24 GB GPUs, unified-memory desktops
70 billion parametersabout 140 GBabout 70 GB35-40 GBMulti-GPU or high-memory workstations
Approximate weight memory by model size and precision

Unified memory architectures, in which CPU and GPU share one pool, are popular for local inference because they allow large models without a discrete GPU, although their bandwidth is still lower than that of high-end graphics memory. On phones, the operating system usually limits a single app to a fraction of total RAM, which keeps practical on-device models in the 1-4 billion parameter range.

Compression techniques that make local inference possible

Weight-only quantization is the workhorse of local deployment. It stores weights in 4 to 8 bits while computing activations at higher precision, which cuts memory and bandwidth by two to four times with modest quality loss. Methods such as GPTQ and AWQ choose quantized values that minimize output error, and formats like GGUF mix bit widths across layers.

Other techniques complement quantization: knowledge distillation trains a smaller model to imitate a larger one, structured pruning removes whole attention heads or channels, and speculative decoding uses a tiny draft model to propose tokens that the main model verifies in parallel. The table below compares their effect.

TechniqueMain benefitMain cost or risk
4-bit quantizationAbout 4x smaller than 16-bitSmall accuracy loss, larger on reasoning tasks
8-bit quantizationAbout 2x smaller, near-losslessLess savings than 4-bit
DistillationSmaller model with retained skillsRequires a training pipeline and data
Structured pruningFewer operations on standard hardwareNeeds fine-tuning to recover quality
Speculative decodingFaster decoding without quality changeExtra memory for the draft model
Compression and acceleration methods for local LLMs

Local LLM compared with cloud and hybrid inference

A local LLM differs from a cloud-hosted model mainly in where data and cost reside. Cloud APIs offer the largest frontier models and elastic capacity, while local execution offers data sovereignty and zero per-token fees. Hybrid inference combines both, routing simple or sensitive requests to the device and hard requests to the cloud.

CriterionLocal LLMCloud APIHybrid inference
Data privacyData stays on deviceData sent to providerDepends on routing policy
Model quality ceilingLimited by local memoryLargest available modelsHigh for routed requests
LatencyNo network round tripNetwork plus queue delayVariable
Cost modelUp-front hardware, energyPer-token or subscriptionMixed
Offline useFully supportedNot supportedPartial
Deployment options for language models

Limitations and common mistakes

Running LLMs locally carries real trade-offs. Smaller quantized models hallucinate more, follow complex instructions less reliably and have weaker multilingual and coding abilities than frontier systems. Sustained inference on laptops and phones causes thermal throttling and battery drain, so benchmark speeds measured over ten seconds rarely hold over ten minutes.

  • Ignoring the KV cache: sizing hardware for weights alone leads to out-of-memory errors with long contexts.
  • Over-quantizing: dropping below 3-4 bits often causes a sharp quality collapse, especially for small models.
  • Assuming local means secure: downloaded weights and runtimes are supply-chain risks and should come from verified sources.
  • Overlooking license terms: open-weight models carry licenses that may restrict commercial use or require attribution.

In simple terms

A local LLM is a chatbot brain that lives on your own computer or phone instead of on a company’s servers. When you type a question, the answer is worked out right there on your device. That means your words never travel over the internet.

To fit on ordinary hardware, these models are squeezed. It is a bit like compressing a large photo so it takes less space. The squeezed version is slightly less capable than the giant cloud models. Still, it works without a connection. It costs nothing per question and keeps private material private.

A practical example

A small law firm wants help summarizing confidential contracts but cannot send client documents to an outside service. Its IT consultant installs an open-weight model with 8 billion parameters, quantized to 4 bits, on a desktop with 32 GB of unified memory, and exposes it through a local chat interface on the office network.

The model occupies about 5 GB, leaving room for a 16,000-token context that holds a typical contract. It generates around 20 tokens per second, so a one-page summary arrives in under a minute. Lawyers still verify every summary, yet the firm cuts first-draft review time noticeably while no client document ever leaves the building.