llama.cpp is an open-source inference engine written in C and C++ that runs large language models on ordinary hardware, from laptops and desktop GPUs to phones and single-board computers. Among AI frameworks & tools, it occupies the local inference layer: it does not train models, but loads already trained weights, usually quantized to 2 to 8 bits per weight, and generates text with minimal dependencies.
The project was started in 2023 by Georgi Gerganov and is released under the MIT license. It is built on ggml, a compact tensor library by the same author, and it defined the GGUF file format that much of the local model ecosystem now uses. Desktop applications such as Ollama and LM Studio rely on llama.cpp or its components internally.
How llama.cpp runs a language model
llama.cpp runs a model by loading a GGUF file, building a computation graph in ggml, and executing that graph token by token on the available hardware. The GGUF file holds the weights, tokenizer and model metadata in a single binary, so no Python runtime, PyTorch installation or separate tokenizer file is needed at inference time.
Generation has two phases. During prompt processing (prefill), the engine processes the whole input in large batches, which is compute-bound and benefits from matrix units on GPUs. During token generation (decode), it produces one token at a time, which is memory-bandwidth-bound: every new token requires reading most of the weights from memory. This is why quantization speeds up decoding almost in proportion to the reduction in model size.
The engine stores attention keys and values in a KV cache so earlier tokens are not recomputed. Files are opened with memory mapping (mmap), which lets the operating system page weights in on demand and share them between processes. The main components are summarized below.
| Component | Role | Practical effect |
|---|---|---|
| ggml | Tensor library and graph executor | Portable kernels for many backends |
| GGUF | Single-file model container | Weights, tokenizer and metadata together |
| KV cache | Stores past keys and values | Memory grows with context length |
| llama-server | HTTP server with OpenAI-style API | Drop-in local endpoint for applications |
| llama-quantize | Converts weights to lower precision | Smaller files and faster decoding |
Quantization formats in llama.cpp
Quantization in llama.cpp means storing weights in blocks of low-bit integers with shared scale factors, which reduces memory use and bandwidth at the cost of some accuracy. The K-quants (for example Q4_K_M) use super-blocks with nested scales, and the i-quants (IQ2, IQ3, IQ4) use lattice-based codebooks for very low bit rates. An importance matrix (imatrix), computed from calibration text, can guide which weights keep more precision.
The table gives approximate sizes for a model with 7 to 8 billion parameters. Exact figures vary by architecture, and quality loss is usually measured as an increase in perplexity compared with the 16-bit original.
| Type | Approximate bits per weight | File size | Typical quality impact |
|---|---|---|---|
| F16 | 16 | About 16 GB | Reference quality |
| Q8_0 | 8.5 | About 8.5 GB | Practically indistinguishable |
| Q6_K | 6.6 | About 6.6 GB | Very small loss |
| Q4_K_M | 4.8 | About 4.9 GB | Small loss, popular default |
| IQ2_XS | 2.3 | About 2.5 GB | Clear degradation, last resort |
A widely used rule is that a larger model at 4 bits often outperforms a smaller model at 8 bits of the same file size, although this depends on the task. Below about 3 bits, degradation becomes noticeable in reasoning and code generation, and an imatrix becomes close to essential.
Hardware backends and GPU offloading
llama.cpp supports many hardware backends through ggml, so the same model file runs on CPUs, Apple silicon and discrete GPUs. On CPUs it uses SIMD instructions such as AVX2, AVX-512 and ARM NEON; on GPUs it offers CUDA, Metal, Vulkan, HIP for AMD and SYCL for Intel hardware.
Partial GPU offloading is a distinctive feature: the --n-gpu-layers option (short form -ngl) places a chosen number of transformer layers in video memory and leaves the rest on the CPU. This lets a model that is larger than available VRAM still run, although every layer left on the CPU slows decoding, because system memory bandwidth is usually 5 to 20 times lower than GPU memory bandwidth.
| Hardware | Backend | Strength | Main constraint |
|---|---|---|---|
| Desktop NVIDIA GPU | CUDA | Fast prefill and decode | VRAM capacity |
| Apple silicon Mac | Metal | Large unified memory | Lower compute than high-end GPUs |
| x86 server CPU | CPU with AVX-512 | Runs very large models in RAM | Slow decoding per user |
| AMD or Intel GPU | HIP, SYCL or Vulkan | Vendor-neutral acceleration | Uneven kernel maturity |
| Phone or Raspberry Pi | CPU with NEON | Fully offline operation | Only small models are practical |
How llama.cpp compares with related tools
llama.cpp is a low-level engine, whereas several neighboring tools wrap it or target different deployment scenarios. Choosing between them depends on whether the priority is control, convenience or data-center throughput.
| Tool | Relationship to llama.cpp | Best suited for |
|---|---|---|
| Ollama | Wraps llama.cpp-derived code with model management | Simple command-line local use |
| LM Studio | Desktop application with a llama.cpp runtime | Graphical model browsing and chat |
| Hugging Face Transformers | Separate Python library on PyTorch | Research and fine-tuning workflows |
| ONNX Runtime | Separate engine for ONNX graphs | Cross-platform production deployment |
| TensorRT | Separate NVIDIA-optimized compiler | Maximum throughput on NVIDIA GPUs |
For serving many concurrent users in a data center, dedicated serving engines with paged attention usually achieve higher aggregate throughput. llama.cpp excels where single-user latency, portability and small footprint matter more than peak batch throughput.
Configuring llama.cpp in practice
Practical configuration in llama.cpp centers on a few parameters that trade memory against speed and context. The defaults are reasonable, but tuning them to the hardware often changes performance substantially.
- Context size (
-c) sets the maximum number of tokens in the KV cache; doubling it roughly doubles cache memory. - GPU layers (
-ngl) should be as high as VRAM allows, leaving headroom for the KV cache. - KV cache quantization to 8-bit types reduces cache memory with modest quality impact.
- Threads (
-t) usually perform best at the number of physical performance cores, not logical threads. - Grammar constraints in GBNF or JSON schema force output into a valid structure.
The bundled llama-server exposes an OpenAI-compatible HTTP API, supports continuous batching of parallel requests, and can use speculative decoding with a small draft model to accelerate generation. The llama-bench and llama-perplexity tools let teams measure speed and quality on their own hardware instead of relying on published figures.
Limitations and common mistakes
llama.cpp has real limitations that experienced users plan around. New model architectures need explicit support in the codebase, so a freshly released model may not run until its conversion script and graph code are added. The project moves quickly, and command-line options, file names and defaults change between releases.
| Symptom | Likely cause | Typical fix |
|---|---|---|
| Very slow generation | Too few layers offloaded to the GPU | Raise -ngl or choose a smaller quantization |
| Out-of-memory error | Context size too large for the KV cache | Reduce -c or quantize the KV cache |
| Garbled or repetitive output | Wrong chat template or broken quantization | Use the template stored in GGUF metadata |
| Poor answers at 2-3 bits | Aggressive quantization without imatrix | Use Q4_K_M or an imatrix-based file |
A common misconception is that llama.cpp is only for Llama models. Despite the name, it supports dozens of architectures, including Mistral, Qwen, Gemma and Phi families, as well as some multimodal and embedding models.
In everyday language
llama.cpp is a free program that lets a chatbot run on your own computer instead of on a company’s servers. Normally, large AI models need expensive data-center machines. llama.cpp makes them fit on ordinary hardware by shrinking the model, a bit like compressing a large photo into a smaller file that still looks almost the same.
The result is that your questions and documents never leave your device. The trade-off is that a home computer is slower and can only hold smaller models than a large cloud service.
Seen in the real world
A small law office wants to summarize confidential contracts without sending them to an external service. Its IT consultant installs llama.cpp on a workstation with a 24 GB graphics card and downloads an 8-billion-parameter model in Q6_K format, about 6.6 GB, which leaves ample room for a 32,000-token context window.
The consultant starts llama-server with all layers offloaded to the GPU and connects the office’s document tool to its OpenAI-compatible endpoint. Prompt processing of a 20-page contract takes a few seconds, and summaries generate at roughly 60 to 90 tokens per second. The office gains fully local, private summarization at the cost of one workstation, while accepting that the model is less capable than the largest cloud models on complex legal reasoning.