llama.cpp

On this page 8

llama.cpp is an open-source inference engine written in C and C++ that runs large language models on ordinary hardware, from laptops and desktop GPUs to phones and single-board computers. Among AI frameworks & tools, it occupies the local inference layer: it does not train models, but loads already trained weights, usually quantized to 2 to 8 bits per weight, and generates text with minimal dependencies.

The project was started in 2023 by Georgi Gerganov and is released under the MIT license. It is built on ggml, a compact tensor library by the same author, and it defined the GGUF file format that much of the local model ecosystem now uses. Desktop applications such as Ollama and LM Studio rely on llama.cpp or its components internally.

How llama.cpp runs a language model

llama.cpp runs a model by loading a GGUF file, building a computation graph in ggml, and executing that graph token by token on the available hardware. The GGUF file holds the weights, tokenizer and model metadata in a single binary, so no Python runtime, PyTorch installation or separate tokenizer file is needed at inference time.

Generation has two phases. During prompt processing (prefill), the engine processes the whole input in large batches, which is compute-bound and benefits from matrix units on GPUs. During token generation (decode), it produces one token at a time, which is memory-bandwidth-bound: every new token requires reading most of the weights from memory. This is why quantization speeds up decoding almost in proportion to the reduction in model size.

The engine stores attention keys and values in a KV cache so earlier tokens are not recomputed. Files are opened with memory mapping (mmap), which lets the operating system page weights in on demand and share them between processes. The main components are summarized below.

ComponentRolePractical effect
ggmlTensor library and graph executorPortable kernels for many backends
GGUFSingle-file model containerWeights, tokenizer and metadata together
KV cacheStores past keys and valuesMemory grows with context length
llama-serverHTTP server with OpenAI-style APIDrop-in local endpoint for applications
llama-quantizeConverts weights to lower precisionSmaller files and faster decoding
Main components of the llama.cpp stack

Quantization formats in llama.cpp

Quantization in llama.cpp means storing weights in blocks of low-bit integers with shared scale factors, which reduces memory use and bandwidth at the cost of some accuracy. The K-quants (for example Q4_K_M) use super-blocks with nested scales, and the i-quants (IQ2, IQ3, IQ4) use lattice-based codebooks for very low bit rates. An importance matrix (imatrix), computed from calibration text, can guide which weights keep more precision.

The table gives approximate sizes for a model with 7 to 8 billion parameters. Exact figures vary by architecture, and quality loss is usually measured as an increase in perplexity compared with the 16-bit original.

TypeApproximate bits per weightFile sizeTypical quality impact
F1616About 16 GBReference quality
Q8_08.5About 8.5 GBPractically indistinguishable
Q6_K6.6About 6.6 GBVery small loss
Q4_K_M4.8About 4.9 GBSmall loss, popular default
IQ2_XS2.3About 2.5 GBClear degradation, last resort
Common GGUF quantization types for an 8B model

A widely used rule is that a larger model at 4 bits often outperforms a smaller model at 8 bits of the same file size, although this depends on the task. Below about 3 bits, degradation becomes noticeable in reasoning and code generation, and an imatrix becomes close to essential.

Hardware backends and GPU offloading

llama.cpp supports many hardware backends through ggml, so the same model file runs on CPUs, Apple silicon and discrete GPUs. On CPUs it uses SIMD instructions such as AVX2, AVX-512 and ARM NEON; on GPUs it offers CUDA, Metal, Vulkan, HIP for AMD and SYCL for Intel hardware.

Partial GPU offloading is a distinctive feature: the --n-gpu-layers option (short form -ngl) places a chosen number of transformer layers in video memory and leaves the rest on the CPU. This lets a model that is larger than available VRAM still run, although every layer left on the CPU slows decoding, because system memory bandwidth is usually 5 to 20 times lower than GPU memory bandwidth.

HardwareBackendStrengthMain constraint
Desktop NVIDIA GPUCUDAFast prefill and decodeVRAM capacity
Apple silicon MacMetalLarge unified memoryLower compute than high-end GPUs
x86 server CPUCPU with AVX-512Runs very large models in RAMSlow decoding per user
AMD or Intel GPUHIP, SYCL or VulkanVendor-neutral accelerationUneven kernel maturity
Phone or Raspberry PiCPU with NEONFully offline operationOnly small models are practical
Typical llama.cpp deployment targets

llama.cpp is a low-level engine, whereas several neighboring tools wrap it or target different deployment scenarios. Choosing between them depends on whether the priority is control, convenience or data-center throughput.

ToolRelationship to llama.cppBest suited for
OllamaWraps llama.cpp-derived code with model managementSimple command-line local use
LM StudioDesktop application with a llama.cpp runtimeGraphical model browsing and chat
Hugging Face TransformersSeparate Python library on PyTorchResearch and fine-tuning workflows
ONNX RuntimeSeparate engine for ONNX graphsCross-platform production deployment
TensorRTSeparate NVIDIA-optimized compilerMaximum throughput on NVIDIA GPUs
llama.cpp and neighboring inference tools

For serving many concurrent users in a data center, dedicated serving engines with paged attention usually achieve higher aggregate throughput. llama.cpp excels where single-user latency, portability and small footprint matter more than peak batch throughput.

Configuring llama.cpp in practice

Practical configuration in llama.cpp centers on a few parameters that trade memory against speed and context. The defaults are reasonable, but tuning them to the hardware often changes performance substantially.

  • Context size (-c) sets the maximum number of tokens in the KV cache; doubling it roughly doubles cache memory.
  • GPU layers (-ngl) should be as high as VRAM allows, leaving headroom for the KV cache.
  • KV cache quantization to 8-bit types reduces cache memory with modest quality impact.
  • Threads (-t) usually perform best at the number of physical performance cores, not logical threads.
  • Grammar constraints in GBNF or JSON schema force output into a valid structure.

The bundled llama-server exposes an OpenAI-compatible HTTP API, supports continuous batching of parallel requests, and can use speculative decoding with a small draft model to accelerate generation. The llama-bench and llama-perplexity tools let teams measure speed and quality on their own hardware instead of relying on published figures.

Limitations and common mistakes

llama.cpp has real limitations that experienced users plan around. New model architectures need explicit support in the codebase, so a freshly released model may not run until its conversion script and graph code are added. The project moves quickly, and command-line options, file names and defaults change between releases.

SymptomLikely causeTypical fix
Very slow generationToo few layers offloaded to the GPURaise -ngl or choose a smaller quantization
Out-of-memory errorContext size too large for the KV cacheReduce -c or quantize the KV cache
Garbled or repetitive outputWrong chat template or broken quantizationUse the template stored in GGUF metadata
Poor answers at 2-3 bitsAggressive quantization without imatrixUse Q4_K_M or an imatrix-based file
Frequent problems and their usual causes

A common misconception is that llama.cpp is only for Llama models. Despite the name, it supports dozens of architectures, including Mistral, Qwen, Gemma and Phi families, as well as some multimodal and embedding models.

In everyday language

llama.cpp is a free program that lets a chatbot run on your own computer instead of on a company’s servers. Normally, large AI models need expensive data-center machines. llama.cpp makes them fit on ordinary hardware by shrinking the model, a bit like compressing a large photo into a smaller file that still looks almost the same.

The result is that your questions and documents never leave your device. The trade-off is that a home computer is slower and can only hold smaller models than a large cloud service.

Seen in the real world

A small law office wants to summarize confidential contracts without sending them to an external service. Its IT consultant installs llama.cpp on a workstation with a 24 GB graphics card and downloads an 8-billion-parameter model in Q6_K format, about 6.6 GB, which leaves ample room for a 32,000-token context window.

The consultant starts llama-server with all layers offloaded to the GPU and connects the office’s document tool to its OpenAI-compatible endpoint. Prompt processing of a 20-page contract takes a few seconds, and summaries generate at roughly 60 to 90 tokens per second. The office gains fully local, private summarization at the cost of one workstation, while accepting that the model is less capable than the largest cloud models on complex legal reasoning.