Ollama

On this page 7

Ollama is an open-source runtime for running large language models locally: it downloads quantized model weights, loads them onto the available CPU or GPU and exposes them through a command-line interface and an HTTP API on port 11434. Among AI frameworks & tools, it occupies the space between a low-level inference engine and a finished chat application, giving developers a single, consistent way to pull, run and serve open-weight models such as Llama, Mistral, Gemma, Qwen or Phi without cloud accounts or per-token fees.

The project is written mainly in Go and builds on the llama.cpp inference engine and the GGUF model format, adding model packaging, memory scheduling and a stable API on top. Its main value is operational simplicity: one command such as ollama run llama3.2 replaces the manual work of finding a compatible file, choosing a quantization, compiling a backend and writing a server.

How Ollama runs a model on local hardware

Ollama works as a background server process that accepts requests from the CLI, desktop apps and other programs, then schedules models into memory on demand. When a request names a model that is not loaded, the server reads its weights from the local model store, offloads as many layers as fit into GPU memory and keeps the remainder on the CPU.

After a response, the model stays resident for a keep-alive period of five minutes by default, so follow-up requests skip the loading cost. Models are stored as content-addressed layers, similar to container images, which lets several model variants share identical weight blobs on disk.

The following table lists the main components that a request passes through and what each one is responsible for.

ComponentRolePractical detail
Server (ollama serve)Accepts API calls, loads and unloads modelsListens on 127.0.0.1:11434 by default
CLI clientPulls, runs, lists and removes modelsTalks to the server over the same HTTP API
Model libraryRegistry of packaged open-weight modelsTags select size and quantization, for example 8b or q4_K_M
Inference backendPerforms the actual token generationBased on llama.cpp, with CUDA, ROCm and Metal acceleration
ModelfileDeclarative recipe for a custom modelSets base weights, prompt template, parameters and system message
Main components of an Ollama installation

Commands, API endpoints and the Modelfile

Everyday work with Ollama relies on a small set of CLI commands and a matching set of REST endpoints. The native API offers /api/generate for single completions, /api/chat for multi-turn conversations and /api/embed for embedding vectors, while an OpenAI-compatible endpoint under /v1 lets existing client libraries switch to a local model by changing only the base URL.

The table below maps common tasks to the command or endpoint that performs them.

TaskCLI commandAPI endpoint
Download a modelollama pull/api/pull
Chat interactivelyollama run/api/chat
List installed modelsollama list/api/tags
Show loaded modelsollama ps/api/ps
Build a custom modelollama create/api/create
Generate embeddingsNot applicable/api/embed
Common Ollama operations

A Modelfile customizes a model without retraining it. The FROM instruction names the base weights, PARAMETER sets values such as temperature or num_ctx, TEMPLATE defines the chat prompt format, SYSTEM fixes a default system message and ADAPTER attaches a LoRA adapter. Running ollama create turns this recipe into a new named model that behaves like any other in the library.

Quantization, memory and context length

The size of the model that Ollama can run is limited mainly by available GPU memory or unified memory. Library models are usually distributed with 4-bit quantization (Q4_K_M) by default, which cuts memory needs to roughly a quarter of 16-bit weights with a modest loss of quality. Higher-precision tags such as q8_0 or fp16 improve fidelity but demand far more memory.

The table gives approximate memory needs for 4-bit models; actual figures depend on architecture, context length and runtime overhead.

Parameter countWeights in memoryTypical hardware
1-4 billionAbout 1-3 GBLaptop CPU or integrated graphics
7-9 billionAbout 4-6 GBConsumer GPU with 8 GB or Apple silicon
12-14 billionAbout 8-10 GBGPU with 12-16 GB
27-34 billionAbout 16-22 GBGPU with 24 GB or a 32 GB Mac
70 billionAbout 40-45 GBTwo large GPUs or a 64 GB Mac
Approximate memory needs for 4-bit quantized models

The context window adds memory on top of the weights because the key-value cache grows with every token held in context. Ollama applies a deliberately modest default context length, which has changed between versions, so long documents are silently truncated unless num_ctx or the OLLAMA_CONTEXT_LENGTH variable is raised. Raising the context too far can push layers off the GPU and sharply reduce speed.

Ollama is often confused with the engine underneath it and with the interfaces built on top of it. The distinction matters when choosing a stack: llama.cpp offers maximum control, LM Studio offers a graphical desktop experience, and Ollama offers a service-oriented middle layer that other software can call.

ToolPrimary formBest suited for
OllamaLocal server with CLI and REST APIDevelopers integrating local models into applications
llama.cppC and C++ inference library and binariesFine-grained tuning, custom builds and embedded use
LM StudioDesktop GUI with an optional local serverExploring and comparing models interactively
Open WebUIBrowser chat interfaceA ChatGPT-like front end over Ollama or other APIs
vLLMHigh-throughput serving engineMulti-user production serving on data center GPUs
How Ollama differs from neighboring tools

Orchestration frameworks such as LangChain and LlamaIndex treat Ollama as one more model provider. Swapping a hosted model for a local one therefore rarely requires changes beyond the connection settings.

Limitations and common mistakes

Ollama is optimized for single-user and small-team workloads, not for high-concurrency production serving. Parallel requests are supported through OLLAMA_NUM_PARALLEL, but throughput falls well short of batching engines such as vLLM or TensorRT-LLM on the same GPU.

  • Exposing the API publicly: the server has no built-in authentication, so binding OLLAMA_HOST to 0.0.0.0 without a reverse proxy lets anyone on the network use the hardware.
  • Ignoring the default context: long prompts are cut without an error, which looks like the model forgetting instructions.
  • Assuming identical quality to hosted models: a quantized 8-billion-parameter model is far weaker at reasoning than frontier systems.
  • Overlooking license terms: Ollama itself uses the MIT license, but each model carries its own license and acceptable-use policy.
  • Running out of memory silently: when a model does not fit, Ollama splits it between GPU and CPU, and generation can slow by an order of magnitude.

In everyday language

Ollama is like an app store and player for AI chat models that lives entirely on your own computer. You type one short command. It downloads a model, and you can start asking questions. After that, no internet connection is needed, and no data leaves the machine.

Other programs on the same computer can talk to it too, much as they would talk to an online AI service. The catch is that your hardware sets the limits. A typical laptop runs small models well. The largest models need expensive graphics cards.

Seen in the real world

A legal department at a mid-sized manufacturer wants to summarize internal contracts but is not allowed to send them to an external AI service. An engineer installs Ollama on a workstation with a 24 GB graphics card, pulls a 14-billion-parameter instruction model in 4-bit quantization and writes a Modelfile that fixes a summarization system prompt and sets num_ctx to 16384 so that full contracts fit.

The team’s internal web tool calls the OpenAI-compatible endpoint, so the existing code changes only its base URL. Summaries arrive in roughly 10-20 seconds per contract, the server is reachable only through an authenticated reverse proxy, and no document ever leaves the company network, which satisfies the data protection review.