vLLM is an open-source inference and serving engine for large language models that maximizes how many tokens a GPU can generate per second while keeping latency acceptable. It began as a research project at the University of California, Berkeley, and is now a widely used community project with an OpenAI-compatible API server.
In AI infrastructure, vLLM fills the model serving layer between the trained model weights and the applications that send prompts. Its central idea, PagedAttention, manages the memory of the KV cache the way an operating system manages virtual memory, which lets one GPU serve many more concurrent requests than naive implementations.
How vLLM manages GPU memory
During generation, a transformer stores the key and value tensors of every previous token so that it does not recompute them. This KV cache grows with sequence length and batch size and often consumes more GPU memory than the weights themselves for long contexts. Older servers reserved one contiguous block per request sized for the maximum length, which wasted most of that space.
PagedAttention splits the KV cache into fixed-size blocks, by default 16 tokens each, and keeps a block table that maps each sequence to physical blocks anywhere in memory. Blocks are allocated only as tokens are produced, so fragmentation drops to a few percent instead of the large waste of contiguous reservations. Sequences that share a prompt prefix can also share the same physical blocks through copy-on-write.
The second pillar is continuous batching. Rather than waiting for a whole batch to finish, the scheduler inserts new requests and removes completed ones at every decoding step. GPUs stay busy, and short requests are not stuck behind long ones.
Core features and configuration parameters
vLLM exposes its behavior through engine arguments passed on the command line or in code. A handful of them decide most of the capacity and latency trade-offs, so operators should understand them before tuning anything else.
| Parameter | Typical value | Effect |
|---|---|---|
| gpu-memory-utilization | 0.85-0.95, default 0.9 | Share of GPU memory for weights and KV cache |
| max-model-len | 4,096 to 128,000 tokens | Longest context; larger values reduce concurrency |
| max-num-seqs | 64-512 | Upper bound on concurrent sequences |
| tensor-parallel-size | 1, 2, 4 or 8 | Splits each layer across GPUs in one node |
| pipeline-parallel-size | 1-4 | Splits layers across nodes or GPU groups |
| enable-prefix-caching | On for shared prompts | Reuses KV blocks of identical prefixes |
| quantization | AWQ, GPTQ, FP8 | Smaller weights, more room for KV cache |
Beyond these settings, vLLM supports chunked prefill, which splits long prompts so they do not stall ongoing decoding, speculative decoding with draft models or n-gram proposals, multi-LoRA serving of many adapters on one base model, and structured output that constrains generation to a JSON schema or grammar.
vLLM compared with other serving options
vLLM is one of several inference servers for language models. The right choice depends on hardware, model family, latency targets and how much operational complexity a team accepts.
| Option | Strength | Trade-off |
|---|---|---|
| vLLM | High throughput, broad model support | Tuning needed for strict latency targets |
| Hugging Face Transformers generate | Simple, flexible for research | Low throughput, no paged KV cache |
| Optimized vendor runtimes | Peak performance on specific GPUs | Model compilation and vendor lock-in |
| Local desktop runtimes | Easy CPU and laptop use | Not built for many concurrent users |
| Managed inference APIs | No infrastructure to run | Less control over data and cost |
Within the Topic, vLLM is a concrete implementation of model serving and GPU inference. It can sit behind an AI gateway or a model routing layer that chooses between several deployments, and its prefix caching is an engine-level relative of prompt caching offered by hosted APIs.
Deploying vLLM in practice
A typical deployment runs the vLLM server in a container on GPU nodes, often under Kubernetes, and exposes the OpenAI-style endpoints for chat and completions. Clients can then switch from a hosted API to self-hosted models by changing only the base URL and the model name.
Capacity planning starts from memory. The weights of a 7 to 8 billion parameter model in 16-bit precision need roughly 15-16 GB; whatever remains under the memory utilization limit becomes KV cache. That remaining space, divided by the per-token cache size, determines how many tokens can be in flight at once.
| Goal | Recommended setting | Reason |
|---|---|---|
| Maximum throughput for batch jobs | Large max-num-seqs, offline mode | Fills the GPU with work |
| Low time to first token | Chunked prefill, smaller batches | Limits queueing behind long prompts |
| Model larger than one GPU | Tensor parallelism | Splits weights across fast links |
| Many requests with one system prompt | Prefix caching | Skips repeated prefill work |
| Limited GPU memory | FP8 or 4-bit quantization | Frees memory for KV cache |
Operators should watch the built-in Prometheus metrics: queue length, KV cache usage, time to first token and inter-token latency. A KV cache that sits near 100 percent usage triggers preemption, where the scheduler pauses sequences and later recomputes or swaps them, which raises latency sharply.
Limitations and common pitfalls
vLLM is powerful but not a drop-in answer for every workload. Most problems come from mismatched expectations about memory and latency.
- Setting max-model-len too high reserves capacity for contexts nobody sends and cuts concurrency.
- Running other processes on the same GPU conflicts with the memory utilization fraction and causes out-of-memory errors at startup.
- Assuming identical outputs across versions, batch sizes or GPUs is risky; batching can change floating-point results slightly.
- Ignoring tokenizer and chat template settings produces subtle quality drops even when the server runs normally.
- Fast release cadence means configuration flags and defaults change, so versions should be pinned and upgrades tested.
In plain words
A language model needs a program that runs it and answers many users at once. vLLM is one such program. Its job is to squeeze as many answers as possible out of each expensive graphics card.
It does this mainly by organizing memory better. Imagine a restaurant that seats guests at any free chair instead of reserving whole tables. Far more people fit in the room. vLLM treats the model’s working memory the same way, so it can serve more people with the same hardware.
A real-world example
A software company wants an internal coding assistant but cannot send source code to an external service. Its platform team deploys an open-weight model of about 8 billion parameters with vLLM on two 80 GB GPUs, one server replica per GPU, behind an internal gateway that handles authentication.
They set max-model-len to 32,000 tokens, enable prefix caching because every request starts with the same long system prompt, and use FP8 weights to free memory. Load tests with 150 simulated developers show stable latency without KV cache preemption, and the assistant goes live without any change to the client tools, which already spoke the OpenAI-style API.