Prompt caching is an inference optimization that stores the computed internal state of a prompt prefix so that later requests beginning with the same prefix can skip recomputing it. Instead of processing a long system prompt, document or conversation history from scratch every time, the server reuses the saved key-value (KV) cache and only processes the new tokens.
In AI infrastructure within artificial intelligence, prompt caching is one of the most effective ways to reduce time to first token and input cost for applications that repeat large contexts, such as agents with long tool definitions, chat with long histories and question answering over a fixed document. It changes neither the model nor its outputs, only how much work the server repeats.
How prompt caching reuses the KV cache
Transformer inference has two phases. During prefill, the model processes all prompt tokens and computes attention keys and values for every layer. During decode, it generates output tokens one at a time, reading those stored keys and values. Prefill cost grows with prompt length, so a 50,000-token prompt creates substantial compute and latency before the first output token appears.
Because each token’s keys and values depend only on the tokens before it, the KV cache for an identical prefix is identical across requests. Prompt caching keeps that state in GPU memory, or offloads it to CPU memory or storage, and attaches it to any new request whose beginning matches token for token. Only the remaining suffix then goes through prefill.
| Event | Prefix handling | Effect on the request |
|---|---|---|
| Cache write | Prefix computed and stored | Normal or slightly higher cost |
| Cache hit | Stored KV state reused | Much faster prefill, lower input cost |
| Partial hit | Longest matching prefix reused | Savings proportional to match length |
| Cache miss | Full prefill from scratch | No savings |
| Eviction | Stored state discarded | Next request misses |
Implementations and key parameters
Prompt caching appears in two main forms. Hosted model APIs offer it as a billing and latency feature, sometimes automatic and sometimes enabled by explicit cache breakpoints in the request. Self-hosted inference servers implement automatic prefix caching, typically by hashing fixed-size blocks of tokens and sharing matching blocks among requests.
| Parameter | Typical values | Practical meaning |
|---|---|---|
| Minimum cacheable length | Often around 1,000 tokens or more | Short prompts may not be cached |
| Time to live | Minutes by default, longer options exist | How long an unused prefix survives |
| Block size | 16 to 256 tokens per block | Granularity of prefix matching |
| Cached input price | A fraction of normal input price | Main cost saving on hosted APIs |
| Cache write price | Equal to or above normal input | Upfront cost of storing a prefix |
| Storage tier | GPU, CPU memory or disk | Capacity versus reload speed |
On self-hosted servers, techniques such as paged KV memory and radix-tree prefix sharing let many concurrent requests share one copy of a common prefix. The limiting factor is GPU memory, because cached prefixes compete with the KV cache of active requests.
Prompt caching compared with other caching layers
Prompt caching is often confused with response caching, which stores finished answers. The two operate at different layers and have very different correctness properties.
| Technique | What is stored | Reused when | Output changes |
|---|---|---|---|
| Prompt caching | KV state of a prefix | Prefix matches exactly | No, outputs remain freshly generated |
| Exact response cache | Final answer text | Entire request identical | Returns old answer |
| Semantic cache | Answers to similar queries | Embedding similarity above threshold | May return a wrong answer |
| In-request KV cache | Keys and values during decoding | Within one generation | No |
| Embedding cache | Vectors of texts | Same text embedded again | No |
Prompt caching also interacts with model routing: if requests with the same prefix are sent to different servers or models, each server must build its own cache. Routers that use prefix-aware or session-affinity routing keep related requests together and raise hit rates.
Structuring prompts for high cache hit rates
Because matching is strictly by prefix, prompt order determines savings. Stable content must come first and changing content last, and even a one-character change early in the prompt invalidates everything after it.
| Situation | Recommended structure | Reason |
|---|---|---|
| Agent with many tools | Tools and system prompt first | Largest stable block is reused |
| Chat with long history | Append new turns, never rewrite old ones | Earlier turns stay a valid prefix |
| Questions over one document | Document before the question | Document becomes the shared prefix |
| Timestamps or user IDs needed | Place them near the end | Avoids breaking the prefix early |
| Bursty traffic | Keep prefixes warm or extend time to live | Prevents eviction between bursts |
Teams should monitor cached token counts reported by the API or server. A sudden drop in hit rate usually points to an innocent-looking change, such as a reordered tool list or a date inserted into the system prompt.
Limitations and common mistakes
Prompt caching brings large gains only for repeated long prefixes. Workloads with short or always-unique prompts see little benefit and may pay extra for cache writes.
- Dynamic content at the top: a timestamp in the first line prevents any reuse.
- Non-deterministic serialization: tool definitions or JSON keys emitted in varying order break exact matching.
- Assuming persistence: cached prefixes can be evicted at any time and must never be treated as storage.
- Ignoring decode cost: caching speeds up prefill, not the generation of long outputs.
- Isolation concerns: shared caches must be scoped so that timing differences cannot leak information between tenants.
In plain words
Before a language model answers, it must read the whole prompt. Long prompts take time and money to read. Prompt caching lets the system remember the part it has already read. On the next request, it skips that part.
It is like a student who has already studied the first ten chapters of a textbook. Before the next class, the student only reads the new chapter. The beginning must be exactly the same for this memory to help. That is why stable instructions go first.
A real-world example
A software company runs a coding assistant whose fixed prefix is about 30,000 tokens: a system prompt, roughly forty tool definitions and project conventions. Every user turn previously reprocessed this prefix, and users waited several seconds before the first word appeared.
The engineers move all stable material to the beginning, sort tool definitions deterministically, place the current date and file excerpts after the conversation history, and mark a cache breakpoint after the tool block. They then track the share of cached input tokens per request.
Once most requests hit the cache, time to first token drops noticeably and the input token bill falls because cached tokens are priced lower. A later regression is traced to a new tool inserted at the top of the list, and moving it to the end restores the hit rate.