Semantic caching is an AI infrastructure technique that stores responses from a language model and reuses them when a new request has the same meaning as an earlier one, even if the wording differs. A semantic cache compares requests by embedding similarity instead of exact text, so “How do I reset my password?” and “I forgot my password, what now?” can be answered from one stored result.
The goal is to avoid repeated model calls for questions that recur in different phrasings. Every hit skips a full generation, which lowers cost per token, removes the time spent in prefill and decoding, and reduces load on GPU inference capacity. The price of that saving is a new failure mode: returning a cached answer to a question that only looks similar.
How a semantic cache handles a request
A semantic cache sits in front of the model, usually inside an AI gateway or application layer, and runs a lookup before any inference happens. The request is converted into a vector by an embedding model, the vector is searched in a vector index of earlier requests, and the closest match is compared with a similarity threshold.
- Normalize the request: strip volatile parts such as timestamps, and decide which context belongs to the cache key.
- Embed the request with a small, fast embedding model, typically in a few milliseconds.
- Search the index for the nearest stored requests using approximate nearest neighbor search.
- Apply the threshold: if similarity exceeds it, return the stored response as a cache hit.
- On a miss, call the model, return its answer and store the request vector with the response.
The main components and their typical settings are summarized below. Exact values vary widely between workloads and must be tuned against real traffic.
| Component | Role | Typical choice or value |
|---|---|---|
| Embedding model | Turns a request into a vector | Compact sentence embedding model |
| Vector index | Finds similar earlier requests | HNSW or IVF approximate index |
| Similarity threshold | Decides hit or miss | Often cosine 0.85 to 0.95 |
| Time to live | Expires stale entries | Minutes to weeks, by content |
| Cache scope | Separates tenants and contexts | Per user, tenant, model or prompt version |
Semantic caching compared with other caches
Semantic caching is one of three distinct caching layers in LLM serving, and they are frequently confused. An exact-match cache returns a response only for byte-identical requests. Prompt caching reuses the computed attention state of a shared prompt prefix, so the model still generates a fresh answer but skips part of the prefill phase. The KV cache stores keys and values inside a single generation so each new token does not recompute earlier ones.
| Cache type | What is reused | Match rule | Model still generates |
|---|---|---|---|
| Exact-match cache | Final response | Identical request text | No |
| Semantic cache | Final response | Similar meaning above threshold | No |
| Prompt caching | Prefix attention state | Identical prompt prefix | Yes |
| KV cache | Keys and values per token | Same ongoing sequence | Yes |
These layers are complementary rather than competing. A typical stack checks an exact-match cache first because it is cheapest and safest, then a semantic cache, and only then sends the request to an inference server where prompt caching and the KV cache accelerate generation.
Tuning the similarity threshold
The similarity threshold is the central trade-off of semantic caching. A low threshold raises the hit rate and the savings but returns more answers to questions that only resemble the cached one, called false hits. A high threshold keeps answers correct but serves few requests from the cache, so the infrastructure cost of the cache may exceed what it saves.
Thresholds are not portable: cosine values depend on the embedding model, the language and the domain. Teams therefore calibrate on labeled request pairs from their own logs and measure precision of hits, not only hit rate. The effects of each setting are summarized below.
| Threshold | Hit rate | Risk of false hits | Suitable for |
|---|---|---|---|
| Low, around 0.80 | High | High | Casual chat, low-stakes FAQ |
| Medium, around 0.90 | Moderate | Moderate | General customer support |
| High, around 0.95 | Low | Low | Policy, pricing, technical answers |
| Exact match only | Lowest | Near zero | Regulated or personalized output |
Realistic hit rates depend on how repetitive the traffic is. FAQ-style assistants with a narrow domain can serve a large share of requests from cache, while open-ended or multi-turn conversations rarely repeat and may see hit rates in the low single digits.
When semantic caching pays off
Semantic caching works best when many users ask the same questions and the correct answer does not depend on who asks or when. Help desks, product documentation assistants, internal policy bots and search-style question answering fit this profile. It is a poor fit for personalized output, conversations whose answer depends on earlier turns, and creative generation where variety is expected.
Conversation context is the most common design question. If only the latest message is embedded, two users who asked “And what about the price?” after different products collide. Robust systems include relevant context in the cache key, such as the product, user role, system prompt version and model version, or bypass the cache for follow-up turns.
Risks and common mistakes
Most production failures come from treating meaning similarity as answer equivalence. Two questions can be close in embedding space while requiring different answers, especially when they differ in a number, negation, date or entity. The table lists frequent problems.
| Symptom | Likely cause | Fix |
|---|---|---|
| Wrong answer for a similar question | Threshold too low | Raise it and recalibrate on labeled pairs |
| One user sees another user’s data | Cache not scoped per tenant or user | Partition the cache by identity |
| Outdated prices or policies served | Missing or long time to live | Expire or invalidate on content change |
| Answers ignore conversation history | Only last message used as key | Add context to key or skip follow-ups |
| Little cost saving despite effort | Traffic rarely repeats | Measure hit rate before scaling the cache |
Data leakage across users is the most serious risk, because a cached answer may contain personal or confidential details. Sensitive workloads should scope caches strictly, avoid caching outputs that contain user data, and log hits so false matches can be audited through LLM observability tooling.
The simple explanation
Semantic caching works like an experienced receptionist who has heard the same questions many times. When someone asks something already answered before, even in different words, the receptionist repeats the earlier answer instead of looking it up again. That is faster and cheaper than asking an expert every time.
The catch is that two questions can sound alike but need different answers. A good semantic cache therefore reuses an answer only when the questions are truly close in meaning, and it forgets answers that may have gone out of date.
How it is used in practice
An online electronics retailer runs a support assistant that handles about 300,000 questions per month, many of them variations of the same topics: delivery times, returns, warranty terms and order tracking. The engineering team places a semantic cache in its AI gateway, scoped by country and by the version of the system prompt, with a threshold of 0.93 chosen after labeling 1,000 real question pairs.
Questions that mention an order number or account details bypass the cache entirely, and entries expire after seven days or immediately when the returns policy page changes. After the first month, roughly a quarter of questions are answered from the cache in under 100 milliseconds instead of two to three seconds, and the monthly model bill falls accordingly. Weekly review of a sample of cache hits finds a small number of false matches, mostly about different product models, so the team adds the product category to the cache key.