Semantic caching

Also known as: semantic cache

On this page 7

Semantic caching is an AI infrastructure technique that stores responses from a language model and reuses them when a new request has the same meaning as an earlier one, even if the wording differs. A semantic cache compares requests by embedding similarity instead of exact text, so “How do I reset my password?” and “I forgot my password, what now?” can be answered from one stored result.

The goal is to avoid repeated model calls for questions that recur in different phrasings. Every hit skips a full generation, which lowers cost per token, removes the time spent in prefill and decoding, and reduces load on GPU inference capacity. The price of that saving is a new failure mode: returning a cached answer to a question that only looks similar.

How a semantic cache handles a request

A semantic cache sits in front of the model, usually inside an AI gateway or application layer, and runs a lookup before any inference happens. The request is converted into a vector by an embedding model, the vector is searched in a vector index of earlier requests, and the closest match is compared with a similarity threshold.

  1. Normalize the request: strip volatile parts such as timestamps, and decide which context belongs to the cache key.
  2. Embed the request with a small, fast embedding model, typically in a few milliseconds.
  3. Search the index for the nearest stored requests using approximate nearest neighbor search.
  4. Apply the threshold: if similarity exceeds it, return the stored response as a cache hit.
  5. On a miss, call the model, return its answer and store the request vector with the response.

The main components and their typical settings are summarized below. Exact values vary widely between workloads and must be tuned against real traffic.

ComponentRoleTypical choice or value
Embedding modelTurns a request into a vectorCompact sentence embedding model
Vector indexFinds similar earlier requestsHNSW or IVF approximate index
Similarity thresholdDecides hit or missOften cosine 0.85 to 0.95
Time to liveExpires stale entriesMinutes to weeks, by content
Cache scopeSeparates tenants and contextsPer user, tenant, model or prompt version
Components of a semantic cache

Semantic caching compared with other caches

Semantic caching is one of three distinct caching layers in LLM serving, and they are frequently confused. An exact-match cache returns a response only for byte-identical requests. Prompt caching reuses the computed attention state of a shared prompt prefix, so the model still generates a fresh answer but skips part of the prefill phase. The KV cache stores keys and values inside a single generation so each new token does not recompute earlier ones.

Cache typeWhat is reusedMatch ruleModel still generates
Exact-match cacheFinal responseIdentical request textNo
Semantic cacheFinal responseSimilar meaning above thresholdNo
Prompt cachingPrefix attention stateIdentical prompt prefixYes
KV cacheKeys and values per tokenSame ongoing sequenceYes
Caching layers in LLM serving

These layers are complementary rather than competing. A typical stack checks an exact-match cache first because it is cheapest and safest, then a semantic cache, and only then sends the request to an inference server where prompt caching and the KV cache accelerate generation.

Tuning the similarity threshold

The similarity threshold is the central trade-off of semantic caching. A low threshold raises the hit rate and the savings but returns more answers to questions that only resemble the cached one, called false hits. A high threshold keeps answers correct but serves few requests from the cache, so the infrastructure cost of the cache may exceed what it saves.

Thresholds are not portable: cosine values depend on the embedding model, the language and the domain. Teams therefore calibrate on labeled request pairs from their own logs and measure precision of hits, not only hit rate. The effects of each setting are summarized below.

ThresholdHit rateRisk of false hitsSuitable for
Low, around 0.80HighHighCasual chat, low-stakes FAQ
Medium, around 0.90ModerateModerateGeneral customer support
High, around 0.95LowLowPolicy, pricing, technical answers
Exact match onlyLowestNear zeroRegulated or personalized output
Effects of the threshold setting

Realistic hit rates depend on how repetitive the traffic is. FAQ-style assistants with a narrow domain can serve a large share of requests from cache, while open-ended or multi-turn conversations rarely repeat and may see hit rates in the low single digits.

When semantic caching pays off

Semantic caching works best when many users ask the same questions and the correct answer does not depend on who asks or when. Help desks, product documentation assistants, internal policy bots and search-style question answering fit this profile. It is a poor fit for personalized output, conversations whose answer depends on earlier turns, and creative generation where variety is expected.

Conversation context is the most common design question. If only the latest message is embedded, two users who asked “And what about the price?” after different products collide. Robust systems include relevant context in the cache key, such as the product, user role, system prompt version and model version, or bypass the cache for follow-up turns.

Risks and common mistakes

Most production failures come from treating meaning similarity as answer equivalence. Two questions can be close in embedding space while requiring different answers, especially when they differ in a number, negation, date or entity. The table lists frequent problems.

SymptomLikely causeFix
Wrong answer for a similar questionThreshold too lowRaise it and recalibrate on labeled pairs
One user sees another user’s dataCache not scoped per tenant or userPartition the cache by identity
Outdated prices or policies servedMissing or long time to liveExpire or invalidate on content change
Answers ignore conversation historyOnly last message used as keyAdd context to key or skip follow-ups
Little cost saving despite effortTraffic rarely repeatsMeasure hit rate before scaling the cache
Common semantic caching problems

Data leakage across users is the most serious risk, because a cached answer may contain personal or confidential details. Sensitive workloads should scope caches strictly, avoid caching outputs that contain user data, and log hits so false matches can be audited through LLM observability tooling.

The simple explanation

Semantic caching works like an experienced receptionist who has heard the same questions many times. When someone asks something already answered before, even in different words, the receptionist repeats the earlier answer instead of looking it up again. That is faster and cheaper than asking an expert every time.

The catch is that two questions can sound alike but need different answers. A good semantic cache therefore reuses an answer only when the questions are truly close in meaning, and it forgets answers that may have gone out of date.

How it is used in practice

An online electronics retailer runs a support assistant that handles about 300,000 questions per month, many of them variations of the same topics: delivery times, returns, warranty terms and order tracking. The engineering team places a semantic cache in its AI gateway, scoped by country and by the version of the system prompt, with a threshold of 0.93 chosen after labeling 1,000 real question pairs.

Questions that mention an order number or account details bypass the cache entirely, and entries expire after seven days or immediately when the returns policy page changes. After the first month, roughly a quarter of questions are answered from the cache in under 100 milliseconds instead of two to three seconds, and the monthly model bill falls accordingly. Weekly review of a sample of cache hits finds a small number of false matches, mostly about different product models, so the team adds the product category to the cache key.