Is semantic caching safe for LLM apps? Hit rates, risks, tests

What semantic caching adds, and what it risks

An exact-match cache returns a stored answer only when everything that decides the answer is identical. INFRO's response cache works this way: the same organization, project and key environment, the same model or route, the same guardrail policy, the same messages and the same generation parameters, and it says plainly that there is no semantic matching.¹ A semantic cache relaxes the "same messages" part. It embeds the prompt, finds the nearest stored prompt, and returns that prompt's answer if the similarity score clears a threshold.

That relaxation is where all the risk lives. Exact matching can only be wrong by being stale. Similarity matching can also be wrong by being different: "Can I cancel after 30 days?" and "Can I cancel within 30 days?" embed very close together and have opposite answers. And if the scope of the lookup is wider than it should be, the wrong answer can belong to another customer.

So the question "is it safe?" splits into three tests you can run before shipping:

1. Does the key keep tenants and policies apart? (a design check)
2. At the threshold you plan to use, how often is a hit wrong? (a measurement)
3. After provider prompt caching, is the remaining saving worth the first two? (arithmetic)

The rest of this article is a procedure for each.

Step 1: scope the key before you compare vectors

Similarity should only ever rank candidates inside a scope that is matched exactly. Never put tenant identity into the embedding text and hope the vector separates it. Treat these as hard filters applied before the nearest-neighbor search:

• Tenant or end customer. Different customers get different cache partitions, always.
• Policy. The guardrail policy in force, and any allowlist or compliance setting that changes what an answer may contain. INFRO's exact-match key includes the guardrail policy for this reason.¹
• Model or route. An answer from one model is not a stand-in for another, and a silent model swap changes the answer distribution.
• System prompt and tool set version. Hash them. A prompt edit must invalidate the partition.
• Generation parameters. Temperature, response format, max tokens.
• Retrieved context identity. If the answer depends on retrieved documents or user data, the document ids and their access-control scope belong in the key. Otherwise a hit replays an answer built from documents the new caller may not be allowed to read.
• Locale and language, if the answer text depends on them.

import hashlib, json

def scope_key(*, tenant_id, policy_id, model, system_prompt, tools,
              params, context_ids, locale):
    material = {
        "tenant": tenant_id,
        "policy": policy_id,
        "model": model,
        "sys": hashlib.sha256(system_prompt.encode()).hexdigest(),
        "tools": hashlib.sha256(json.dumps(tools, sort_keys=True).encode()).hexdigest(),
        "params": params,            # temperature, response_format, max_tokens
        "ctx": sorted(context_ids),  # retrieved doc ids the answer depends on
        "locale": locale,
    }
    blob = json.dumps(material, sort_keys=True).encode()
    return hashlib.sha256(blob).hexdigest()

# Nearest-neighbor search runs only over entries stored under this exact key.

OWASP's 2025 list names Sensitive Information Disclosure and Vector and Embedding Weaknesses as separate risks for LLM applications.² A shared vector index across tenants sits on both. Treat any hit that crosses a tenant partition as a security incident with its own alert, not as a quality defect, and test it directly: seed tenant A with a distinctive prompt and answer, send the near-identical prompt as tenant B, and assert a miss.

Also decide what is never eligible. INFRO's exact cache already refuses to store truncated or filtered answers (length, content_filter), responses that output guardrails modified, and requests with images behind a URL, because the content behind a URL can change.¹ Copy those exclusions, and add your own: tool-calling turns with side effects, answers that quote user-specific data, and anything non-deterministic where variety is the point.

Step 2: measure the false-hit rate by similarity band

A hit rate alone tells you how often the cache fires. It says nothing about whether it fired correctly. The number that matters is the false-hit rate: of the requests answered from the cache, the share where the cached answer would not have been acceptable for the new prompt.

Build the measurement from your own traffic, since embedding models, domains and phrasing differ.

1. Sample a few thousand production prompts per route, within one scope (one tenant, one policy, one prompt version).
2. For each prompt, find its nearest neighbor among the others in the same scope, and record the similarity score.
3. Bucket the pairs by score into bands, for example 0.80–0.85 up to 0.98–1.00.
4. Sample pairs from every band, weighting toward the bands you might choose as a threshold.
5. For each pair, get the real provider answer for the second prompt and ask: would the first prompt's answer have been acceptable here? Label with humans, or with an LLM judge you have calibrated against humans first.¹ See calibrating graders.
6. Compute the false-hit rate per band with a confidence interval.

Report the upper bound. A band with zero observed bad hits in 60 pairs does not have a zero false-hit rate; with 95% confidence it could be near 5%.

from math import sqrt

def wilson_upper(bad, n, z=1.96):
    if n == 0:
        return 1.0
    p = bad / n
    centre = p + z * z / (2 * n)
    margin = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return (centre + margin) / (1 + z * z / n)

# For each band: wilson_upper(bad_hits, labeled_pairs)

Two refinements matter in practice. First, thresholds differ by intent: short factual lookups tolerate a looser threshold than prompts containing numbers, dates, negations or named entities. A cheap guard is a pre-check that extracts numbers, dates and identifiers from both prompts and refuses a hit when they differ. Second, a change of embedding model changes every score, so every threshold must be re-measured after one.

Step 3: run in shadow mode before serving a single hit

The offline sample estimates the false-hit rate on historical traffic. Shadow mode confirms it on live traffic without exposing users to it.

In shadow mode the lookup runs and logs "would have hit, cached entry X, score s", but the request still goes to the provider. You then compare the provider's fresh answer with the cached one for a sampled share of would-be hits, again with a calibrated judge or human review. Record, per request: the scope key, score, whether a would-be hit occurred, the label, and the estimated provider cost of the call you were not going to avoid yet.

Per-request cost and latency come from your gateway's request records; INFRO records model, latency, token counts and estimated provider cost for each request,³ and prompt and completion logging is off by default with a retention window you choose,³ so decide deliberately which comparison data you store and for how long. Keep the labeled comparison set under the same retention rules as the rest of your prompt logs.

Exit criteria for shadow mode, written before it starts:

• would-be hit rate stable across at least a full weekly cycle,
• upper-bound false-hit rate under budget in every band you intend to serve,
• zero cross-tenant would-be hits,
• no partition whose hit rate is dominated by one stale entry.

Step 4: price the saving against prompt caching, not a cold call

Finance owners often compute the benefit as "hit rate × cost of the call". That overstates it if the workload already benefits from provider prompt caching. With prompt caching the provider still serves the request but bills the repeated prefix at its cached-input rate, and INFRO's docs give ballpark cached reads of roughly 0.1× the input rate for OpenAI, Anthropic and Google, and lower for DeepSeek, with writes costing more than ordinary input.⁴ Those rates are the provider's, change over time, and your invoice is the authority.

A semantic hit avoids the remaining cost: the uncached part of the input (the new user message), all output tokens, and the latency. On a long-prefix, short-answer workload most input tokens are already cheap, so the marginal saving is smaller than the headline. On a short-prompt, long-answer workload, output dominates and the cache matters more.

Embedding lookups are model calls too. If you embed through your gateway, INFRO's /v1/embeddings takes OpenAI's request shape on your own connection,³ so the lookup cost shows up in the same per-request records as everything else. For the FinOps view, the FinOps Foundation lists a working-group asset on managing AI token costs from direct model provider APIs,⁵ which is the right place to align on how to report cache-avoided spend against the actual invoice.

Step 5: operate it like a data store with a wrong-answer budget

After launch, three things drift: your prompts, your documents and your embedding model.

• TTL and invalidation. Pick the TTL from how fast the underlying facts change, not from the longest the system allows. INFRO's exact cache has a 60-second floor and plan-dependent ceilings, from one hour to 30 days,¹ and a semantic cache deserves a shorter TTL than an exact one because each entry can be reused for many different phrasings. Purge by partition when the system prompt, policy or source documents change.
• Continuous sampling. Keep auditing a small share of served hits against fresh answers, and alert when the upper-bound false-hit rate crosses budget. This is the same loop as shadow mode, kept running.
• Attribution. Tag hits by tenant and feature so a bad partition is findable. Show hit rate, false-hit rate and avoided cost per route side by side; a rising hit rate with a rising false-hit rate is a threshold set too low, not a success.

The INFRO exact-match cache is a safer first step for most teams: a hit there is by construction the same request, and the gateway records the cache status per request.¹ See exact-match caching and the prompt caching break-even for those two layers. Add a semantic layer only on routes where the measured numbers in steps 2 to 4 justify it.

When to say no

Skip semantic caching, or exclude the route, when any of these hold:

• answers depend on per-user state, entitlements or retrieved private documents you cannot put in the key,
• a wrong answer is costly (money movement, medical, legal, account changes),
• prompts are long and unique, so few neighbors exist and the hit rate will be low,
• the workload is already dominated by cheap cached-prefix input and short outputs.

Good candidates are the opposite: high-volume, public-knowledge questions inside one tenant's partition, with a stable system prompt, where a slightly different phrasing really does deserve the same answer.