# Is semantic caching safe for LLM apps? Hit rates, risks, tests

URL: https://infro.io/blog/is-semantic-caching-safe-for-llm-apps-hit-rates-risks-tests
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-11-04
Last modified: 2026-11-04
Last reviewed: 2026-11-04 (next review due 2027-05-03)
Topic: Cost and FinOps
Format: Methodology

> Semantic caching is safe only when the cache key is scoped by tenant, policy and model before similarity is computed, and when you have measured a false-hit rate per similarity band on your own traffic. Run it in shadow mode first, and compare the marginal saving against provider prompt caching, not against a cold call.

Key takeaways:
- Scope the cache key by tenant, policy, model and prompt version as hard filters, and only then compare embeddings inside that scope.
- Measure false-hit rate per similarity band on labeled pairs from your own traffic, and report an upper confidence bound, not the point estimate.
- Run the cache in shadow mode first: compute would-be hits, still call the provider, and compare the two answers.
- Price the saving against a request that already benefits from provider prompt caching, because a cached-prefix read is already a fraction of an input token.
- Treat a cross-tenant hit as a security incident, not a quality defect.

## What semantic caching adds, and what it risks

An exact-match cache returns a stored answer only when everything that decides the answer is identical. INFRO's response cache works this way: the same organization, project and key environment, the same model or route, the same guardrail policy, the same messages and the same generation parameters, and it says plainly that there is no semantic matching.[1] A semantic cache relaxes the "same messages" part. It embeds the prompt, finds the nearest stored prompt, and returns that prompt's answer if the similarity score clears a threshold.

That relaxation is where all the risk lives. Exact matching can only be wrong by being stale. Similarity matching can also be wrong by being *different*: "Can I cancel after 30 days?" and "Can I cancel within 30 days?" embed very close together and have opposite answers. And if the scope of the lookup is wider than it should be, the wrong answer can belong to another customer.

So the question "is it safe?" splits into three tests you can run before shipping:

1. Does the key keep tenants and policies apart? (a design check)
2. At the threshold you plan to use, how often is a hit wrong? (a measurement)
3. After provider prompt caching, is the remaining saving worth the first two? (arithmetic)

The rest of this article is a procedure for each.

## Step 1: scope the key before you compare vectors

Similarity should only ever rank candidates *inside* a scope that is matched exactly. Never put tenant identity into the embedding text and hope the vector separates it. Treat these as hard filters applied before the nearest-neighbor search:

- **Tenant or end customer.** Different customers get different cache partitions, always.
- **Policy.** The guardrail policy in force, and any allowlist or compliance setting that changes what an answer may contain. INFRO's exact-match key includes the guardrail policy for this reason.[1]
- **Model or route.** An answer from one model is not a stand-in for another, and a silent model swap changes the answer distribution.
- **System prompt and tool set version.** Hash them. A prompt edit must invalidate the partition.
- **Generation parameters.** Temperature, response format, max tokens.
- **Retrieved context identity.** If the answer depends on retrieved documents or user data, the document ids and their access-control scope belong in the key. Otherwise a hit replays an answer built from documents the new caller may not be allowed to read.
- **Locale and language,** if the answer text depends on them.

```python
import hashlib, json

def scope_key(*, tenant_id, policy_id, model, system_prompt, tools,
              params, context_ids, locale):
    material = {
        "tenant": tenant_id,
        "policy": policy_id,
        "model": model,
        "sys": hashlib.sha256(system_prompt.encode()).hexdigest(),
        "tools": hashlib.sha256(json.dumps(tools, sort_keys=True).encode()).hexdigest(),
        "params": params,            # temperature, response_format, max_tokens
        "ctx": sorted(context_ids),  # retrieved doc ids the answer depends on
        "locale": locale,
    }
    blob = json.dumps(material, sort_keys=True).encode()
    return hashlib.sha256(blob).hexdigest()

# Nearest-neighbor search runs only over entries stored under this exact key.
```

OWASP's 2025 list names Sensitive Information Disclosure and Vector and Embedding Weaknesses as separate risks for LLM applications.[2] A shared vector index across tenants sits on both. Treat any hit that crosses a tenant partition as a security incident with its own alert, not as a quality defect, and test it directly: seed tenant A with a distinctive prompt and answer, send the near-identical prompt as tenant B, and assert a miss.

Also decide what is never eligible. INFRO's exact cache already refuses to store truncated or filtered answers (`length`, `content_filter`), responses that output guardrails modified, and requests with images behind a URL, because the content behind a URL can change.[1] Copy those exclusions, and add your own: tool-calling turns with side effects, answers that quote user-specific data, and anything non-deterministic where variety is the point.

## Step 2: measure the false-hit rate by similarity band

A hit rate alone tells you how often the cache fires. It says nothing about whether it fired correctly. The number that matters is the **false-hit rate**: of the requests answered from the cache, the share where the cached answer would not have been acceptable for the new prompt.

Build the measurement from your own traffic, since embedding models, domains and phrasing differ.

1. Sample a few thousand production prompts per route, within one scope (one tenant, one policy, one prompt version).
2. For each prompt, find its nearest neighbor among the others in the same scope, and record the similarity score.
3. Bucket the pairs by score into bands, for example 0.80–0.85 up to 0.98–1.00.
4. Sample pairs from every band, weighting toward the bands you might choose as a threshold.
5. For each pair, get the real provider answer for the second prompt and ask: would the first prompt's answer have been acceptable here? Label with humans, or with an LLM judge you have calibrated against humans first.[1] See [calibrating graders](https://infro.io/blog/can-an-llm-judge-be-trusted-calibrating-graders-against-humans).
6. Compute the false-hit rate per band with a confidence interval.

Report the upper bound. A band with zero observed bad hits in 60 pairs does not have a zero false-hit rate; with 95% confidence it could be near 5%.

```python
from math import sqrt

def wilson_upper(bad, n, z=1.96):
    if n == 0:
        return 1.0
    p = bad / n
    centre = p + z * z / (2 * n)
    margin = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return (centre + margin) / (1 + z * z / n)

# For each band: wilson_upper(bad_hits, labeled_pairs)
```

> **Example: Illustrative labeling result**
> These counts are made up to show the shape of the output, not measured results. Your bands will differ.
> 
> | Similarity band | Labeled pairs | Unacceptable | Observed rate | 95% upper bound |
> | --- | --- | --- | --- | --- |
> | 0.85–0.90 | 200 | 31 | 15.5% | 21% |
> | 0.90–0.95 | 200 | 9 | 4.5% | 8% |
> | 0.95–0.98 | 300 | 3 | 1.0% | 3% |
> | 0.98–1.00 | 300 | 0 | 0.0% | 1.3% |
> 
> With a budget of 2% on this route, only the top band qualifies, and the threshold becomes 0.98, not the 0.90 a library default might suggest.

Two refinements matter in practice. First, thresholds differ by intent: short factual lookups tolerate a looser threshold than prompts containing numbers, dates, negations or named entities. A cheap guard is a pre-check that extracts numbers, dates and identifiers from both prompts and refuses a hit when they differ. Second, a change of embedding model changes every score, so every threshold must be re-measured after one.

## Step 3: run in shadow mode before serving a single hit

The offline sample estimates the false-hit rate on historical traffic. Shadow mode confirms it on live traffic without exposing users to it.

In shadow mode the lookup runs and logs "would have hit, cached entry X, score s", but the request still goes to the provider. You then compare the provider's fresh answer with the cached one for a sampled share of would-be hits, again with a calibrated judge or human review. Record, per request: the scope key, score, whether a would-be hit occurred, the label, and the estimated provider cost of the call you were *not* going to avoid yet.

Per-request cost and latency come from your gateway's request records; INFRO records model, latency, token counts and estimated provider cost for each request,[3] and prompt and completion logging is off by default with a retention window you choose,[3] so decide deliberately which comparison data you store and for how long. Keep the labeled comparison set under the same retention rules as the rest of your prompt logs.

Exit criteria for shadow mode, written before it starts:

- would-be hit rate stable across at least a full weekly cycle,
- upper-bound false-hit rate under budget in every band you intend to serve,
- zero cross-tenant would-be hits,
- no partition whose hit rate is dominated by one stale entry.

## Step 4: price the saving against prompt caching, not a cold call

Finance owners often compute the benefit as "hit rate × cost of the call". That overstates it if the workload already benefits from provider prompt caching. With prompt caching the provider still serves the request but bills the repeated prefix at its cached-input rate, and INFRO's docs give ballpark cached reads of roughly 0.1× the input rate for OpenAI, Anthropic and Google, and lower for DeepSeek, with writes costing more than ordinary input.[4] Those rates are the provider's, change over time, and your invoice is the authority.

A semantic hit avoids the *remaining* cost: the uncached part of the input (the new user message), all output tokens, and the latency. On a long-prefix, short-answer workload most input tokens are already cheap, so the marginal saving is smaller than the headline. On a short-prompt, long-answer workload, output dominates and the cache matters more.

> **Example: Illustrative break-even**
> All figures are invented to show the arithmetic.
> 
> Take one request that costs 1.00 unit with no caching. With prompt-cache hits on the prefix, assume it costs 0.55 units. Assume a semantic lookup (embedding call plus vector search) costs 0.01 units per request, and that each served false hit costs 3.00 units in support time and rework.
> 
> At a 15% served hit rate and a 1% false-hit rate among hits:
> 
> - avoided provider cost: 0.15 × 0.55 = 0.0825
> - lookup cost on every request: 0.01
> - false-hit cost: 0.15 × 0.01 × 3.00 = 0.0045
> 
> Net: about 0.068 units per request, roughly 12% of the 0.55 baseline. Against the uncached 1.00 baseline, the same hit rate looks like a 15% gain, which is the number that gets quoted and is wrong for this workload.

Embedding lookups are model calls too. If you embed through your gateway, INFRO's `/v1/embeddings` takes OpenAI's request shape on your own connection,[3] so the lookup cost shows up in the same per-request records as everything else. For the FinOps view, the FinOps Foundation lists a working-group asset on managing AI token costs from direct model provider APIs,[5] which is the right place to align on how to report cache-avoided spend against the actual invoice.

## Step 5: operate it like a data store with a wrong-answer budget

After launch, three things drift: your prompts, your documents and your embedding model.

- **TTL and invalidation.** Pick the TTL from how fast the underlying facts change, not from the longest the system allows. INFRO's exact cache has a 60-second floor and plan-dependent ceilings, from one hour to 30 days,[1] and a semantic cache deserves a shorter TTL than an exact one because each entry can be reused for many different phrasings. Purge by partition when the system prompt, policy or source documents change.
- **Continuous sampling.** Keep auditing a small share of served hits against fresh answers, and alert when the upper-bound false-hit rate crosses budget. This is the same loop as shadow mode, kept running.
- **Attribution.** Tag hits by tenant and feature so a bad partition is findable. Show hit rate, false-hit rate and avoided cost per route side by side; a rising hit rate with a rising false-hit rate is a threshold set too low, not a success.

The INFRO exact-match cache is a safer first step for most teams: a hit there is by construction the same request, and the gateway records the cache status per request.[1] See [exact-match caching](https://infro.io/blog/llm-response-cache-exact-match) and the [prompt caching break-even](https://infro.io/blog/prompt-caching-break-even) for those two layers. Add a semantic layer only on routes where the measured numbers in steps 2 to 4 justify it.

## When to say no

Skip semantic caching, or exclude the route, when any of these hold:

- answers depend on per-user state, entitlements or retrieved private documents you cannot put in the key,
- a wrong answer is costly (money movement, medical, legal, account changes),
- prompts are long and unique, so few neighbors exist and the hit rate will be low,
- the workload is already dominated by cheap cached-prefix input and short outputs.

Good candidates are the opposite: high-volume, public-knowledge questions inside one tenant's partition, with a stable system prompt, where a slightly different phrasing really does deserve the same answer.

## Questions

### What false-hit rate is acceptable?

There is no universal number. Set it per route from the cost of a wrong answer: a store-hours FAQ can tolerate more than anything touching billing, legal or account data. Write the budget down before the test, and require the upper confidence bound, not the observed rate, to sit under it.

### Can a semantic layer sit in front of an exact-match gateway cache?

They are independent layers. INFRO's response cache is exact-match and does no similarity matching, so a semantic layer would be your own code in front of it, with its own key scoping, TTL and purge. Anything it returns never reaches the gateway, so those requests do not appear in gateway usage records unless you log them yourself.

## Sources

1. Response cache — Features — INFRO Docs. https://infro.io/docs/features/caching (accessed 2026-11-04)
2. OWASP Top 10 for LLM Applications 2025 — OWASP Gen AI Security Project. https://genai.owasp.org/llm-top-10/ (accessed 2026-11-04)
3. INFRO documentation overview — INFRO Docs. https://infro.io/docs (accessed 2026-11-04)
4. Prompt caching — Features — INFRO Docs. https://infro.io/docs/features/prompt-caching (accessed 2026-11-04)
5. The FinOps Foundation — FinOps Foundation. https://www.finops.org/ (accessed 2026-11-04)