# LLM response caching: when exact-match pays, when it's wrong

URL: https://infro.io/blog/llm-response-cache-exact-match
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-10-13
Last modified: 2026-10-13
Last reviewed: 2026-10-13 (next review due 2026-11-12)
Topic: Cost and FinOps
Format: Guide

> An exact-match response cache stores the whole answer to an LLM request and replays it when an identical request arrives inside a time-to-live, so the provider is never called for the repeat. It is correct for deterministic, idempotent work — pinned lookups, classification, evaluation runs, client retries — and wrong whenever the answer depends on something outside the key: the caller, the time, data behind a URL, or the variety of sampled output. Key on tenant, target, policy, messages and every generation parameter; keep TTLs no longer than the facts last; measure hits against lookups and report provider calls avoided, never a guess.

Key takeaways:
- A response cache and provider prompt caching are different mechanisms: a response-cache hit never reaches the provider, while a prompt-cache hit is still a provider call that generates a fresh answer.
- The cache key must hold everything that decides the answer — tenant, target, policy, the messages as sent and every generation parameter — and nothing that does not, or it either serves wrong answers or never hits.
- Cache deterministic, idempotent work. Do not cache answers that depend on the caller, the clock, a URL's contents or sampling, unless that dependency is in the key.
- Semantic (similar-prompt) caching trades correctness for hit rate through a similarity threshold, and fixed thresholds give no guarantee about how often a hit is wrong.
- Measure hits over lookups, and report provider calls avoided with an estimated provider cost that stays empty for unpriced requests rather than defaulting to zero.

Most LLM traffic is not repeated. Some of it is, exactly: the same classification prompt over the same ticket after a retry, the same evaluation suite run twice in an afternoon, the same templated lookup that a hundred users trigger with the same inputs. For those requests a response cache can return the stored answer without calling the model at all. The provider is not asked, so it does not bill the repeat, and the caller gets an answer without waiting for generation.

The hard part is not storing answers. It is knowing when a stored answer is still the right answer for the request in front of you. This guide covers the cache key, the cases where a hit is correct and the cases where it quietly serves the wrong thing, semantic caching, expiry, isolation, and how to measure what a cache did. The last section describes INFRO's response cache exactly.

## Two caches with the same word in their name

Provider prompt caching and a gateway response cache solve different problems, and confusing them leads to double-counting the benefit.

Prompt caching happens at the provider. OpenAI reuses a processed prompt prefix when "the entire rendered prefix" matches, from 1,024 input tokens on GPT-5.6 and later, and reports the reused part in `usage.input_tokens_details.cached_tokens`[1]. Anthropic caches the prefix of `tools`, `system` and `messages` up to a breakpoint, for five minutes by default or one hour at a higher write cost[2]. In both cases the model still generates a new answer: OpenAI states that only input prefixes are cached and identical requests are not guaranteed identical outputs[1], and Anthropic that the response "is identical to what you would get if prompt caching were not used"[2].

A response cache sits in front of the provider and stores the finished answer. A hit means no provider call happened. That makes it far more consequential in both directions: the whole call disappears from the provider's bill, and the caller receives an answer the model did not produce for this request.

[Diagram: What each cache stores and who answers on a hit. Provider behaviour from OpenAI's and Anthropic's prompt-caching documentation, read 2026-10-13.]

The two compose. A response-cache miss goes to the provider, where the prompt prefix can still be a prompt-cache hit. The arithmetic of when provider prompt caching pays is in [Prompt caching across providers: when it pays](https://infro.io/blog/prompt-caching-break-even); this article is about the other layer.

## What goes into the key

An exact-match cache is only as correct as its key. The rule is simple to state: every input that can change the answer must be in the key, and nothing that cannot change the answer should be, or identical requests will miss for no reason.

HTTP caching has the same problem in a different form. RFC 9111 defines the cache key as "at a minimum" the request method and target URI, and lets a cache add more material to it[3]. For an LLM call the URI is nearly the same for every request, so almost all of the key has to come from the body.

[Diagram: The parts of a correct exact-match key for a chat completion, from the outside in. The last row is what to leave out.]

A few entries need explanation:

- **Tenant scope.** Organization and project at minimum, plus the environment, so a staging prompt never answers production. If you serve several customers of your own from one project, add a namespace per customer. RFC 9111 makes the same point for shared HTTP caches: a response to a request carrying an `Authorization` header must not be reused for another request unless the response explicitly allows shared storage[3].
- **Target, not the server that answered.** Key on the model or route the caller asked for, including a version for the route, so editing a route's targets retires its old entries. Do not key on which provider account served the first request; that would split identical requests across entries for no reason.
- **The messages as sent.** If a guardrail redacts the prompt before it leaves, key on the redacted version — that is what the model saw. Include tool results and inline images byte for byte. An image referenced by URL is different: what sits behind the URL can change while the request stays identical, so such requests should not be cached.
- **Every forwarded parameter.** `temperature`, `top_p`, `seed`, `stop`, `max_tokens`, `tools`, `tool_choice`, `response_format`, `n`, the provider's `user` field, and any parameter your gateway forwards without modelling. Missing one means two requests that differ only in that parameter share an answer.

What to leave out: the `stream` flag (a streamed and a non-streamed request can share an answer if the cache replays it as a stream), your own metadata and tracing headers, logging switches, and routing hints that choose where a request goes rather than what it asks.

## When a cached answer is correct

A cache is correct where replaying an earlier answer to the same request is what you would have wanted the model to do anyway. That holds for four kinds of traffic:

1. **Deterministic lookups and classifications.** A pinned model asked at `temperature: 0` to classify a fixed text, extract fields from a document, or answer a templated question whose inputs are all in the prompt.
2. **Evaluation and regression runs.** Rerunning a suite after an unrelated change should not pay for the unchanged cases again, and it gives you a stable baseline to compare against.
3. **Client retries.** A client that times out and resends the identical request, or a user who presses the button twice, should get one answer, not two different ones billed twice.
4. **Fan-out duplicates.** Batch jobs and agents frequently ask the same sub-question more than once within a run.

There is a subtlety in "deterministic". Setting temperature to zero does not make a hosted model deterministic. Thinking Machines Lab sampled 1,000 completions at temperature 0 and got 80 distinct outputs, tracing the cause to kernels whose numerical results change with batch size, which varies with server load[4]. For a cache this cuts in a useful direction: replaying the first answer makes repeated requests *more* consistent than calling the model again would. It also means a cache hit is a specific earlier sample, not "the" answer, which matters when you compare cached and uncached runs.

## When a cached answer is wrong

A hit is wrong whenever the answer depended on something that was not in the key. The common cases:

- **Personalised answers.** If the user's identity, plan or history shaped the answer but is not in the messages or the key, one user receives another's answer. Either put the user in the key or do not cache the endpoint.
- **Time-sensitive answers.** "What is today's status", anything that reads a clock, a price feed or a ticket's current state from outside the prompt. If the prompt does not carry the time, the key does not either.
- **Content behind a URL.** The request is byte-for-byte identical while the document or image it points to has changed.
- **Sampled output where variety is the point.** Brainstorming, copy variants, synthetic data. Replaying one sample to every caller defeats the feature.
- **Truncated, filtered or failed answers.** An answer that stopped on the length limit or a content filter, or a stream that was cancelled half way, should never be stored and replayed as if it were complete.
- **Tool-calling turns, with care.** A cached assistant turn that requests a tool call is the model's decision, and the tool results that follow change the next request's messages, so later turns key differently. That is safe for the decision itself. It is not safe if anything downstream treats "the model asked for this tool" as a fresh event and re-executes a side effect.

[Diagram: A first-pass test for whether an endpoint or workload should be cached at all.]

## Semantic caching moves the risk

A semantic cache embeds each prompt and treats a new prompt as a hit when it is close enough to a stored one. It raises the hit rate because paraphrases match. It also changes what a hit means: the cached answer was produced for a *different* question, and you are betting that the difference does not matter.

The authors of vCache, a 2025 paper revised in February 2026, argue that the usual design — one fixed similarity threshold for every request — is the weak point, because "static thresholds do not give formal correctness guarantees" and produce unexpected error rates[5]. Their remedy learns a threshold per cached prompt against an error-rate target, which is a reasonable direction, but the underlying trade does not go away: a lower threshold hits more and is wrong more often. Two prompts that differ by a negation, an account type or a date can sit very close in embedding space.

If you adopt semantic caching, restrict it to answers that tolerate approximation (FAQ-style help text, not account actions), sample hits for review, and keep the same tenant scoping and TTL discipline an exact-match cache needs. A semantic cache does nothing about staleness either; a correct match to an outdated answer is still outdated.

## Expiry and invalidation

Every entry needs a time-to-live, and the right TTL is the length of time the underlying facts stay true, not the longest the store allows. A classification of a frozen document can live for weeks. An answer about a policy that legal edits monthly should not outlive the next edit.

TTL alone is not enough, because the things that make old answers wrong are usually your own changes: a new system prompt, a different model behind a route, a changed guardrail. Put the versions of those things in the key, so a change makes old entries unreachable without deleting anything. For the remaining cases, keep a purge that works by bumping a generation number in the key rather than scanning the store; it takes effect everywhere at once.

Tell the caller what happened. HTTP requires a cache that serves a stored response to add an `Age` header, whose presence implies the response "was not generated or validated by the origin server for this request"[3]. An LLM gateway should do the equivalent: mark the response as a hit, say how old it is, and name the request that produced it.

## Isolation and privacy

A response cache stores model output, which can contain anything the prompt contained. Treat it as stored customer content:

- **Isolation by construction.** Put the tenant identifiers inside the hashed key so that a lookup cannot match another tenant's entry even if the lookup code is wrong elsewhere.
- **Timing is a signal.** A hit is faster than a miss. Researchers auditing provider prompt caches detected "global cache sharing across users in seven API providers", where timing differences could leak information about other users' prompts[6]. A response cache shared across tenants would leak in the same way, and far more directly; scope it per tenant and you remove the channel.
- **Store less.** The prompt only needs to exist as an input to the hash. Store the answer, its token counts and what produced it.
- **Respect opt-outs.** A request that asks not to be logged should not be written to the cache, and turning the cache on should not quietly turn on prompt capture.

## Measuring what the cache did

Two numbers describe a cache honestly:

- **Hit rate** is hits divided by lookups, where lookups are hits plus misses. Requests that were never eligible — sampled, too large, opted out — do not belong in the denominator, but they belong in a report of their own, because they are the reason a hit rate is low.
- **Provider calls avoided** is the hit count. Each hit can carry an *estimated provider cost avoided*: what the request that produced the stored answer was estimated to cost at the provider's rate. When that source request had no known price, the avoided cost is unknown, not zero, and should be counted separately rather than folded into a total.

Keep latency apart as well. A hit's latency is a storage lookup; mixing it into provider latency percentiles makes the provider look faster than it is.

> **Example: Reading a week of cache metrics**
> Suppose a support-classification endpoint makes 40,000 cache lookups in a week and 9,200 are hits: a 23% hit rate, and 9,200 provider calls avoided. If 9,000 of those hits came from priced source requests estimated at $0.0012 each, the estimated provider cost avoided is about $10.80, and the remaining 200 hits are reported as unpriced. Another 6,000 requests on the endpoint were sampled at temperature 0.7 and never eligible; they appear in neither number.

## Response caching in INFRO

INFRO's gateway has an opt-in, exact-match response cache for `POST /v1/chat/completions`, streaming and non-streaming. A hit is answered by INFRO and never reaches your provider, so your provider does not bill it. It is separate from provider prompt caching, which still happens at your provider on every miss. There is no semantic matching.

**Turning it on.** The cache is off by default. An owner or admin turns it on for the organization in the console settings, and each project can `inherit`, turn it `on` or turn it `off`. The organization also sets a default TTL and a scope. The same fields exist on the management API: `response_cache_enabled`, `response_cache_default_ttl_seconds` and `response_cache_scope` on `org_settings`, and `response_cache_override` on a project.

**Plans and TTLs.** Every plan includes the cache; the plan sets the longest TTL an entry may have. The shortest is 60 seconds. A per-request TTL above the plan's ceiling is refused with `400 cache_ttl_too_long`.

[Diagram: The longest response-cache TTL each published plan allows, from INFRO's plan definitions on 2026-10-13.]

**Scope.** `deterministic_only`, the default, caches only requests that set `temperature: 0` or a `seed`. `all` also caches sampled requests, and then replays the first sampled answer to every identical request until it expires — choose it only where one fixed answer is acceptable.

**The key.** A SHA-256 hash over the organization, project, the calling key's environment, organization and project purge generations, your optional namespace, the target (the model, or the route's id, version and the if-rule that matched, with any `routing.providers` selection), a fingerprint of the guardrail policies in force, the messages after input guardrails, and every forwarded generation parameter. `stream`, `metadata`, `logging`, routing hints, the `cache` field itself and `X-INFRO-*` headers are not in it. `max_tokens` is keyed as you sent it.

**Per-request control.** Send a top-level `cache` object (`extra_body` in the OpenAI SDKs) or the equivalent headers. A request can narrow what settings allow and never turn the cache on where they have it off.

```json
{
  "model": "support-answer",
  "temperature": 0,
  "messages": [{ "role": "user", "content": "How long do refunds take?" }],
  "cache": { "mode": "default", "ttl_seconds": 3600, "namespace": "tenant-42" }
}
```

`mode` is `default`, `bypass`, `refresh` (call the provider and replace the entry) or `read_only`. The headers are `X-INFRO-Cache`, `X-INFRO-Cache-TTL` and `X-INFRO-Cache-Namespace`; the body wins field by field, and a malformed header is `400 invalid_cache_control`.

**What is never cached.** A request carrying an image by `http(s)` URL; a sampled request under `deterministic_only`; an answer that finished with `length` or `content_filter`; one an output guardrail changed, or one whose guardrail check failed open; a cancelled or failed stream; an answer without reported usage; anything over 1 MiB stored; and any request sent with `logging: false`, which may still be answered from the cache but is never written to it. Each appears on its request record as `uncacheable` with the reason.

**What the response says.** `X-INFRO-Cache` is `HIT`, `MISS`, `BYPASS`, `DISABLED` or `UNCACHEABLE`; a hit adds `X-INFRO-Cache-Age` and `X-INFRO-Cache-Source`. The body carries a `cache` object, and on a hit `usage` keeps the original token counts while `usage.cost` is `0` with `price_source: "gateway_cache"`.

**Budgets and limits.** The lookup runs after authentication, allowlists and input guardrails, so a refused request is refused even when an answer is cached. A hit takes no budget reservation, but it counts against the key's rate limit and toward the plan's monthly request allowance.

**Isolation and storage.** Entries live in Cloudflare Workers KV, one namespace per environment, under keys that include your organization's id. The prompt is not stored in plaintext; only the hash is kept. Purge an organization or project from the console or `POST /manage/v1/org_settings/{organization_id}/purge_cache` and `POST /manage/v1/projects/{id}/purge_cache`; a purge raises a generation in the key, so older entries stop matching at once. The cache fails open: a lookup that errors or takes more than 50 ms is a miss, and your provider answers.

**Measurement.** Request records carry `cache.status`, `source_request_id`, `provider_billed: false` and `estimated_provider_cost_avoided`, which is `null` when the source request was unpriced; `GET /v1/requests` filters on `cache=hit|miss|bypass|uncacheable|disabled`. Usage analytics add `cache_hits`, `cache_lookups`, `estimated_provider_cost_avoided` and `cache_hits_unpriced`, and report hit latency separately from provider latency. Hits are left out of cost reconciliation and the FOCUS export, because your provider never saw them. The full contract is on the [response cache docs page](https://infro.io/docs/features/caching).

## Sources

1. Prompt caching — OpenAI. https://developers.openai.com/api/docs/guides/prompt-caching (accessed 2026-10-13)
2. Prompt caching — Anthropic. https://platform.claude.com/docs/en/build-with-claude/prompt-caching (accessed 2026-10-13)
3. RFC 9111: HTTP Caching — IETF. https://www.rfc-editor.org/rfc/rfc9111.html (accessed 2026-10-13)
4. Defeating Nondeterminism in LLM Inference — Thinking Machines Lab. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ (accessed 2026-10-13)
5. vCache: Verified Semantic Prompt Caching — arXiv. https://arxiv.org/abs/2502.03771 (accessed 2026-10-13)
6. Auditing Prompt Caching in Language Model APIs — arXiv. https://arxiv.org/abs/2502.07776 (accessed 2026-10-13)