LLM response caching: when exact-match pays, when it's wrong

Most LLM traffic is not repeated. Some of it is, exactly: the same classification prompt over the same ticket after a retry, the same evaluation suite run twice in an afternoon, the same templated lookup that a hundred users trigger with the same inputs. For those requests a response cache can return the stored answer without calling the model at all. The provider is not asked, so it does not bill the repeat, and the caller gets an answer without waiting for generation.

The hard part is not storing answers. It is knowing when a stored answer is still the right answer for the request in front of you. This guide covers the cache key, the cases where a hit is correct and the cases where it quietly serves the wrong thing, semantic caching, expiry, isolation, and how to measure what a cache did. The last section describes INFRO's response cache exactly.

Two caches with the same word in their name

Provider prompt caching and a gateway response cache solve different problems, and confusing them leads to double-counting the benefit.

Prompt caching happens at the provider. OpenAI reuses a processed prompt prefix when "the entire rendered prefix" matches, from 1,024 input tokens on GPT-5.6 and later, and reports the reused part in usage.input_tokens_details.cached_tokens¹. Anthropic caches the prefix of tools, system and messages up to a breakpoint, for five minutes by default or one hour at a higher write cost². In both cases the model still generates a new answer: OpenAI states that only input prefixes are cached and identical requests are not guaranteed identical outputs¹, and Anthropic that the response "is identical to what you would get if prompt caching were not used"².

A response cache sits in front of the provider and stores the finished answer. A hit means no provider call happened. That makes it far more consequential in both directions: the whole call disappears from the provider's bill, and the caller receives an answer the model did not produce for this request.

The two compose. A response-cache miss goes to the provider, where the prompt prefix can still be a prompt-cache hit. The arithmetic of when provider prompt caching pays is in Prompt caching across providers: when it pays; this article is about the other layer.

What goes into the key

An exact-match cache is only as correct as its key. The rule is simple to state: every input that can change the answer must be in the key, and nothing that cannot change the answer should be, or identical requests will miss for no reason.

HTTP caching has the same problem in a different form. RFC 9111 defines the cache key as "at a minimum" the request method and target URI, and lets a cache add more material to it³. For an LLM call the URI is nearly the same for every request, so almost all of the key has to come from the body.

A few entries need explanation:

• Tenant scope. Organization and project at minimum, plus the environment, so a staging prompt never answers production. If you serve several customers of your own from one project, add a namespace per customer. RFC 9111 makes the same point for shared HTTP caches: a response to a request carrying an Authorization header must not be reused for another request unless the response explicitly allows shared storage³.
• Target, not the server that answered. Key on the model or route the caller asked for, including a version for the route, so editing a route's targets retires its old entries. Do not key on which provider account served the first request; that would split identical requests across entries for no reason.
• The messages as sent. If a guardrail redacts the prompt before it leaves, key on the redacted version — that is what the model saw. Include tool results and inline images byte for byte. An image referenced by URL is different: what sits behind the URL can change while the request stays identical, so such requests should not be cached.
• Every forwarded parameter. temperature, top_p, seed, stop, max_tokens, tools, tool_choice, response_format, n, the provider's user field, and any parameter your gateway forwards without modelling. Missing one means two requests that differ only in that parameter share an answer.

What to leave out: the stream flag (a streamed and a non-streamed request can share an answer if the cache replays it as a stream), your own metadata and tracing headers, logging switches, and routing hints that choose where a request goes rather than what it asks.

When a cached answer is correct

A cache is correct where replaying an earlier answer to the same request is what you would have wanted the model to do anyway. That holds for four kinds of traffic:

1. Deterministic lookups and classifications. A pinned model asked at temperature: 0 to classify a fixed text, extract fields from a document, or answer a templated question whose inputs are all in the prompt.
2. Evaluation and regression runs. Rerunning a suite after an unrelated change should not pay for the unchanged cases again, and it gives you a stable baseline to compare against.
3. Client retries. A client that times out and resends the identical request, or a user who presses the button twice, should get one answer, not two different ones billed twice.
4. Fan-out duplicates. Batch jobs and agents frequently ask the same sub-question more than once within a run.

There is a subtlety in "deterministic". Setting temperature to zero does not make a hosted model deterministic. Thinking Machines Lab sampled 1,000 completions at temperature 0 and got 80 distinct outputs, tracing the cause to kernels whose numerical results change with batch size, which varies with server load⁴. For a cache this cuts in a useful direction: replaying the first answer makes repeated requests more consistent than calling the model again would. It also means a cache hit is a specific earlier sample, not "the" answer, which matters when you compare cached and uncached runs.

When a cached answer is wrong

A hit is wrong whenever the answer depended on something that was not in the key. The common cases:

• Personalised answers. If the user's identity, plan or history shaped the answer but is not in the messages or the key, one user receives another's answer. Either put the user in the key or do not cache the endpoint.
• Time-sensitive answers. "What is today's status", anything that reads a clock, a price feed or a ticket's current state from outside the prompt. If the prompt does not carry the time, the key does not either.
• Content behind a URL. The request is byte-for-byte identical while the document or image it points to has changed.
• Sampled output where variety is the point. Brainstorming, copy variants, synthetic data. Replaying one sample to every caller defeats the feature.
• Truncated, filtered or failed answers. An answer that stopped on the length limit or a content filter, or a stream that was cancelled half way, should never be stored and replayed as if it were complete.
• Tool-calling turns, with care. A cached assistant turn that requests a tool call is the model's decision, and the tool results that follow change the next request's messages, so later turns key differently. That is safe for the decision itself. It is not safe if anything downstream treats "the model asked for this tool" as a fresh event and re-executes a side effect.

Semantic caching moves the risk

A semantic cache embeds each prompt and treats a new prompt as a hit when it is close enough to a stored one. It raises the hit rate because paraphrases match. It also changes what a hit means: the cached answer was produced for a different question, and you are betting that the difference does not matter.

The authors of vCache, a 2025 paper revised in February 2026, argue that the usual design — one fixed similarity threshold for every request — is the weak point, because "static thresholds do not give formal correctness guarantees" and produce unexpected error rates⁵. Their remedy learns a threshold per cached prompt against an error-rate target, which is a reasonable direction, but the underlying trade does not go away: a lower threshold hits more and is wrong more often. Two prompts that differ by a negation, an account type or a date can sit very close in embedding space.

If you adopt semantic caching, restrict it to answers that tolerate approximation (FAQ-style help text, not account actions), sample hits for review, and keep the same tenant scoping and TTL discipline an exact-match cache needs. A semantic cache does nothing about staleness either; a correct match to an outdated answer is still outdated.

Expiry and invalidation

Every entry needs a time-to-live, and the right TTL is the length of time the underlying facts stay true, not the longest the store allows. A classification of a frozen document can live for weeks. An answer about a policy that legal edits monthly should not outlive the next edit.

TTL alone is not enough, because the things that make old answers wrong are usually your own changes: a new system prompt, a different model behind a route, a changed guardrail. Put the versions of those things in the key, so a change makes old entries unreachable without deleting anything. For the remaining cases, keep a purge that works by bumping a generation number in the key rather than scanning the store; it takes effect everywhere at once.

Tell the caller what happened. HTTP requires a cache that serves a stored response to add an Age header, whose presence implies the response "was not generated or validated by the origin server for this request"³. An LLM gateway should do the equivalent: mark the response as a hit, say how old it is, and name the request that produced it.

Isolation and privacy

A response cache stores model output, which can contain anything the prompt contained. Treat it as stored customer content:

• Isolation by construction. Put the tenant identifiers inside the hashed key so that a lookup cannot match another tenant's entry even if the lookup code is wrong elsewhere.
• Timing is a signal. A hit is faster than a miss. Researchers auditing provider prompt caches detected "global cache sharing across users in seven API providers", where timing differences could leak information about other users' prompts⁶. A response cache shared across tenants would leak in the same way, and far more directly; scope it per tenant and you remove the channel.
• Store less. The prompt only needs to exist as an input to the hash. Store the answer, its token counts and what produced it.
• Respect opt-outs. A request that asks not to be logged should not be written to the cache, and turning the cache on should not quietly turn on prompt capture.

Measuring what the cache did

Two numbers describe a cache honestly:

• Hit rate is hits divided by lookups, where lookups are hits plus misses. Requests that were never eligible — sampled, too large, opted out — do not belong in the denominator, but they belong in a report of their own, because they are the reason a hit rate is low.
• Provider calls avoided is the hit count. Each hit can carry an estimated provider cost avoided: what the request that produced the stored answer was estimated to cost at the provider's rate. When that source request had no known price, the avoided cost is unknown, not zero, and should be counted separately rather than folded into a total.

Keep latency apart as well. A hit's latency is a storage lookup; mixing it into provider latency percentiles makes the provider look faster than it is.

Response caching in INFRO

INFRO's gateway has an opt-in, exact-match response cache for POST /v1/chat/completions, streaming and non-streaming. A hit is answered by INFRO and never reaches your provider, so your provider does not bill it. It is separate from provider prompt caching, which still happens at your provider on every miss. There is no semantic matching.

Turning it on. The cache is off by default. An owner or admin turns it on for the organization in the console settings, and each project can inherit, turn it on or turn it off. The organization also sets a default TTL and a scope. The same fields exist on the management API: response_cache_enabled, response_cache_default_ttl_seconds and response_cache_scope on org_settings, and response_cache_override on a project.

Plans and TTLs. Every plan includes the cache; the plan sets the longest TTL an entry may have. The shortest is 60 seconds. A per-request TTL above the plan's ceiling is refused with 400 cache_ttl_too_long.

Scope. deterministic_only, the default, caches only requests that set temperature: 0 or a seed. all also caches sampled requests, and then replays the first sampled answer to every identical request until it expires — choose it only where one fixed answer is acceptable.

The key. A SHA-256 hash over the organization, project, the calling key's environment, organization and project purge generations, your optional namespace, the target (the model, or the route's id, version and the if-rule that matched, with any routing.providers selection), a fingerprint of the guardrail policies in force, the messages after input guardrails, and every forwarded generation parameter. stream, metadata, logging, routing hints, the cache field itself and X-INFRO-* headers are not in it. max_tokens is keyed as you sent it.

Per-request control. Send a top-level cache object (extra_body in the OpenAI SDKs) or the equivalent headers. A request can narrow what settings allow and never turn the cache on where they have it off.

{
  "model": "support-answer",
  "temperature": 0,
  "messages": [{ "role": "user", "content": "How long do refunds take?" }],
  "cache": { "mode": "default", "ttl_seconds": 3600, "namespace": "tenant-42" }
}

mode is default, bypass, refresh (call the provider and replace the entry) or read_only. The headers are X-INFRO-Cache, X-INFRO-Cache-TTL and X-INFRO-Cache-Namespace; the body wins field by field, and a malformed header is 400 invalid_cache_control.

What is never cached. A request carrying an image by http(s) URL; a sampled request under deterministic_only; an answer that finished with length or content_filter; one an output guardrail changed, or one whose guardrail check failed open; a cancelled or failed stream; an answer without reported usage; anything over 1 MiB stored; and any request sent with logging: false, which may still be answered from the cache but is never written to it. Each appears on its request record as uncacheable with the reason.

What the response says. X-INFRO-Cache is HIT, MISS, BYPASS, DISABLED or UNCACHEABLE; a hit adds X-INFRO-Cache-Age and X-INFRO-Cache-Source. The body carries a cache object, and on a hit usage keeps the original token counts while usage.cost is 0 with price_source: "gateway_cache".

Budgets and limits. The lookup runs after authentication, allowlists and input guardrails, so a refused request is refused even when an answer is cached. A hit takes no budget reservation, but it counts against the key's rate limit and toward the plan's monthly request allowance.

Isolation and storage. Entries live in Cloudflare Workers KV, one namespace per environment, under keys that include your organization's id. The prompt is not stored in plaintext; only the hash is kept. Purge an organization or project from the console or POST /manage/v1/org_settings/{organization_id}/purge_cache and POST /manage/v1/projects/{id}/purge_cache; a purge raises a generation in the key, so older entries stop matching at once. The cache fails open: a lookup that errors or takes more than 50 ms is a miss, and your provider answers.

Measurement. Request records carry cache.status, source_request_id, provider_billed: false and estimated_provider_cost_avoided, which is null when the source request was unpriced; GET /v1/requests filters on cache=hit|miss|bypass|uncacheable|disabled. Usage analytics add cache_hits, cache_lookups, estimated_provider_cost_avoided and cache_hits_unpriced, and report hit latency separately from provider latency. Hits are left out of cost reconciliation and the FOCUS export, because your provider never saw them. The full contract is on the response cache docs page.