Response cache

The response cache stores the answer to a chat completion and returns it when the same request arrives again inside the time-to-live (TTL). Identical requests are answered from the cache and never reach your provider, so your provider does not bill them. INFRO charges nothing per hit: there is no per-token price and no markup, and the cache is part of the platform subscription.

It is exact-match. A request is answered from the cache only when everything that decides the answer is identical: the same organization, project and key environment, the same model or route, the same guardrail policy, the same messages and the same generation parameters. A request that differs by one character is a different request. There is no semantic (similar-prompt) matching.

Turn it on

The cache is off by default. An owner or admin turns it on for the organization in the console settings, and each project can inherit that, turn it on, or turn it off (inherit, on, off). The organization setting also holds the default TTL (empty means the plan's maximum) and the scope described below. The same fields are on the management API: response_cache_enabled, response_cache_default_ttl_seconds and response_cache_scope on org_settings, and response_cache_override on a project.

Every plan includes the cache. The plan sets the longest TTL an entry may have. Turning the cache on, or setting a project to on, needs a plan that includes it; turning it off never needs anything. A stored default above the plan's maximum (after a plan change) is shortened to the maximum, never refused.

The shortest TTL is 60 seconds. A per-request TTL above the plan's maximum is refused with 400 and code cache_ttl_too_long. Setting a default above it in the console or the management API is refused with the same code.

What is cached

Only POST /v1/chat/completions, streaming and non-streaming. A cache field sent to any other endpoint is accepted and ignored. A response is stored when all of these hold:

• The response was an HTTP 200 and every choice finished with stop or tool_calls.
• The provider reported usage.
• Output guardrails left the response untouched.
• The stored entry is at most 1 MiB.
• The request was cacheable in the first place (next section), and it was not sent with logging: false or in read_only mode.

What is never cached

A request or response that cannot be cached is served by your provider as usual. Its request record says cache.status: "uncacheable" with one of these reasons:

A request is also looked up and stored only when the cache is on for its project. A lookup that fails or takes longer than 50 ms counts as a miss with reason lookup_error or lookup_timeout, and your provider answers — see When the cache is unavailable.

The cache key

The key is a SHA-256 hash over the fields below. Two requests share an entry only when every one of them is identical.

user is in the key, so requests for different end users do not share entries unless they send the same user. Leave it out, or send the same value, only where the same answer is right for everyone.

The key names what you asked for, not which connection answered. On a route with several targets, a failover can answer on another target, and that answer is then replayed to later identical requests on the same route version; the response's provider and connection name the connection that produced it originally.

Scope: deterministic only, or all

The organization's scope decides which requests are eligible. deterministic_only, the default, caches only requests that set temperature: 0 or a seed. all caches every otherwise cacheable request, including sampled ones: the first answer is stored and the same sampled answer is then replayed to every identical request until the entry expires. Choose all only where one fixed answer is acceptable for every caller.

Per-request control

A request can narrow what the settings allow; it can never turn the cache on where the settings have it off. A cache field sent while the cache is off for the project is accepted, has no effect, and the response says DISABLED, so code that sends it can ship before the setting is turned on. Send it as a top-level body field (extra_body in the OpenAI SDKs), which other OpenAI-compatible backends ignore:

cache.modestring

default reads and writes. bypass neither reads nor writes. refresh skips the read, calls your provider and replaces the stored answer. read_only reads, and never writes.

cache.ttl_secondsinteger

60 to 2592000, and at most your plan's maximum; above it is 400 cache_ttl_too_long. Applies to the entry this request writes.

cache.namespacestring

1 to 128 characters of letters, digits and . _ : -. Partitions the key, for example per tenant of your own product.

When both are sent, the body wins, field by field. A malformed header is refused with 400 and code invalid_cache_control. logging: false means "do not store this request's content": the request can still be answered from the cache, and its answer is never written to it.

curl https://api.infro.io/v1/chat/completions \
  -H "Authorization: Bearer $INFRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "support-answer",
    "temperature": 0,
    "messages": [{"role": "user", "content": "How long do refunds take?"}],
    "cache": {"ttl_seconds": 3600, "namespace": "tenant-42"}
  }'

What the response says

Every chat response that reached the cache step carries X-INFRO-Cache: HIT, MISS, BYPASS, DISABLED or UNCACHEABLE. A hit also carries X-INFRO-Cache-Age (seconds since the entry was written) and X-INFRO-Cache-Source (the req_ id of the request that produced it). A request refused before that step, by authentication or policy, carries none.

The body has a top-level cache field, absent when the cache is off for the request: {"status": "miss"}, "bypass" or "uncacheable", or on a hit {"status": "hit", "age_seconds", "source_request_id", "provider_billed": false}.

{
  "id": "req_5c0e1f7a2b9d4c83",
  "object": "chat.completion",
  "created": 1790000000,
  "model": "openai/gpt-5.6-terra",
  "route": "primary",
  "route_name": "support-answer",
  "provider": "openai",
  "connection": "conn_4b1e9a07",
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "Refunds are processed within 5 business days." },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 412,
    "completion_tokens": 11,
    "total_tokens": 423,
    "cost": 0,
    "price_source": "gateway_cache"
  },
  "cache": {
    "status": "hit",
    "age_seconds": 12,
    "source_request_id": "req_9f27c1d0a4b84e6f",
    "provider_billed": false
  }
}

• id is this request's own id and created is now. Everything else is the stored answer.
• usage carries the original token counts, so token totals stay comparable. usage.cost is 0 and usage.price_source is gateway_cache: a known zero, because your provider was not called.
• model, provider, connection and route describe what produced the answer originally. route_name is the route this request named.

Streaming

A streamed hit is replayed as a stream: each choice's whole message arrives as one delta, then its finish chunk, then — with stream_options.include_usage — the usage chunk carrying usage and cache, then [DONE]. Time to first token on a hit is not comparable with a live stream. The X-INFRO-Cache header on a stream is what is known when headers are sent: a MISS can still end uncacheable (a cancelled or failed stream), and the usage chunk and the request record say which.

Budgets, rate limits and your plan

The lookup happens after the request is authenticated and checked, and before the budget check. So a request from a revoked key, an archived project or a lapsed subscription, a model outside an allowlist, a request an input guardrail blocks, or a region conflict is refused even when an answer is cached. The model or route must still be servable, because the lookup happens after it is resolved.

• A hit takes no budget reservation and is served even when a blocking budget is exhausted. A spend control limits what your provider can bill you; a hit is not sent to your provider and adds nothing to that bill.
• Rate limits still apply. A hit counts against the key's requests per minute like any other request.
• Hits count toward your plan's monthly request allowance. The request log, analytics and /v1/usage show them separately.

Purge

Purge the whole organization or one project from the console settings, or through the management API: POST /manage/v1/org_settings/{organization_id}/purge_cache and POST /manage/v1/projects/{id}/purge_cache. A purge raises a generation number that is part of every key, so every older entry stops matching immediately, everywhere. The old entries are not read again and expire on their own TTL. A purge is recorded in the audit log as response_cache.purged, and it does not change the resource's version or ETag.

Editing a route, or changing a guardrail policy, has the same effect for the requests it covers, because both are in the key.

When the cache is unavailable

The cache fails open. If a lookup errors or takes longer than 50 ms, the request goes to your provider as if there were no cache, and is recorded as a miss with reason lookup_error or lookup_timeout. A failed write is logged and not reported to the caller. An unavailable cache store never makes a request fail.

Where hits show up

Estimated provider cost avoided is, for each hit, the estimated provider cost of the request that produced the answer — what your provider would likely have billed had the request reached it, at the same rates INFRO uses for every estimate. It is an estimate, never a figure your provider reports. It is null for a hit whose source was unpriced, and cache_hits_unpriced counts those, so the total is never padded with a guess.

What INFRO stores

A cache entry holds the response content, its token counts, the connection that produced it and that request's estimated provider cost. The prompt is not stored in plaintext: it is one input to the key's SHA-256 hash, and the hash is all that is kept of it. Entries live in Cloudflare Workers KV, in a namespace per environment. Each key is a hash that includes your organization's id, so one organization's requests can never match another's entries.

• Every entry expires at its TTL — at most the plan's maximum above. A purge makes entries unreachable at once; they are then deleted when their TTL runs out.
• The cache is a separate opt-in from payload logging. Turning the cache on does not turn on prompt capture, and turning capture off does not turn the cache off.
• A request sent with logging: false is never written to the cache.
• Turning the cache off stops reads and writes for that scope; existing entries are not read and expire on their TTL. Purge first if you want them unreachable immediately.