Prompt caching

Most providers cache the repeated portion of a prompt and bill it at their cached-input rate when a later request reuses it. INFRO passes this through automatically — OpenAI, Anthropic, DeepSeek, and Google style caching all work with no INFRO-side configuration. The cached rate is your provider's, on your provider's own invoice, and INFRO takes no part of it. It shows up in usage.cost on every response.

There is nothing to enable and no INFRO-specific cache flag. If your requests share a stable prefix and land on a provider that caches, your provider bills the cached part at its cached-input rate. This page covers structuring prompts so caches actually hit, how Anthropic's explicit cache_control blocks are handled today, and how to confirm the cache is being read.

How it works

Caching happens at the provider, not INFRO. When a provider recognizes the start of an incoming request as one it processed recently, it skips recomputing that prefix and bills those input tokens at a reduced rate. INFRO forwards your request unchanged, reads the cached-token counts the provider reported, and prices them at the cached rate in usage.cost — see Chat completions for the full response shape.

Providers set these rates and change them over time, in both directions and without notice — treat the numbers above as ballpark, and note that a cache write costs more than an ordinary input token while a cache read costs less. The authoritative figure is your provider's own invoice; usage.cost estimates it per request. Per-model list rates come from the pricing object in GET /v1/models; see Models for the catalog format.

Caches live with the upstream that served the request, so hits depend on consecutive requests reaching the same connection. That is the common case — the first target serves while it is healthy — but a failover to another connection resets the cache for that prefix, and a cold prefix is billed as a cold prefix.

Structure prompts for cache hits

Provider caches match on the exact token prefix of a request. Only the portion identical to a recent request — starting from the very beginning — can hit the cache. The practical rule: stable content first, volatile content last.

1. System prompt — identical on every request. No timestamps, request IDs, or user names interpolated into it.
2. Tool definitions — same tools, in the same order, with the same descriptions.
3. Conversation history — append-only, so earlier turns keep matching the cached prefix as the conversation grows.
4. The newest user message — the volatile part goes last.

Anthropic cache_control

Anthropic models use explicit cache breakpoints rather than fully automatic caching. Mark where the cacheable prefix ends with a cache_control field on a text content part. INFRO keeps it when it translates the request for Anthropic: on text parts of system, user, assistant and tool messages, and on a tool definition (next to function or inside it) to cache every tool declared up to that one. cache_control must be on a content part, not on the message object, so give the message an array of text parts rather than a plain string. Anthropic allows up to four breakpoints per request; one at the end of your stable prefix is usually enough.

cache_controlobject

Attach to a text content part (system, user, assistant or tool message) or to a tool definition to mark the end of a cacheable prefix. Meaningful on anthropic/... models only — providers with automatic caching ignore it.

cache_control.typestringrequired

The cache type. Use "ephemeral".

{
  "model": "anthropic/claude-sonnet-5",
  "messages": [
    {
      "role": "system",
      "content": [
        {
          "type": "text",
          "text": "Support policy for Acme: <several thousand tokens of policy>",
          "cache_control": {
            "type": "ephemeral"
          }
        }
      ]
    },
    {
      "role": "user",
      "content": "How do I reset my password?"
    }
  ]
}

The first request writes the cache; later requests within its lifetime that share the exact prefix read from it at the cached-input rate. Everything up to and including the marked block is cached — content after it is processed normally.

Verify caching is working

The reliable signal is usage.cost. Send the same request twice in quick succession and compare: with a prompt long enough to cache, the second call should estimate noticeably lower. prompt_tokens does not move — cached tokens are still prompt tokens — but usage.prompt_tokens_details.cached_tokens reports how many of them the provider served from its cache, on both streaming and non-streaming responses. Watch that and cost, not prompt_tokens.

# request.json holds a long, stable system prompt plus a user message
for i in 1 2; do
  curl -s https://api.infro.io/v1/chat/completions \
    -H "Authorization: Bearer $INFRO_API_KEY" \
    -H "Content-Type: application/json" \
    -d @request.json | jq '.usage.cost'
done

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.infro.io/v1",
    api_key=os.environ["INFRO_API_KEY"],
)

long_prompt = open("policy.md").read()  # several thousand tokens, identical every call

for attempt in (1, 2):
    response = client.chat.completions.create(
        model="openai/gpt-5.6-sol",
        messages=[
            {"role": "system", "content": long_prompt},
            {"role": "user", "content": "Summarize section 4."},
        ],
    )
    print(f"call {attempt}: ${response.usage.cost}")

import fs from "node:fs";
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.infro.io/v1",
  apiKey: process.env.INFRO_API_KEY,
});

const longPrompt = fs.readFileSync("policy.md", "utf8"); // identical every call

for (const attempt of [1, 2]) {
  const response = await client.chat.completions.create({
    model: "openai/gpt-5.6-sol",
    messages: [
      { role: "system", content: longPrompt },
      { role: "user", content: "Summarize section 4." },
    ],
  });
  console.log(`call ${attempt}: $${(response.usage as any).cost}`);
}

On streaming requests, pass stream_options with {"include_usage": true} to receive a final chunk whose usage includes prompt_tokens, completion_tokens, and cost — see Streaming. If the second call's estimate does not drop, check that the prefix is byte-for-byte identical and that both requests hit the same provider — the top-level provider field on the response tells you.

How cached input is billed

Providers generally bill cached input at around a tenth of their uncached input rate, lower on some models, varying by provider and model. Output tokens are always billed at the full rate — the cached rate applies only to the cached portion of the prompt.

• Caches are short-lived — typically minutes, depending on the provider. Steady traffic benefits most; a request arriving long after the previous one usually pays full price.
• Short prompts may never cache. Providers enforce minimum prefix lengths, commonly around 1,024 tokens.
• Cache writes are billed ABOVE the normal input rate, not below it — 1.25× on Anthropic and on OpenAI's GPT-5.6 models, and 2× for Anthropic's one-hour entries — so the request that populates a cache costs more than an uncached one. Caching pays for itself from the first read at the five-minute rate, and from the second at the one-hour rate.
• Caching is provider-side compute reuse, independent of INFRO logging: requests sent with logging: false are still billed at the provider's cached rate when they hit its cache. See Privacy.