The production LLM cost optimization playbook

An LLM bill grows for reasons that are hard to see from the invoice: a prompt template gained a paragraph, a retrieval step started returning twice as many chunks, an agent loop got one step longer, a feature launched on the model someone picked for the prototype. The invoice reports the total. The decisions that moved it were made in code.

This playbook orders cost work by risk. The early steps change nothing a user would notice; the later ones change behaviour and need evidence. It makes no promise about how far your spend will fall, because that depends entirely on how your traffic is shaped — but every lever here is one you can measure before and after.

Start with the right unit: cost per successful task

Price per million tokens is a supplier's unit. What matters to a product is the cost of getting a task done: a resolved ticket, a valid extraction, a finished agent run. Consumption-based pricing is exactly what makes AI spend hard to manage with traditional budgeting, since billing is tied to units such as tokens processed or requests made¹.

cost per successful task = (total estimated cost of all attempts for a workload)
                           ÷ (number of tasks that met the workload's success criteria)

The numerator includes retries, failed attempts and validation failures. The denominator only counts tasks that passed. That is what makes it honest: a model that costs little per call but fails often, or needs a second pass, cannot hide.

Step 1: Measure and attribute before changing anything

You cannot optimize traffic you cannot see. For one representative week, record for every request:

• the workload or feature that sent it, as metadata on the request;
• input and output tokens separately, and cached tokens where the provider reports them;
• the model and the provider account that served it;
• whether the task succeeded, from your application's own validation.

Then look for the usual shapes: a few workloads producing most of the tokens; output-heavy workloads where the output rate dominates; a long tail of very large prompts; retries that nobody counted. Attribution by owner is the foundation of the wider practice covered in the LLM FinOps guide. When the numbers go to people who will not read a request log, report them as estimates against budget with the unpriced share stated; the AI spend report leadership reads describes that format.

Step 2: Remove waste before changing models

These changes rarely affect quality and usually take less work than a model migration:

• Trim conversation history. Resending a whole transcript each turn grows input with every message. Keep recent turns plus a running summary.
• Retrieve less, better. A retrieval step can easily send far more context than the answer needs. Measure answer quality as you reduce the number of chunks, and add a reranker before raising the count.
• Cap output. Set a maximum output length per workload, ask for structured output instead of prose where a program reads the result, and remove instructions that invite long preambles.
• Prune tool results. Agent loops often carry every previous tool result forward. Keep what the next step needs.
• Delete dead prompt text. Instructions for features that no longer exist still cost tokens on every request.

Step 3: Use provider prompt caching deliberately

Providers bill a repeated prompt prefix at a fraction of their ordinary input rate, and the mechanics are published:

• OpenAI enables prompt caching by default on supported models. On GPT-5.6 and later, a cache read costs 0.1 times the uncached input rate (0.05 times on GPT-6.1 Sol) and a cache write 1.25 times, with a 30-minute lifetime and a 1,024-token minimum; earlier models bill reads at a model-specific cached-input rate and charge nothing extra for writes².
• Anthropic prices writing to the cache at 1.25 times the base input rate for a five-minute lifetime and 2 times for one hour, and reading from it at 0.1 times the base input rate — 0.05 times on Claude Opus 5.5 and Sonnet 5.5, and 0.025 times on Claude Fable 5.1 and Mythos 5.1³.

Those multipliers make the break-even point easy to check. For a cached prefix of a given size, at the standard 1.25 write and 0.1 read:

The write premium is new on OpenAI's side, and it changes one old habit: a prefix that is written and seldom read again now costs more than no cache at all, on both providers.

Three practices make caching work in production:

1. Put stable content first. Caches match prefixes, so system instructions, tool definitions and reference documents belong before anything that changes per request.
2. Keep prefixes byte-identical. A timestamp or request id near the top of a prompt defeats the cache on every call.
3. Keep cached traffic on one account. A cache belongs to the account that wrote it. Spreading a cache-heavy workload across accounts for capacity reasons can raise cost; weigh the two deliberately.

Step 4: Answer exact repeats once

Prompt caching makes a repeated prefix less expensive to process; the provider still runs the model. A response cache goes further for one narrow class of traffic: when an identical request arrives again, it returns the stored answer and the provider is never called. That is correct for deterministic, idempotent work — a classification at temperature 0, a lookup with pinned inputs, a client retry of a request that already succeeded — and wrong whenever the answer depends on anything outside the request: the caller, the time, a document behind a URL, or the variety a sampled answer is supposed to have.

Three rules keep it honest:

• Key on everything that changes the answer: tenant, model or route, the full messages and every generation parameter.
• Keep the time-to-live no longer than the facts last. An answer about a refund policy is wrong the day the policy changes.
• Count what it avoided, not what it might have. Report hits against lookups, and the estimated provider cost of the calls that did not happen.

Where it fits and where it does not is covered in detail in LLM response caching: when exact-match pays.

Step 5: Move non-interactive work to batch

Work with no user waiting — nightly summaries, enrichment, evaluation runs, backfills — rarely needs real-time pricing. OpenAI prices its Batch API at half the cost of its synchronous APIs, with a 24-hour completion window⁴. Anthropic charges batch usage at 50% of standard prices, and batches expire if processing does not complete within 24 hours⁵.

The trade is latency and a different failure mode: a batch returns results per item, some of which may fail and need resubmission. If a job runs on a schedule, it is worth checking whether it needs synchronous pricing at all.

Step 6: Right-size models with evidence

Only now change models, one workload at a time. Build an evaluation from real requests, run the current model and a lower-rate candidate on the same inputs, and compare cost per successful task, not rates. Three patterns tend to hold up:

• Tier by task. Classification, extraction and routing decisions are often served well by smaller models; multi-step reasoning and customer-facing prose may not be. Decide per workload, with results.
• Escalate on signals. Start with the efficient model and send only failed or high-risk requests to the stronger one. See quality-aware routing.
• Classify before routing. When one workload mixes easy and hard requests, a small classifier can label each prompt and send it to the model its class needs. It adds a call to every request, so it needs a timeout, a default class, and its own line in the cost per successful task. The trade-offs are in classifiers in the request path.

The method for building the evaluation is in how to evaluate LLMs on real workloads.

Step 7: Estimate the worst case, then put a ceiling on it

Cost work lowers the typical bill; ceilings bound the bad day. A looping agent, a leaked key or a misconfigured batch job can consume a month's budget in hours. Unbounded consumption is recognised as its own risk category for LLM applications⁶.

A ceiling can only refuse a request before it runs if something estimates that request's cost before it runs. For text that estimate is a range: input tokens can be counted, but output is unknown until the model stops, so admission has to use the worst case — the output ceiling at the output rate — while previews and forecasts use the expected figure. A request with no max_tokens has a worst case set by the model's maximum output, which is why setting one is itself a cost control. When no rate is known, the honest estimate is "unpriced", not zero. The arithmetic for each modality is in estimating LLM cost before the request.

• Use alerts for planned growth and hard ceilings for containment.
• Scope ceilings narrowly — per key, per project, per environment — so one runaway workload does not stop unrelated production traffic.
• Decide what happens at the ceiling: refuse, queue, or degrade to a smaller model you have evaluated. Details are in LLM spend controls.

Step 8: Reconcile estimates against what providers bill

Per-request cost estimates are what budgets and attribution run on, and they are only as good as the rates behind them. Several providers publish organization-level cost reporting you can compare them against:

• OpenAI exposes an organization Costs endpoint with daily buckets⁷, which requires an Admin API key that cannot be used for non-administration endpoints⁸.
• Anthropic's Usage and Cost API requires Admin API credentials, reports costs in USD as decimal strings in cents at daily granularity, and excludes Priority Tier costs from the cost endpoint⁹.

The useful output is the variance between your estimate and the provider's figure for the same period. A small, scattered difference is expected. A sustained gap in one direction usually means a rate is wrong — a negotiated rate not entered, a cache-write premium not priced — and fixing the rate fixes every future estimate.

Levers in order of risk

What not to do

• Do not compare models on list price alone. Failure rates, output length and retries change the answer.
• Do not trust a reduction percentage you did not measure — including one in an article like this.
• Do not spread cached traffic for tidiness. Keep it where its cache lives unless capacity forces otherwise.
• Do not cut context blindly. Measure answer quality as you reduce it.
• Do not response-cache sampled or personalised answers. One user's answer replayed to another is an incident, not a cost control.

Cost control in INFRO: estimates, rates, caps and the response cache

INFRO does not change what your providers charge: requests run on the accounts you connect, your providers invoice you directly, and INFRO charges a subscription with no share of model spend. What it adds is the measurement and control around that spend.

• Per-request estimates. Every response carries usage.cost and every request record its token counts and estimated cost, with price_source saying whether the rate came from your own rate for that provider (custom_provider), your rate for the model on any provider (custom), or the model vendor's list rate (list). A model with no rate is recorded as unpriced, null rather than zero.
• Estimates before the call. POST /v1/estimates answers what a request would cost on your own provider account for text, image, video, speech or transcription quantities, without calling the provider.
• Your negotiated rates. From the Team plan, you can enter the rates your contracts carry so estimates use them.
• Attribution. GET /v1/usage groups estimated cost by model, project, key, provider, connection or route, and request metadata such as a feature tag or user id is filterable in the request log.
• Budgets. Budgets at organization, project, member and key level can alert or block. A blocking budget admits a request only if its worst-case estimated cost fits what remains, and otherwise refuses it with 402 and error.type budget_exceeded. A request that could reach a model with no price is refused while a blocking budget applies (code unpriced_model_under_budget), unless the organization's owner chooses to serve unpriced traffic uncounted; entering a rate is the better fix.
• Response cache. An opt-in, exact-match cache for POST /v1/chat/completions, off until an owner or admin turns it on. By default it caches only requests with temperature: 0 or a seed. A hit is answered without a provider call, so your provider does not bill it; its record carries provider_billed: false and estimated_provider_cost_avoided, which is null when the original request was unpriced. Every plan includes it, and the plan sets the longest time-to-live.
• Classifier routing. A route can classify each prompt on a decision model you connect and send each class to its own targets, with a fallback class on a timeout or low confidence. Each classifier call is its own row in the request log with its own estimated cost.
• Reconciliation. On the Business plan, estimated spend is compared with the cost reported by OpenAI's and Anthropic's cost APIs, using a separate admin credential.

Where a provider reports cached or cache-write tokens, the estimate prices them at the corresponding rate rather than the plain input rate. See spend controls, billing and rates, response cache and analytics.