A finance partner asks what the model spend was last month. Somebody opens the gateway dashboard and reads out a number to the cent. Somebody else opens the provider console and reads out a different one. The meeting resolves the way these meetings always resolve — the invoice is the real number, the dashboard is "roughly right", and everyone moves on.
Nobody writes down the gap. Nobody checks whether it was the same gap last month. And because nobody checks, a rate that the dashboard has never modelled correctly can sit there for two quarters, quietly making every per-customer margin figure in the company slightly wrong in the same direction.
This is not an argument that dashboards are bad. It is an argument that a number produced one way and a number produced another way should be compared, and that the comparison is rarely made.
Your cost figure is a model, not a measurement
Nearly every AI gateway and LLM observability tool computes cost the same way. The provider returns token counts on the response. The tool multiplies those counts by a rate it holds in a table. It stores the product and shows it to you. Some upstreams put a cost figure in the response itself; most return only the counts.
That is the whole mechanism. There is no billing feed involved and no charge being observed. What you are looking at is a rate table applied to token counts — a model of the provider's price list, maintained by people who do not work at the provider, evaluated against counts the provider reported for its own purposes.
You can see the shape of this in how the tools describe themselves. Cloudflare AI Gateway lets you enter custom costs, and its documentation says plainly that they "can reflect your negotiated input, output, cache-read, and cache-write rates" and "override the default or public model costs"1. The default, in other words, is a public list rate. The same page notes that a response served from its cache always shows a cost of zero, even when a custom cost is configured1 — a sensible choice, and one more place where the figure on the screen and the figure on the invoice are answering different questions.
None of that is a defect. A rate table is the right design: it is immediate, it is per-request, and it can be attributed to a customer or a feature, which an invoice never can. It is also the only thing that can answer before a request is made, which is what estimating cost before the request depends on. The defect is treating its output as though it were the bill.
Why this stopped being an accounting curiosity
The FinOps Foundation's 2026 survey — 1,192 respondents, more than $83 billion in annual cloud spend between them — found that 98% now manage AI spend, against 63% a year earlier2. FinOps for AI is the report's top forward-looking priority, and AI cost management is the top skill respondents say their teams need to build2.
The three obstacles they name, in order, are visibility into AI costs, allocating those costs to business units, and determining what the spend returned2. Visibility comes first. Not optimisation, not commitment management — knowing what the number is.
Two years ago an AI bill was small enough that being 7% out did not matter to anyone. Now it is a line item that a finance team has opinions about, and "our dashboard says a different thing from your invoice" is a conversation that ends with the dashboard being distrusted rather than fixed.
The seven places an estimate drifts
| Source of drift | Direction | Why the table misses it |
|---|---|---|
| Prompt caching | Both ways at once | Cache writes cost more than plain input, cache reads cost far less, and the counts arrive in separate fields |
| Negotiated rates | Estimate too high | The table holds the public list price; your contract does not |
| Service tiers | Estimate too high or absent | Batch and priority pricing differ from standard, and at least one tier is missing from a cost endpoint entirely |
| Rates set by the request | Either | Claude Haiku 5.5 prices a prompt over 100,000 tokens at five times the rate below it3; DeepSeek charges twice its off-peak rate in published peak hours4. A rate held per model cannot follow either |
| Day boundaries and settlement | Either, then cancels | A request at 23:59 UTC lands in different days in the two systems; some charges settle late |
| Unpriced models | Estimate too low | A model with no rate contributes zero to the estimate and a real number to the invoice |
| Traffic the gateway never sees | Estimate too low | Console use, playgrounds, evaluation scripts, another team's key on the same account |
Two of these deserve more than a row.
Prompt caching is the big one
Caching is where most rate tables are wrong, because it is priced as a multiple of the base input rate rather than as its own number, and because the token counts arrive split across fields that are easy to add up incorrectly.
Anthropic's published multipliers are precise: a cache write costs 1.25× the base input price at a five-minute TTL and 2× at one hour, and a cache read costs 0.1× — except on Claude Opus 5.5 and Sonnet 5.5, where reads are 0.05×, and on Claude Fable 5.1 and Mythos 5.1, where they are 0.025×5. The read multiplier is a property of the model, not of the vendor, so a table that stores one multiplier per vendor is wrong for some of that vendor's models. The usage object reports cache_creation_input_tokens and cache_read_input_tokens separately from input_tokens, and the documentation is explicit that input_tokens counts only the tokens after the last cache breakpoint, so the true total is the sum of all three5.
Now consider a table that models the read discount but not the write premium — a very easy table to write, because the discount is the part everyone markets. Every cached token is priced correctly on the way out and 25% too cheap on the way in. The estimate is low, and it is low specifically on the workloads that cache hardest, which are the large-system-prompt agent workloads that dominate a modern bill. The error is not random; it is concentrated exactly where the money is.
Rates you were given are not rates you were quoted
A negotiated rate is invisible to any default table by construction. So is a committed-use discount, a promotional balance, a regional premium and a tier you were moved onto. Every one of these makes the estimate high, which is the less alarming direction and therefore the one nobody investigates.
What the providers will actually tell you
Both major providers publish organization-level cost reporting, behind a credential that is deliberately not the one your application uses. This is the part most teams do not know exists.
| OpenAI Costs API | Anthropic Cost API | |
|---|---|---|
| Endpoint | /v1/organization/costs6 | /v1/organizations/cost_report7 |
| Credential | Admin key6 | Admin key; workspace keys are refused7 |
| Time parameters | start_time, end_time in Unix seconds6 | starting_at, ending_at as RFC 33397 |
| Bucket width | Daily only on this endpoint6 | Daily only7 |
| Grouping | Line item, project6 | Workspace, or description, which adds parsed model and inference_geo fields7 |
| Amount | amount.value as a number, with amount.currency6 | Decimal strings in cents7 |
| Known exclusions | — | Priority Tier costs are not in the cost endpoint7 |
| Pagination | next_page cursor, passed back as page6 | has_more and next_page7 |
Three details in that table have bitten us and will bite you.
The units differ, and one of them is unusual. OpenAI gives you an amount object whose value is a number in the currency named beside it6. Anthropic gives you decimal strings "in lowest units (cents)"7. Read one as the other and you are out by a factor of a hundred — in a figure whose entire purpose is to be surprising, so the usual instinct that a number looks wrong does not fire. Convert at the edge of each adapter, and write a test that asserts the two conversions do not collapse into one.
Daily is as fine as it gets. Neither cost endpoint will tell you what a request cost. That is not a limitation to work around; it is the reason the two numbers are complementary. The provider knows the total and nothing about your customers. Your gateway knows the customer and is estimating the total. Reconciliation is the join.
There are documented holes. Anthropic states that Priority Tier costs use a different billing model and are not in the cost endpoint7, that playground usage carries a null api_key_id, and that the default workspace reports a null workspace_id7. It also notes that the programmatic endpoints are not currently available on Claude Platform on AWS, that code execution appears in the cost endpoint but not in the usage endpoint, and that Claude Enterprise organizations read their cost through a separate Analytics API7. An account using any of these has a variance that will never close, and a reconciliation that does not say so is lying by omission.
Worth noting, because it settles the question of whether this is an exotic use of the API: Anthropic's own documentation lists "Cost reconciliation: Match internal records with Anthropic billing for finance and accounting teams" as an intended purpose of the endpoint7. The capability is not obscure. The practice is.
What a variance report is, and what it is not
It is not a dashboard tile. A single day's gap is almost always timing — a request served at 23:59 UTC that lands on one side in one system and the other side in the other, a charge that settles hours late, a batch that finalises. Those cancel out across a week and mean nothing individually. Publishing them as a daily percentage produces a number that moves for reasons unrelated to what it measures, which is the fastest way to teach people to ignore it.
What means something is direction. If your estimate is consistently 7% under on one provider for most of a week, that is not scatter averaging out. That is a rate you do not model.
So the useful test has four conditions, and each one excludes a specific false positive:
- Enough days. Five out of seven, so a day-boundary effect cannot carry it.
- A money floor per day. A day with forty cents of spend can be 80% out for entirely uninteresting reasons, and reporting it buries the account spending four thousand dollars.
- All leaning the same way. Ordinary scatter produces days in both directions; an unmodelled rate does not.
- Every request in the window priced. An unpriced request makes the estimate low by an unknown amount, so a window containing one has a variance that is partly a coverage gap. "Your rates are wrong" and "we have no rate for this model" are different findings with different fixes.
That fourth condition is the one most likely to be dropped for being inconvenient, and it is the one that keeps the report honest. Report unpriced requests as their own number, next to the variance, always.
A worked example of being wrong
The cache-write premium above is not a hypothetical we chose for the article. It is our own defect.
Our own rate table modelled the cache read discount from the beginning, because it is the part every provider publishes prominently. It did not model the cache write premium. Anthropic folds cache_creation_input_tokens into the prompt total — which it must, because that sum is the published total5 — and our pricing then charged every one of those tokens at the plain input rate. The estimate came out low, by 25% on the affected tokens, on precisely the workloads that cache hardest.
It was found by widening the rate model by hand, not by anything watching. It could have been found in a week by a variance report, because it has exactly the signature described above: consistent, one-directional, concentrated on the days the cached workload ran. Instead it was correct-looking for as long as nobody looked.
We publish this for a specific reason. A vendor writing about cost accuracy is making an implicit claim that its own numbers are right. Ours were not, in a way we can describe and date, and the mechanism that would have caught it is the one this article is about. A cost tool that has never found itself wrong has either been lucky or has not checked.
If you are not going to build this
Most teams should not build a reconciliation pipeline. The monthly version takes half an hour and captures most of the value:
- Mint an admin key at each provider. It is a different credential class from the one your application uses — OpenAI's is created in organization settings, Anthropic's is an Admin API key (
sk-ant-admin01-) or another organization-level admin credential, and workspace keys are refused76. - Pull last month's daily costs from the cost endpoint into a spreadsheet.
- Put your own tool's daily totals beside them.
- Compute the difference per day and for the month, as a percentage of the provider's figure.
- Write down the monthly percentage. The single number matters less than having last month's to compare it to.
- Note anything the comparison cannot cover: Priority Tier, playground use, another team on the same account.
If the percentage is stable, you have learned that your estimate is biased by a known amount, which is worth more than an estimate you believe is exact. If it moves, something changed — a model, a tier, a workload, or a provider's price.
Two cautions. Do not reconcile the current day: a partial day written as a whole one makes the variance look enormous every morning and shrink through the afternoon. And re-read a few days back each time rather than only the newest — Anthropic notes data typically appears within five minutes but that delays can be longer7, and a late-settling figure should correct itself rather than stand.
How INFRO reconciles
INFRO estimates cost on every request, from the organization's own negotiated rate where one has been entered — for that provider, or for the model on any provider — otherwise the vendor's published list rate, and records in price_source which it used. Where a provider reports a request's cost in its own response, that figure is recorded instead and marked provider. A model with no rate is recorded as unpriced rather than as zero.
Separately, and only when an organization connects a read-only billing credential, INFRO reads the provider's own cost API once a day and stores the reported figure per provider per day beside its own estimate for the same UTC day; the current day is left out until it ends. A day counts as reconciled only when both figures are present; a day carrying one and not the other is recorded and is explicitly not a reconciliation, for the same reason an unpriced request is not a variance. The comparison is the product: totals, a per-day table, and a variance expressed as a share of the provider's number.
A gap that holds becomes something you can act on. When the estimate leans the same way by more than 5% on at least five of the last seven days, the organization gets a cost.drift alert. A gap that persists becomes a finding that names one likely cause — a negotiated rate nobody entered, a discount that changed, a vendor price change, a premium the rate table does not model, traffic reaching the same account without passing through INFRO, or unpriced requests — with a confidence and the figures it rests on. A negotiated-rate finding needs the ratio of reported to estimated cost to hold within 5% over at least ten days, and can suggest a rate per model once a model has at least seven days and $20 of reported spend. Applying a suggestion writes a rate for that model on that provider, from that day forward. Findings are readable as GET /v1/rate-findings.
Three properties of that design are deliberate and worth stating plainly.
The billing credential is separate from the inference credential, permanently. The key that serves your requests never reads your billing. The admin-scoped key that reads your cost is granted separately, stored encrypted, and never shown again.
Reconciliation never rewrites a request's recorded cost. Both cost APIs report daily and organization-wide. Attributing a daily total back to individual requests would invent a precision the provider did not give, so the per-request estimate stands as the estimate it is, a suggested rate applies only to requests served after it, and the variance is reported alongside.
A provider with no integration shows no variance rather than a variance of zero. Bedrock, Vertex AI and Azure keep their cost in the surrounding cloud's billing system. Zero would read as agreement, and there is nothing to agree with.
Reconciliation covers OpenAI and Anthropic and is included on the Business plan; below Business the console shows a preview of how much of each provider's estimate used the list rate and how many requests were unpriced. The availability of everything described here is listed below.
Why it sits in the gateway
What distinguishes this from the tools in the next section is not the comparison but where it runs: in the same gateway that routes each request, fails over between your connections, and refuses spend before the provider is called. A rate corrected from a finding is the rate the next budget check uses.
That matters because a budget can only be as right as the estimate it counts. In a test on INFRO's sandbox on 2026-09-25 (code at 1ad89a5, a test connection named echo, a $1 organization cap), 50, 200 and 500 concurrent requests — plain, streaming with client disconnects, with key, project and organization caps together, and with injected upstream failures — held in 12 of 12 runs: the most recorded was $0.9720 against $1, every request was either served and recorded or refused with 402 budget_exceeded, and no reservation was left behind; the test write-up has the scenarios and the method. Read it with its limits. The cap counts INFRO's estimate of provider cost, not the provider's invoice, so a provider that bills above the reserved worst case passes the cap by that excess — which is the gap this article is about. A model with no known price is refused while a blocking cap applies (402 unpriced_model_under_budget) unless the organization allows unpriced traffic. It was a sandbox test with a test connection, not production traffic. And admission serializes under load: p95 latency ran from 1.6 to 4.5 seconds, and streams at 500 concurrent requests had a 42-second p95.
Other tools that do this
INFRO is not the only product that compares an estimate against the bill, and a reader choosing one should know the others. Revenium shows an invoiced figure from the provider's billing data beside its own metered figure and describes them as reconciliation points8. Kenda says it reconciles spend against the actual invoice using read-only keys, names OpenAI, Anthropic, Gemini and Bedrock in its examples, and was running a design-partner programme ahead of general availability on 2026-10-139. LLM SpendGuard is an open-source tool for the OpenAI and Anthropic SDKs that compares a local ledger with the provider's own bill and enforces caps before submission10. What differs is where each sits: INFRO does it inside the gateway that also routes the traffic and enforces budgets, which suits a team that wants those in one place, and does not suit a team that wants none of its traffic to pass through a proxy.
The number worth having
The point of reconciling is not to make the estimate exact. It cannot be exact; it is a different kind of number from the invoice and it is more useful than the invoice for everything except paying it.
The point is to know the size and the direction of the gap, so that when someone asks whether the cost-per-customer figure can be trusted, the answer is a measured percentage and a date rather than a shrug. That percentage belongs in the spend report leadership reads, next to the estimate it qualifies. Any tool will tell you what your AI costs. Ask whichever one you use how far off it is, and how it knows.