Does an AI spend cap hold under 500 concurrent requests?

A spend cap is a promise about money that has not been spent yet. That makes it a concurrency problem before it is a finance problem. A cap that is exact for one request at a time can be passed by a wide margin when a hundred requests arrive in the same second, and most of the ways that happens are invisible in a test that sends requests one by one.

This article explains where caps leak, how a reservation-based cap closes those leaks, and what a test of INFRO's cap on its sandbox showed: the method, every number, the defects the test found first, and the things it does not prove.

Why spend caps leak under concurrency

Check, then act

The obvious implementation reads the spend so far, compares it with the cap, and serves the request if there is room. Each request is correct on its own. Two requests that read the total before either has recorded its cost both see the same room, and both are served.

Any limit that records cost after the response and enforces on the next request has this gap, and the platforms that work this way say so. OpenAI writes of its own hard spend limits that "enforcement is not instantaneous, so recorded spend can slightly exceed the configured amount"¹. Azure API Management's token limit policy notes that "concurrent or near-concurrent requests can temporarily exceed the configured token limit"². These are honest descriptions of a design choice, not defects: recording after the fact is exact about cost and cheap to run. It is a ceiling that can be crossed by whatever is in flight when it is reached.

The cost is unknown at admission

A chat request's price depends on how many tokens the model generates, which nobody knows when the request is admitted. A cap that wants to refuse before the provider call has to assume the worst: the whole billed request body at the most expensive input rate the model could apply, plus the most output the model could produce, on the most expensive model the request could fall back to. That worst case is what gets reserved, and it is usually several times the actual cost.

Requests that outlive their reservation

A reservation needs an expiry, or a crashed process leaves headroom booked forever. But an expiry creates three new leaks:

• Streams. A stream usually has an idle timeout but no total deadline. If the reservation expires while the stream is still producing tokens, the cap stops counting it.
• Asynchronous jobs. A video render may be reserved under the HTTP request's id but metered under the job's id, so settlement never releases the reservation, which then lapses while the job runs for hours.
• Delayed usage records. If a usage row cannot be written and goes to a retry queue, the reservation can expire before the row lands. For that window the request is counted as neither.

Retries, failover and unpriced models

A request that fails over to a second provider must be counted once, at the cost of the attempt that was billed. A request that fails upstream after admission must release its reservation, or the cap refuses traffic it has room for. And a model with no known price has a worst case of zero: unless something refuses it, a "hard cap" silently becomes a hard cap on priced models only.

The definition used

INFRO's test held its cap to four properties, written down before the runs. With a blocking cap C and N concurrent requests:

1. Final recorded spend is at most C plus, for each admitted request, any amount by which the actual cost exceeded the worst case reserved for it. In other words, at most C unless a provider bills more than the reserved worst case, and that excess is measured.
2. Every request is either served and recorded, or refused with 402 budget_exceeded.
3. No request is served without a usage row.
4. No reservation is left behind after the run.

The first property is deliberately narrower than "the invoice never exceeds the cap". A gateway can bound what it estimates; it cannot bound what a provider decides to bill.

How INFRO's blocking cap works

Budgets in INFRO can alert or block. An alert-only budget notifies and never refuses, and it is outside everything below. A blocking budget at the organization, project, member or key level, a key's lifetime limit, and an end-customer cap all go through the same admission path.

1. Price the worst case. The gateway prices the whole request body the provider bills, including tool definitions, schemas and every requested choice, at the most expensive input rate the model could apply, such as a prompt-cache write, on the most expensive model and provider the request could fall back to. The output side is the caller's max_tokens, else the catalog's maximum output, else 32,768 tokens.
2. Admit under a lock. One call to the database function budget_admit_v2 takes a transaction-scoped Postgres advisory lock per blocking ceiling, in sorted order so overlapping admissions cannot deadlock³, deletes expired reservations, reads live reservations held by other requests, then reads recorded spend, and admits only if the worst case fits every blocking ceiling. If it fits, it inserts this request's reservation before the lock is released. If it does not, the request is refused with 402 and a code naming the ceiling, such as organization_budget or key_budget.
3. Keep the hold while running. A reservation lives 600 seconds. A running request renews it every two minutes, so a long stream stays counted; a video job's reservation moves to the job id with a longer hold; a usage row handed to the retry outbox extends its reservation to 24 hours.
4. Settle, then release. When the request finishes, the usage row is inserted first and the reservation released second. For a moment the request is counted twice, which is the safe direction.

Two more rules close the gaps described above. Under a blocking budget, when a chat request sets no max_tokens, INFRO sends the reserved ceiling to the provider, so the response cannot run past what was set aside. And when a request could reach a model with no price, it is refused with 402 unpriced_model_under_budget unless the organization's owner has chosen to allow unpriced traffic, which blocking budgets then do not govern.

The admission function is the mechanism the test exercised. Since the test it has gained counters for end-customer caps (migration 0078) and for rotated keys (0079), on the same lock, read and insert structure.

The method

The work had two stages before anything ran on shared infrastructure.

A test suite of 31 tests ran the production admission and settlement functions in eight simulated isolates, each with its own Postgres pool and its own copy of the spend modules, so only the database coordinated them, as in production. It covered scenarios a load test cannot reach easily: a stream outliving the reservation's lifetime, a usage row stuck in the outbox, a process crashing after admission, a daily key cap across midnight UTC, and a provider billing three times its worst case.

A load harness then drove the real gateway Worker. For the sandbox stage it ran against sandbox-api.infro.io on 2026-09-25, serving commit 1ad89a5 on its Neon database. Each run created a fresh organization, key and a test connection named echo, which returns a fixed response priced from the catalog as anthropic/claude-opus-5; the worst case reserved per request was about $0.037 and the actual cost about $0.012. The cap was a $1 organization budget. After each run, recorded spend, usage rows and leftover reservations were read from the gateway's own database.

The scenarios at 50, 200 and 500 concurrent requests:

• basic: plain chat requests.
• stream: streaming requests, one in five disconnecting after the first chunk.
• multi: three keys under a $0.30 daily key cap on key A, a $0.50 project cap and the $1 organization cap.
• failures: one in three requests failing upstream after admission.

Results

Twelve of twelve runs held. Recorded spend never exceeded the $1 cap.

How to read it:

• Why $0.9720 and not $1.00. With $0.028 of room left, no request could be admitted, because each reserved about $0.037. A reservation-based cap refuses the last request that might not fit, even when its actual cost would have.
• Undelivered requests received no response from INFRO at all, from client or edge socket limits when one machine opens 500 connections. They left no row and were never admitted.
• failures at 200. The injected failures got the echo connection ejected by health checks, so later requests were refused with 503 connection_out_of_scope before admission. That is the health ejection working, not a cap effect.
• Every delivered request was served and recorded, refused before the provider call (with 402, or the 503 above), or failed upstream with its own 502 row. No reservation was left behind in any run.
• Latency. p95 ran 1.6 to 4.5 seconds, except streams at 500, which had a 42-second p95: they queued behind the admission lock.

Defects found and fixed first

The local stage found five defects, all fixed before the sandbox runs, each with a test that fails when the fix is reverted.

D4 has a stated residual: when the database is down, the extension cannot be written, but no admission can run either; after recovery the gap lasts until the next outbox drain.

Guards proven by removing them

A passing test only matters if it fails without the thing it tests. Each guard was removed on purpose and the suite run again.

Removing the advisory lock produced $1.10, $1.04 and $1.01 recorded on a $1 organization cap: the size of a check-then-act race in this codebase under the suite's load, a few percent rather than a multiple. Removing the reservation read broke eight tests at once, which is the clearest evidence that reserving the worst case, not the lock alone, is what holds the line.

What this does not show

• It bounds an estimate, not an invoice. The cap counts INFRO's estimate of what your provider will bill, from your negotiated rate where you have entered one, else the vendor's list rate. A provider that bills above the reserved worst case, by ignoring max_tokens, adding a surcharge INFRO cannot see, or counting tokens differently, passes the cap by exactly that excess. The test suite measured this directly: with every fourth request billed at three times its worst case, $1.50 was recorded on a $1 cap at 200 concurrent requests, an overshoot equal to the excess billed.
• Unpriced models are refused, not counted. While a blocking cap applies, a model with no known price is refused with 402 unpriced_model_under_budget unless unpriced traffic is allowed. If it is allowed, that traffic is not governed by the cap.
• It was a sandbox with a test connection. The echo connection returns a fixed response; no real provider was called and no production traffic was involved.
• Admission serializes. Every blocking admission re-sums the period under the ceiling's lock. That never admits past the cap, but it bounds how fast a busy organization under a blocking cap can be served, as the latency figures show.
• A cancelled job may still be billed upstream. INFRO releases the reservation when a render is cancelled; a provider with no cancel operation may bill anyway.
• No other product was tested. Nothing here says how any other gateway behaves under the same load.

How to test a spend cap yourself

The method transfers to any gateway, including one you build. A cap that has not been tested under concurrency should be assumed to leak.

• Use a priced test model with a fixed response, so actual cost is known exactly per request.
• Set a small cap, such as $1, so a few hundred requests can cross it.
• Fire 50, 200 and 500 requests at once from several connections, not a loop.
• Include streams that disconnect after the first chunk, and requests that fail upstream after admission.
• Stack caps at two or three levels and check each one.
• Afterwards, compare recorded spend with the cap, count usage rows against served requests, and look for reservations or holds still open.
• Hold a stream open longer than the reservation's expiry, and see whether it is still counted.
• Remove your lock, or your reservation read, and confirm the test fails.

The last step is the one most often skipped, and the one that tells you whether the rest measured anything.