# Does an AI spend cap hold under 500 concurrent requests?

URL: https://infro.io/blog/ai-spend-cap-concurrency-test
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-10-13
Last modified: 2026-10-13
Last reviewed: 2026-10-13 (next review due 2026-11-12)
Topic: Cost and FinOps
Format: Methodology

> In a test on INFRO's sandbox on 2026-09-25, a $1 blocking organization cap held in 12 of 12 runs at 50, 200 and 500 concurrent requests: the most recorded was $0.9720, every request was served and recorded or refused with 402 budget_exceeded, and no reservation was left behind. The cap counts INFRO's estimate of provider cost, not the provider's invoice, and refuses models with no known price unless unpriced traffic is allowed; it was a sandbox test with a test connection, not production traffic, and admission serializes under load.

Key takeaways:
- A cap that reads a running total and then serves the request is passed by any burst that reads the same total; the fix is to reserve each request's worst case under a lock before the provider call.
- Streams with no total deadline, async jobs metered under another id and usage rows delayed in an outbox each let a reservation lapse while money is still being spent.
- In INFRO's sandbox test, a $1 organization cap held in 12 of 12 runs at 50, 200 and 500 concurrent requests; the most recorded was $0.9720.
- The cap counts INFRO's estimate of provider cost. A provider that bills above the reserved worst case passes the cap by exactly that excess.
- Serializing admission costs latency: p95 ran 1.6 to 4.5 seconds in the test, and 42 seconds for streams at 500.

A spend cap is a promise about money that has not been spent yet. That makes it a concurrency problem before it is a finance problem. A cap that is exact for one request at a time can be passed by a wide margin when a hundred requests arrive in the same second, and most of the ways that happens are invisible in a test that sends requests one by one.

This article explains where caps leak, how a reservation-based cap closes those leaks, and what a test of INFRO's cap on its sandbox showed: the method, every number, the defects the test found first, and the things it does not prove.

## Why spend caps leak under concurrency

### Check, then act

The obvious implementation reads the spend so far, compares it with the cap, and serves the request if there is room. Each request is correct on its own. Two requests that read the total before either has recorded its cost both see the same room, and both are served.

[Diagram: Two requests read the same total before either records its cost. Each sees room; together they pass the cap.]

Any limit that records cost after the response and enforces on the next request has this gap, and the platforms that work this way say so. OpenAI writes of its own hard spend limits that "enforcement is not instantaneous, so recorded spend can slightly exceed the configured amount"[1]. Azure API Management's token limit policy notes that "concurrent or near-concurrent requests can temporarily exceed the configured token limit"[2]. These are honest descriptions of a design choice, not defects: recording after the fact is exact about cost and cheap to run. It is a ceiling that can be crossed by whatever is in flight when it is reached.

### The cost is unknown at admission

A chat request's price depends on how many tokens the model generates, which nobody knows when the request is admitted. A cap that wants to refuse before the provider call has to assume the worst: the whole billed request body at the most expensive input rate the model could apply, plus the most output the model could produce, on the most expensive model the request could fall back to. That worst case is what gets reserved, and it is usually several times the actual cost.

### Requests that outlive their reservation

A reservation needs an expiry, or a crashed process leaves headroom booked forever. But an expiry creates three new leaks:

- **Streams.** A stream usually has an idle timeout but no total deadline. If the reservation expires while the stream is still producing tokens, the cap stops counting it.
- **Asynchronous jobs.** A video render may be reserved under the HTTP request's id but metered under the job's id, so settlement never releases the reservation, which then lapses while the job runs for hours.
- **Delayed usage records.** If a usage row cannot be written and goes to a retry queue, the reservation can expire before the row lands. For that window the request is counted as neither.

### Retries, failover and unpriced models

A request that fails over to a second provider must be counted once, at the cost of the attempt that was billed. A request that fails upstream after admission must release its reservation, or the cap refuses traffic it has room for. And a model with no known price has a worst case of zero: unless something refuses it, a "hard cap" silently becomes a hard cap on priced models only.

## The definition used

INFRO's test held its cap to four properties, written down before the runs. With a blocking cap C and N concurrent requests:

1. Final recorded spend is at most C plus, for each admitted request, any amount by which the actual cost exceeded the worst case reserved for it. In other words, at most C unless a provider bills more than the reserved worst case, and that excess is measured.
2. Every request is either served and recorded, or refused with `402 budget_exceeded`.
3. No request is served without a usage row.
4. No reservation is left behind after the run.

The first property is deliberately narrower than "the invoice never exceeds the cap". A gateway can bound what it estimates; it cannot bound what a provider decides to bill.

## How INFRO's blocking cap works

Budgets in INFRO can alert or block. An alert-only budget notifies and never refuses, and it is outside everything below. A blocking budget at the organization, project, member or key level, a key's lifetime limit, and an end-customer cap all go through the same admission path.

[Diagram: The coordinated path for a request under any blocking ceiling, as implemented in `limits/spend.ts` and `metering/record.ts`.]

1. **Price the worst case.** The gateway prices the whole request body the provider bills, including tool definitions, schemas and every requested choice, at the most expensive input rate the model could apply, such as a prompt-cache write, on the most expensive model and provider the request could fall back to. The output side is the caller's `max_tokens`, else the catalog's maximum output, else 32,768 tokens.
2. **Admit under a lock.** One call to the database function `budget_admit_v2` takes a transaction-scoped Postgres advisory lock per blocking ceiling, in sorted order so overlapping admissions cannot deadlock[3], deletes expired reservations, reads live reservations held by other requests, then reads recorded spend, and admits only if the worst case fits every blocking ceiling. If it fits, it inserts this request's reservation before the lock is released. If it does not, the request is refused with `402` and a code naming the ceiling, such as `organization_budget` or `key_budget`.
3. **Keep the hold while running.** A reservation lives 600 seconds. A running request renews it every two minutes, so a long stream stays counted; a video job's reservation moves to the job id with a longer hold; a usage row handed to the retry outbox extends its reservation to 24 hours.
4. **Settle, then release.** When the request finishes, the usage row is inserted first and the reservation released second. For a moment the request is counted twice, which is the safe direction.

Two more rules close the gaps described above. Under a blocking budget, when a chat request sets no `max_tokens`, INFRO sends the reserved ceiling to the provider, so the response cannot run past what was set aside. And when a request could reach a model with no price, it is refused with `402 unpriced_model_under_budget` unless the organization's owner has chosen to allow unpriced traffic, which blocking budgets then do not govern.

The admission function is the mechanism the test exercised. Since the test it has gained counters for end-customer caps (migration 0078) and for rotated keys (0079), on the same lock, read and insert structure.

## The method

The work had two stages before anything ran on shared infrastructure.

**A test suite** of 31 tests ran the production admission and settlement functions in eight simulated isolates, each with its own Postgres pool and its own copy of the spend modules, so only the database coordinated them, as in production. It covered scenarios a load test cannot reach easily: a stream outliving the reservation's lifetime, a usage row stuck in the outbox, a process crashing after admission, a daily key cap across midnight UTC, and a provider billing three times its worst case.

**A load harness** then drove the real gateway Worker. For the sandbox stage it ran against `sandbox-api.infro.io` on 2026-09-25, serving commit `1ad89a5` on its Neon database. Each run created a fresh organization, key and a test connection named `echo`, which returns a fixed response priced from the catalog as `anthropic/claude-opus-5`; the worst case reserved per request was about $0.037 and the actual cost about $0.012. The cap was a $1 organization budget. After each run, recorded spend, usage rows and leftover reservations were read from the gateway's own database.

The scenarios at 50, 200 and 500 concurrent requests:

- **basic**: plain chat requests.
- **stream**: streaming requests, one in five disconnecting after the first chunk.
- **multi**: three keys under a $0.30 daily key cap on key A, a $0.50 project cap and the $1 organization cap.
- **failures**: one in three requests failing upstream after admission.

## Results

Twelve of twelve runs held. Recorded spend never exceeded the $1 cap.

| Scenario | N | Undelivered | Served | Refused | Other | Recorded spend | Leftover reservations |
| --- | --- | --- | --- | --- | --- | --- | --- |
| basic | 50 | 0 | 50 | 0 | — | $0.6000 | 0 |
| basic | 200 | 0 | 81 | 119 | — | $0.9720 | 0 |
| basic | 500 | 53 | 81 | 366 | — | $0.9720 | 0 |
| stream (1 in 5 disconnect) | 50 | 0 | 40 (+10) | 0 | — | $0.6000 | 0 |
| stream | 200 | 0 | 68 (+13) | 119 | — | $0.9720 | 0 |
| stream | 500 | 1 | 66 (+15) | 418 | — | $0.9720 | 0 |
| multi (key A $0.30 daily, project $0.50, org $1) | 50 | 0 | 50 | 0 | — | $0.6000 (A $0.204, P $0.408) | 0 |
| multi | 200 | 0 | 79 | 72 org + 49 project | — | $0.9480 (A $0.048, P $0.468) | 0 |
| multi | 500 | 6 | 81 | 361 org + 51 project + 1 key | — | $0.9720 (A $0.252, P $0.468) | 0 |
| failures (1 in 3 fail) | 50 | 0 | 33 | 0 | 17 × 502 | $0.3960 | 0 |
| failures | 200 | 0 | 57 | 0 | 27 × 502, 116 × 503 `connection_out_of_scope` | $0.6840 | 0 |
| failures | 500 | 0 | 81 | 379 | 40 × 502 | $0.9720 | 0 |

Table: INFRO sandbox, 2026-09-25, code at 1ad89a5, `echo` test connection, $1 organization cap. "(+N)" counts streams that disconnected and were still recorded. Refusals are `402 budget_exceeded`.

[Diagram: Recorded spend per run against the $1 cap. Runs that stop at $0.9720 stop because the $0.028 left is less than one request's worst case.]

How to read it:

- **Why $0.9720 and not $1.00.** With $0.028 of room left, no request could be admitted, because each reserved about $0.037. A reservation-based cap refuses the last request that might not fit, even when its actual cost would have.
- **Undelivered** requests received no response from INFRO at all, from client or edge socket limits when one machine opens 500 connections. They left no row and were never admitted.
- **failures at 200.** The injected failures got the echo connection ejected by health checks, so later requests were refused with `503 connection_out_of_scope` before admission. That is the health ejection working, not a cap effect.
- **Every delivered request** was served and recorded, refused before the provider call (with `402`, or the `503` above), or failed upstream with its own `502` row. No reservation was left behind in any run.
- **Latency.** p95 ran 1.6 to 4.5 seconds, except streams at 500, which had a 42-second p95: they queued behind the admission lock.

## Defects found and fixed first

The local stage found five defects, all fixed before the sandbox runs, each with a test that fails when the fix is reverted.

|  | Defect | Fix |
| --- | --- | --- |
| D1 | The admission function read recorded spend, then reservations, in two snapshots. A request settling between the two reads was counted as neither; reproduced 30 of 30 times on a large history. | Migration 0070 reads reservations first, so a missed release means the usage row was already visible. |
| D2 | A video reserved under the request id but metered under the job id, so its reservation lapsed after ten minutes of a render that can run for hours. | The reservation moves to the job id with a longer hold, released on metering or cancel. |
| D3 | A stream longer than 600 seconds stopped being counted while still spending: $1.80 recorded on a $1 cap in the test. | Every coordinated reservation is renewed every two minutes while the request runs. |
| D4 | A usage row held in the outbox longer than 600 seconds was counted as neither. | Hand-off to the outbox extends the reservation to 24 hours. |
| D5 | With no `max_tokens`, OpenAI-compatible and Responses adapters sent no output ceiling, so a response could run past its reservation. | Under a blocking budget, the gateway sends the reserved ceiling when the caller sets none. |

Table: Summarised from the internal report, section 5.

D4 has a stated residual: when the database is down, the extension cannot be written, but no admission can run either; after recovery the gap lasts until the next outbox drain.

## Guards proven by removing them

A passing test only matters if it fails without the thing it tests. Each guard was removed on purpose and the suite run again.

[Diagram: Each row is a guard removed from the code and what the concurrency suite then recorded.]

Removing the advisory lock produced $1.10, $1.04 and $1.01 recorded on a $1 organization cap: the size of a check-then-act race in this codebase under the suite's load, a few percent rather than a multiple. Removing the reservation read broke eight tests at once, which is the clearest evidence that reserving the worst case, not the lock alone, is what holds the line.

## What this does not show

[Diagram: The claim, and its limits, in one place.]

- **It bounds an estimate, not an invoice.** The cap counts INFRO's estimate of what your provider will bill, from your negotiated rate where you have entered one, else the vendor's list rate. A provider that bills above the reserved worst case, by ignoring `max_tokens`, adding a surcharge INFRO cannot see, or counting tokens differently, passes the cap by exactly that excess. The test suite measured this directly: with every fourth request billed at three times its worst case, $1.50 was recorded on a $1 cap at 200 concurrent requests, an overshoot equal to the excess billed.
- **Unpriced models are refused, not counted.** While a blocking cap applies, a model with no known price is refused with `402 unpriced_model_under_budget` unless unpriced traffic is allowed. If it is allowed, that traffic is not governed by the cap.
- **It was a sandbox with a test connection.** The `echo` connection returns a fixed response; no real provider was called and no production traffic was involved.
- **Admission serializes.** Every blocking admission re-sums the period under the ceiling's lock. That never admits past the cap, but it bounds how fast a busy organization under a blocking cap can be served, as the latency figures show.
- **A cancelled job may still be billed upstream.** INFRO releases the reservation when a render is cancelled; a provider with no cancel operation may bill anyway.
- **No other product was tested.** Nothing here says how any other gateway behaves under the same load.

## How to test a spend cap yourself

The method transfers to any gateway, including one you build. A cap that has not been tested under concurrency should be assumed to leak.

- [ ] Use a priced test model with a fixed response, so actual cost is known exactly per request.
- [ ] Set a small cap, such as $1, so a few hundred requests can cross it.
- [ ] Fire 50, 200 and 500 requests at once from several connections, not a loop.
- [ ] Include streams that disconnect after the first chunk, and requests that fail upstream after admission.
- [ ] Stack caps at two or three levels and check each one.
- [ ] Afterwards, compare recorded spend with the cap, count usage rows against served requests, and look for reservations or holds still open.
- [ ] Hold a stream open longer than the reservation's expiry, and see whether it is still counted.
- [ ] Remove your lock, or your reservation read, and confirm the test fails.

The last step is the one most often skipped, and the one that tells you whether the rest measured anything.

## Sources

1. Spend limits — OpenAI. https://developers.openai.com/api/docs/guides/spend-limits (accessed 2026-10-13)
2. llm-token-limit policy — Microsoft. https://learn.microsoft.com/en-us/azure/api-management/llm-token-limit-policy (accessed 2026-10-13)
3. Explicit locking: advisory locks — PostgreSQL Global Development Group. https://www.postgresql.org/docs/current/explicit-locking.html (accessed 2026-10-13)