# Provider quota is short: which AI requests should survive?

URL: https://infro.io/blog/provider-quota-is-short-which-ai-requests-should-survive
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-11-20
Last modified: 2026-11-20
Last reviewed: 2026-11-20 (next review due 2026-12-20)
Topic: Routing and reliability
Format: Guide

> Give every request a lane (interactive, background or batch) at the edge, reserve quota per lane, and shed in a fixed order: batch first, then background, then degrade interactive, and reject last. Cap queues by deadline rather than by size, and keep standby capacity for the interactive lane only.

Key takeaways:
- Tag every request with a lane at the edge, because a gateway cannot tell a chat turn from a nightly backfill after the fact.
- Reserve quota per lane in the provider's own units (tokens and requests per minute), and let lower lanes borrow only what interactive is not using.
- Shed in a fixed order, batch then background then interactive degradation, and set queue limits from the deadline, not from memory size.
- A failover target with a smaller quota turns a failover into a second outage unless background lanes are kept off it.
- Shared quota pools mean a new region or deployment in the same subscription may add no capacity, so check the pool before counting it as a fallback.

## Why quota shortage is a priority problem, not a retry problem

A 429 storm has a simple cause: demand for tokens or requests per minute exceeds what your provider account will admit. RFC 6585 defines 429 as the user having sent too many requests in a given amount of time, and says only that the response MAY carry a Retry-After header. It does not say how the server identifies the user or counts requests.[1] In practice that means the limit you hit may be shared with every other workload on the same account, and the response will not tell you which of your callers consumed it.

When that happens, someone's request gets served and someone's does not. If you have not decided who, the decision is made by arrival order: whichever caller retries fastest wins. Background agents and backfills usually retry fastest and hold the most requests in flight, so they win. The person waiting on a chat answer loses.

Circuit breakers and per-key rate limits do not fix this. A breaker answers "is the provider healthy?" and a throttled provider is healthy but full. A rate limit caps one caller but says nothing about relative importance. What is missing is an explicit rule for which work survives when capacity is short. This guide builds that rule in four parts: lanes, quota shares, a shed ladder and queue limits. Then it covers the case that breaks most designs, a failover target with less quota than the primary.

## The three lanes

Assign every request to exactly one lane at the edge, before it reaches the gateway. Do not infer the lane from the model name or prompt length; a gateway sees a payload, not intent.

| Lane | Who is waiting | Deadline | On shortage |
| --- | --- | --- | --- |
| Interactive | A person, with a stream open or a spinner showing | Seconds | Degrade, then reject with a clear message |
| Background | A job, agent step or webhook handler | Minutes | Queue with a hard age limit, then defer |
| Batch | Nobody; backfills, evals, nightly summaries | Hours | Pause entirely; resume when quota recovers |

Table: Lane definitions. The deadline is what the caller can tolerate, not what the system usually achieves.

Three rules keep the model honest:

- **The lane is a property of the caller's intent.** A support agent's final answer is interactive; the same agent's overnight re-indexing is batch, even if both use the same model and prompt template.
- **Lanes do not promote themselves.** A background job that fails and retries stays in background. Retries must not climb into a higher lane.
- **Work that can wait an hour belongs on a path built for waiting.** If a provider offers a batch interface or a lower-rate background window, send batch-lane work there and take it off the synchronous quota altogether; see [batch or synchronous](https://infro.io/blog/batch-vs-synchronous-llm) for how to decide which workloads move.

Tag the lane on every request. INFRO routes can match a key in the request's `metadata` object, and the same attribution values arrive from metadata or `X-INFRO-*` headers.[2] Using a `lane` key means the same value drives routing, appears on the trace and can be filtered later.

```python
from openai import OpenAI

client = OpenAI(base_url="https://api.infro.io/v1", api_key=INFRO_KEY)

resp = client.chat.completions.create(
    model="chat-interactive",          # a route name, see the lane routes below
    messages=[{"role": "user", "content": "Where is my order?"}],
    metadata={"lane": "interactive", "feature": "support-chat"},
    stream=True,
)
```

## Reserve quota per lane, in the provider's units

Provider limits are not one number. Azure OpenAI's quota page lists both requests per minute and tokens per minute per model and deployment type, and for some model versions the ratio between them differs: the page says versions of `gpt-chat-latest` from 2026-05-05 to 2026-06-24 use 10 RPM per 1,000 TPM, while 2026-08-06 and 2026-10-07 use 1 RPM per 1,000 TPM.[3] On the newer ratio, a lane of short interactive requests can exhaust the request limit long before it uses its tokens. Your limiter has to track both, and admit a request only if both buckets can pay for it.

Split the quota into three kinds of share:

1. **Reserved share.** Capacity only that lane may use. Interactive gets the largest reserve, because it is the lane that cannot wait.
2. **Borrowable share.** Capacity a lane may use only when higher lanes are not using it. Batch borrows; it never reserves.
3. **Headroom.** A slice nobody uses in steady state, so a burst of interactive traffic does not have to preempt work already running.

> **Example: Illustrative split (made-up numbers)**
> Suppose your effective limit on a model is 600,000 tokens per minute (TPM) and 600 requests per minute (RPM). One reasonable starting split: interactive reserves 50% of both, background reserves 20%, batch reserves 0% and borrows up to 30%, and 10% is headroom. Batch can use the other lanes' idle capacity, and is the first thing cut when they need it back. These figures are not from any provider; derive yours from your real quota and traffic mix.

Admission uses a token estimate, not a measurement. INFRO exposes its own upper-bound estimate of prompt size as `input_tokens` for route conditions, alongside the request's `max_tokens`.[2] Reserve prompt estimate plus `max_tokens` at admission, then return the difference when the response reports real usage. Without a `max_tokens` cap on background and batch calls, one long completion can consume a lane's whole minute.

```python
def admit(lane, est_tokens, now):
    b = buckets[lane]                    # token + request buckets for this lane
    if b.try_take(tokens=est_tokens, requests=1, now=now):
        return "ok"
    # Borrow only if every higher lane is idle enough and shed level allows it.
    if lane != "interactive" and shed_level < BORROW_CUTOFF[lane]:
        if borrow_pool.try_take(tokens=est_tokens, requests=1, now=now,
                                leave_for=higher_lanes(lane)):
            return "ok"
    return "queue" if queue_has_room(lane) else "shed"
```

## A shed ladder with fixed order

Do not decide at 3 a.m. what to drop. Define levels, the signal that moves between them, and what each level cuts. Drive the level from signals you can read: the rate of provider 429s over a short window, queue wait at the 95th percentile, and any Retry-After value.

| Level | Trigger (examples) | Batch | Background | Interactive |
| --- | --- | --- | --- | --- |
| 0 Normal | 429 rate near zero | Borrow idle capacity | Reserved share | Reserved share |
| 1 Pressure | Sustained provider 429s, or queue wait over half the deadline | Paused, held in its queue | Reserved share only, no borrowing | Unchanged |
| 2 Shortage | 429s continue after level 1 | Paused | New work queued; oldest past its age limit dropped | Lower `max_tokens`, route to a smaller model |
| 3 Emergency | Interactive wait breaches its deadline | Paused | Deferred; callers told to retry later | Admit only up to the reserved share; reject the rest with a clear error |

Table: Shed ladder. Thresholds are yours to set from your own baselines.

Two rules make the ladder safe:

- **Move down fast, move up slowly.** Raise the shed level on the first confirmed signal. Lower it only after a quiet window several times longer than the one that raised it. Otherwise the level flaps and the lanes oscillate.
- **Degradation comes before rejection for interactive.** A shorter answer from a smaller model is usually better for the user than an error. Test that the smaller model is acceptable first; see [fallback equivalence](https://infro.io/blog/is-your-fallback-model-good-enough-test-fallback-equivalence).

Tell the caller what happened. INFRO's error model separates a limit INFRO enforces (`rate_limit_exceeded`) from a 429 your own provider returned, which carries the code `provider_rate_limited`.[4] Your own shed responses should follow the same idea: a distinct code naming the lane and reason, so a client does not read "the platform is shedding background work" as "I personally sent too much". The RFC also notes that answering every request in an overload with a 429 itself consumes resources, and that dropping connections may be more appropriate under attack.[1] At level 3, reject cheaply and early, before parsing large bodies.

## Queue limits come from deadlines

A queue converts overload into latency. That is useful only while the wait is shorter than the caller's deadline. Past that, a queued request is wasted work: it still consumes quota when it finally runs, and nobody is waiting for the answer.

Size each queue from the deadline and the drain rate, not from a round number:

```text
max_depth = admitted_rate_per_second × (deadline_seconds − p50_service_seconds)
max_age   = deadline_seconds − p50_service_seconds
```

> **Example: Worked queue limit (illustrative numbers)**
> An interactive lane is admitted at 200 requests per minute, about 3.3 per second. The deadline is 8 seconds and median service time is 4 seconds. Then `max_age` is 4 seconds and `max_depth` is about 13 requests. The 14th arrival cannot be served inside its deadline, so it is rejected immediately instead of waiting for a timeout. A background lane at the same rate with a 5-minute deadline would allow a depth in the hundreds, which is fine only if its callers genuinely tolerate it.

Rules for queues:

- **Enforce age as well as depth.** Check the age at dequeue time and drop expired entries without calling the provider.
- **Under overload, serve the newest interactive request first.** The oldest is the most likely to have been abandoned by the user. Keep first-in-first-out order for background, where fairness matters more than recency.
- **Never let a lower lane queue ahead of a higher one.** Dequeue by lane priority, then by order within the lane.
- **Bound in-flight work too.** Queued requests are not the only risk; streams already open hold quota for their whole duration. A per-key limit on concurrent requests, streams included, keeps a single lane from occupying every slot; INFRO offers a concurrency limit per gateway key.

## Retries are where storms come from

Every retry is new demand on the quota that just refused you. Three settings matter:

- **Background and batch retries must not use the interactive reserve.** Charge retries to the original lane's bucket.
- **Keep gateway retries off lower lanes.** INFRO routes can retry the same target 0–3 times, with no backoff between retries.[5] On a background or batch route, set retries to 0 and put the backoff in your queue worker, where you can honor Retry-After and add jitter.
- **Use failure rules by kind.** A route's failure rules decide per failure kind (rate limit, server error, timeout, connection, rejected key) whether to retry, move to the next target at once, or stop.[5] For the batch lane, a rate-limit failure should stop. For interactive, it should move on.

A related caution: INFRO documents `X-RateLimit-Remaining` as a best-effort upper bound, because rate limiting is enforced at the edge location serving the request rather than from one global counter. It says to back off as the value nears zero but not to schedule against it.[6] Treat any remaining-quota header the same way, as a hint for the shed level, not as admission control.

## When failover lands on less quota

A degraded failover is where lanes pay for themselves. Failover itself is straightforward: INFRO moves a request to the next eligible connection when the first fails before the first token, including on a provider 429, and a provider 429 does not count against the connection's health.[7] The trouble is what the next connection can absorb.

Three failure modes recur:

1. **The standby is smaller.** Azure's quota tiers differ by an order of magnitude. In the Tier 1 table, `gpt-6-astra` on Global Standard lists 250 RPM and 250,000 TPM, and 100 RPM and 100,000 TPM on Data Zone Standard; the Tier 2 row for the same model lists 1,000 RPM.[3] If the primary is a Tier 2 subscription and the standby is a Tier 1, a failover sends full traffic into a quarter of the capacity.
2. **The pool is shared.** For Global Standard, deployments of the same model and version share one quota pool across all regions in a subscription, and Data Zone Standard shares one per data zone.[3] A second region in the same subscription is not extra capacity for that model.
3. **Every lane fails over together.** Without lane-aware routing, batch work lands on the standby along with everything else and consumes the capacity interactive needed.

The fix is to give lanes different target lists, so standby capacity is reserved for the lane that needs it. INFRO named routes accept priority or weighted targets over your own connections, and route rules can match on `metadata`.[2] A layout:

| Route | Targets | Retries | Notes |
| --- | --- | --- | --- |
| `chat-interactive` | Primary, then standby | 1 | A smaller-model rule at shed level 2 |
| `chat-background` | Primary only | 0 | Never touches the standby |
| `chat-batch` | Primary only, or a batch interface | 0 | Paused from your side at level 1 |

Table: Lane routes. Route names are examples.

Note the limits of this. A request that names a model directly is tried across every eligible connection for that vendor, so lane routing only works if each lane calls its own route name.[7] Also, INFRO does not add capacity: it states that your own provider accounts keep their model, token and request quotas and that it adds no shared pool of purchased capacity.[6] Routes decide where requests go. The queue, the quota shares and the shed ladder in this guide live in your application or worker layer, and a per-key request ceiling is a coarse lane control, not a reservation. It helps only when the sum of lane ceilings stays below the provider's quota with interactive headroom left over.

Provider-side overflow features can be lane-aware too. Azure's spillover sends overage from a provisioned deployment to a standard deployment, and it can be switched on per request with the `x-ms-spillover-deployment` header. If the header is not set, spillover does not start. Spillover requests are billed at the standard deployment's token rates.[8] Setting the header only on the interactive lane keeps background traffic from spilling onto pay-as-you-go capacity.

## See which lane was shed

You cannot tune what you cannot see. Record, per lane: admitted, queued, shed and deferred counts, queue wait at p50 and p95, provider 429 rate, shed level over time, and the failover and spillover rate. Group by the `lane` tag you set at the edge.

INFRO records model, latency, token counts and estimated provider cost on every request, and traces carry the route and connection that served each one; spans can be sent to your own OpenTelemetry collector. OpenTelemetry's GenAI semantic conventions now live in their own repository, so check there for current attribute names before you standardize spans, and add `lane` and `shed.level` as your own attributes.[9] Alert on the interactive lane's queue wait breaching half its deadline, not on the raw 429 count, which mostly measures background noise.

Rehearse it. Cut your effective limit in a staging environment and confirm that level 1 pauses batch within one window, that interactive wait stays under its deadline, and that nothing retried its way into a higher lane. [Failover drills](https://infro.io/blog/llm-failover-drills) covers the provider-failure side of the same exercise.

## Checklist before the next storm

- [ ] Every caller sets a lane; requests without one default to background, not interactive.
- [ ] Reserved and borrowable shares are written down per model, in both RPM and TPM.
- [ ] `max_tokens` is capped on background and batch calls.
- [ ] Each queue has a maximum age and depth derived from its deadline.
- [ ] Retries are charged to the original lane; background and batch routes retry 0 times at the gateway.
- [ ] Standby connections appear only in the interactive route.
- [ ] The standby's quota was checked against interactive peak, and any shared pool counted once.
- [ ] The shed ladder has named levels, triggers and a slow way back up.
- [ ] Shed responses carry a distinct code, lane and reason.
- [ ] A drill has cut the quota and the dashboard showed the right lane shed.

## Questions

### Should shed requests get a 429 or a 503?

Use a status your clients do not already treat as their own fault, and add a machine-readable code that names the lane and the reason. RFC 6585 defines 429 as the user having sent too many requests in a given amount of time and allows a Retry-After header[1]; a shed caused by upstream capacity is a different condition, so give it a distinct error code even if you keep the status.

### Does a per-key rate limit enforce lane priority?

No. A key limit caps what one key can send; it does not reserve provider capacity for another key. It becomes a lane control only when the lane ceilings together stay below the provider quota with interactive headroom left over.

## Sources

1. RFC 6585: Additional HTTP Status Codes — IETF / RFC Editor. https://www.rfc-editor.org/rfc/rfc6585 (accessed 2026-11-20)
2. Routes — INFRO Docs. https://infro.io/docs/routes (accessed 2026-11-20)
3. Azure OpenAI in Microsoft Foundry Models quotas and limits — Microsoft Learn. https://learn.microsoft.com/en-us/azure/ai-foundry/openai/quotas-limits (accessed 2026-11-20)
4. Errors — INFRO Docs. https://infro.io/docs/api/errors (accessed 2026-11-20)
5. Smart routing — INFRO Docs. https://infro.io/docs/features/routing (accessed 2026-11-20)
6. Rate & spend limits — INFRO Docs. https://infro.io/docs/api/limits (accessed 2026-11-20)
7. Failover & fallbacks — INFRO Docs. https://infro.io/docs/features/fallbacks (accessed 2026-11-20)
8. Manage traffic with spillover for provisioned deployments — Microsoft Learn. https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/spillover-traffic-management (accessed 2026-11-20)
9. Generative AI semantic conventions (moved) — OpenTelemetry. https://opentelemetry.io/docs/specs/semconv/gen-ai/ (accessed 2026-11-20)
