Skip to content

Routing and reliabilityGuide

Provider quota is short: which AI requests should survive?

When provider quota runs out, the gateway decides who gets the remaining capacity, whether or not you designed that decision. This guide gives a three-lane model, a shed ladder and queue limits, and shows how to build them on top of routes and per-lane keys.

By INFRO Editorial

Published

11 min read

Luminous routing paths connecting one application node to many AI model endpoints

In short

Give every request a lane (interactive, background or batch) at the edge, reserve quota per lane, and shed in a fixed order: batch first, then background, then degrade interactive, and reject last. Cap queues by deadline rather than by size, and keep standby capacity for the interactive lane only.

Key takeaways

  • Tag every request with a lane at the edge, because a gateway cannot tell a chat turn from a nightly backfill after the fact.
  • Reserve quota per lane in the provider's own units (tokens and requests per minute), and let lower lanes borrow only what interactive is not using.
  • Shed in a fixed order, batch then background then interactive degradation, and set queue limits from the deadline, not from memory size.
  • A failover target with a smaller quota turns a failover into a second outage unless background lanes are kept off it.
  • Shared quota pools mean a new region or deployment in the same subscription may add no capacity, so check the pool before counting it as a fallback.

Why quota shortage is a priority problem, not a retry problem

A 429 storm has a simple cause: demand for tokens or requests per minute exceeds what your provider account will admit. RFC 6585 defines 429 as the user having sent too many requests in a given amount of time, and says only that the response MAY carry a Retry-After header. It does not say how the server identifies the user or counts requests.1 In practice that means the limit you hit may be shared with every other workload on the same account, and the response will not tell you which of your callers consumed it.

When that happens, someone's request gets served and someone's does not. If you have not decided who, the decision is made by arrival order: whichever caller retries fastest wins. Background agents and backfills usually retry fastest and hold the most requests in flight, so they win. The person waiting on a chat answer loses.

Circuit breakers and per-key rate limits do not fix this. A breaker answers "is the provider healthy?" and a throttled provider is healthy but full. A rate limit caps one caller but says nothing about relative importance. What is missing is an explicit rule for which work survives when capacity is short. This guide builds that rule in four parts: lanes, quota shares, a shed ladder and queue limits. Then it covers the case that breaks most designs, a failover target with less quota than the primary.

The three lanes

Assign every request to exactly one lane at the edge, before it reaches the gateway. Do not infer the lane from the model name or prompt length; a gateway sees a payload, not intent.

LaneWho is waitingDeadlineOn shortage
InteractiveA person, with a stream open or a spinner showingSecondsDegrade, then reject with a clear message
BackgroundA job, agent step or webhook handlerMinutesQueue with a hard age limit, then defer
BatchNobody; backfills, evals, nightly summariesHoursPause entirely; resume when quota recovers
Lane definitions. The deadline is what the caller can tolerate, not what the system usually achieves.

Three rules keep the model honest:

  • The lane is a property of the caller's intent. A support agent's final answer is interactive; the same agent's overnight re-indexing is batch, even if both use the same model and prompt template.
  • Lanes do not promote themselves. A background job that fails and retries stays in background. Retries must not climb into a higher lane.
  • Work that can wait an hour belongs on a path built for waiting. If a provider offers a batch interface or a lower-rate background window, send batch-lane work there and take it off the synchronous quota altogether; see batch or synchronous for how to decide which workloads move.

Tag the lane on every request. INFRO routes can match a key in the request's metadata object, and the same attribution values arrive from metadata or X-INFRO-* headers.2 Using a lane key means the same value drives routing, appears on the trace and can be filtered later.

tag_lane.py
from openai import OpenAI

client = OpenAI(base_url="https://api.infro.io/v1", api_key=INFRO_KEY)

resp = client.chat.completions.create(
    model="chat-interactive",          # a route name, see the lane routes below
    messages=[{"role": "user", "content": "Where is my order?"}],
    metadata={"lane": "interactive", "feature": "support-chat"},
    stream=True,
)

Reserve quota per lane, in the provider's units

Provider limits are not one number. Azure OpenAI's quota page lists both requests per minute and tokens per minute per model and deployment type, and for some model versions the ratio between them differs: the page says versions of gpt-chat-latest from 2026-05-05 to 2026-06-24 use 10 RPM per 1,000 TPM, while 2026-08-06 and 2026-10-07 use 1 RPM per 1,000 TPM.3 On the newer ratio, a lane of short interactive requests can exhaust the request limit long before it uses its tokens. Your limiter has to track both, and admit a request only if both buckets can pay for it.

Split the quota into three kinds of share:

  1. Reserved share. Capacity only that lane may use. Interactive gets the largest reserve, because it is the lane that cannot wait.
  2. Borrowable share. Capacity a lane may use only when higher lanes are not using it. Batch borrows; it never reserves.
  3. Headroom. A slice nobody uses in steady state, so a burst of interactive traffic does not have to preempt work already running.

Admission uses a token estimate, not a measurement. INFRO exposes its own upper-bound estimate of prompt size as input_tokens for route conditions, alongside the request's max_tokens.2 Reserve prompt estimate plus max_tokens at admission, then return the difference when the response reports real usage. Without a max_tokens cap on background and batch calls, one long completion can consume a lane's whole minute.

admit.py
def admit(lane, est_tokens, now):
    b = buckets[lane]                    # token + request buckets for this lane
    if b.try_take(tokens=est_tokens, requests=1, now=now):
        return "ok"
    # Borrow only if every higher lane is idle enough and shed level allows it.
    if lane != "interactive" and shed_level < BORROW_CUTOFF[lane]:
        if borrow_pool.try_take(tokens=est_tokens, requests=1, now=now,
                                leave_for=higher_lanes(lane)):
            return "ok"
    return "queue" if queue_has_room(lane) else "shed"

A shed ladder with fixed order

Do not decide at 3 a.m. what to drop. Define levels, the signal that moves between them, and what each level cuts. Drive the level from signals you can read: the rate of provider 429s over a short window, queue wait at the 95th percentile, and any Retry-After value.

LevelTrigger (examples)BatchBackgroundInteractive
0 Normal429 rate near zeroBorrow idle capacityReserved shareReserved share
1 PressureSustained provider 429s, or queue wait over half the deadlinePaused, held in its queueReserved share only, no borrowingUnchanged
2 Shortage429s continue after level 1PausedNew work queued; oldest past its age limit droppedLower max_tokens, route to a smaller model
3 EmergencyInteractive wait breaches its deadlinePausedDeferred; callers told to retry laterAdmit only up to the reserved share; reject the rest with a clear error
Shed ladder. Thresholds are yours to set from your own baselines.

Two rules make the ladder safe:

  • Move down fast, move up slowly. Raise the shed level on the first confirmed signal. Lower it only after a quiet window several times longer than the one that raised it. Otherwise the level flaps and the lanes oscillate.
  • Degradation comes before rejection for interactive. A shorter answer from a smaller model is usually better for the user than an error. Test that the smaller model is acceptable first; see fallback equivalence.

Tell the caller what happened. INFRO's error model separates a limit INFRO enforces (rate_limit_exceeded) from a 429 your own provider returned, which carries the code provider_rate_limited.4 Your own shed responses should follow the same idea: a distinct code naming the lane and reason, so a client does not read "the platform is shedding background work" as "I personally sent too much". The RFC also notes that answering every request in an overload with a 429 itself consumes resources, and that dropping connections may be more appropriate under attack.1 At level 3, reject cheaply and early, before parsing large bodies.

Queue limits come from deadlines

A queue converts overload into latency. That is useful only while the wait is shorter than the caller's deadline. Past that, a queued request is wasted work: it still consumes quota when it finally runs, and nobody is waiting for the answer.

Size each queue from the deadline and the drain rate, not from a round number:

Text
max_depth = admitted_rate_per_second × (deadline_seconds − p50_service_seconds)
max_age   = deadline_seconds − p50_service_seconds

Rules for queues:

  • Enforce age as well as depth. Check the age at dequeue time and drop expired entries without calling the provider.
  • Under overload, serve the newest interactive request first. The oldest is the most likely to have been abandoned by the user. Keep first-in-first-out order for background, where fairness matters more than recency.
  • Never let a lower lane queue ahead of a higher one. Dequeue by lane priority, then by order within the lane.
  • Bound in-flight work too. Queued requests are not the only risk; streams already open hold quota for their whole duration. A per-key limit on concurrent requests, streams included, keeps a single lane from occupying every slot; INFRO offers a concurrency limit per gateway key.

Retries are where storms come from

Every retry is new demand on the quota that just refused you. Three settings matter:

  • Background and batch retries must not use the interactive reserve. Charge retries to the original lane's bucket.
  • Keep gateway retries off lower lanes. INFRO routes can retry the same target 0–3 times, with no backoff between retries.5 On a background or batch route, set retries to 0 and put the backoff in your queue worker, where you can honor Retry-After and add jitter.
  • Use failure rules by kind. A route's failure rules decide per failure kind (rate limit, server error, timeout, connection, rejected key) whether to retry, move to the next target at once, or stop.5 For the batch lane, a rate-limit failure should stop. For interactive, it should move on.

A related caution: INFRO documents X-RateLimit-Remaining as a best-effort upper bound, because rate limiting is enforced at the edge location serving the request rather than from one global counter. It says to back off as the value nears zero but not to schedule against it.6 Treat any remaining-quota header the same way, as a hint for the shed level, not as admission control.

When failover lands on less quota

A degraded failover is where lanes pay for themselves. Failover itself is straightforward: INFRO moves a request to the next eligible connection when the first fails before the first token, including on a provider 429, and a provider 429 does not count against the connection's health.7 The trouble is what the next connection can absorb.

Three failure modes recur:

  1. The standby is smaller. Azure's quota tiers differ by an order of magnitude. In the Tier 1 table, gpt-6-astra on Global Standard lists 250 RPM and 250,000 TPM, and 100 RPM and 100,000 TPM on Data Zone Standard; the Tier 2 row for the same model lists 1,000 RPM.3 If the primary is a Tier 2 subscription and the standby is a Tier 1, a failover sends full traffic into a quarter of the capacity.
  2. The pool is shared. For Global Standard, deployments of the same model and version share one quota pool across all regions in a subscription, and Data Zone Standard shares one per data zone.3 A second region in the same subscription is not extra capacity for that model.
  3. Every lane fails over together. Without lane-aware routing, batch work lands on the standby along with everything else and consumes the capacity interactive needed.

The fix is to give lanes different target lists, so standby capacity is reserved for the lane that needs it. INFRO named routes accept priority or weighted targets over your own connections, and route rules can match on metadata.2 A layout:

RouteTargetsRetriesNotes
chat-interactivePrimary, then standby1A smaller-model rule at shed level 2
chat-backgroundPrimary only0Never touches the standby
chat-batchPrimary only, or a batch interface0Paused from your side at level 1
Lane routes. Route names are examples.

Note the limits of this. A request that names a model directly is tried across every eligible connection for that vendor, so lane routing only works if each lane calls its own route name.7 Also, INFRO does not add capacity: it states that your own provider accounts keep their model, token and request quotas and that it adds no shared pool of purchased capacity.6 Routes decide where requests go. The queue, the quota shares and the shed ladder in this guide live in your application or worker layer, and a per-key request ceiling is a coarse lane control, not a reservation. It helps only when the sum of lane ceilings stays below the provider's quota with interactive headroom left over.

Provider-side overflow features can be lane-aware too. Azure's spillover sends overage from a provisioned deployment to a standard deployment, and it can be switched on per request with the x-ms-spillover-deployment header. If the header is not set, spillover does not start. Spillover requests are billed at the standard deployment's token rates.8 Setting the header only on the interactive lane keeps background traffic from spilling onto pay-as-you-go capacity.

See which lane was shed

You cannot tune what you cannot see. Record, per lane: admitted, queued, shed and deferred counts, queue wait at p50 and p95, provider 429 rate, shed level over time, and the failover and spillover rate. Group by the lane tag you set at the edge.

INFRO records model, latency, token counts and estimated provider cost on every request, and traces carry the route and connection that served each one; spans can be sent to your own OpenTelemetry collector. OpenTelemetry's GenAI semantic conventions now live in their own repository, so check there for current attribute names before you standardize spans, and add lane and shed.level as your own attributes.9 Alert on the interactive lane's queue wait breaching half its deadline, not on the raw 429 count, which mostly measures background noise.

Rehearse it. Cut your effective limit in a staging environment and confirm that level 1 pauses batch within one window, that interactive wait stays under its deadline, and that nothing retried its way into a higher lane. Failover drills covers the provider-failure side of the same exercise.

Checklist before the next storm

  • Every caller sets a lane; requests without one default to background, not interactive.
  • Reserved and borrowable shares are written down per model, in both RPM and TPM.
  • max_tokens is capped on background and batch calls.
  • Each queue has a maximum age and depth derived from its deadline.
  • Retries are charged to the original lane; background and batch routes retry 0 times at the gateway.
  • Standby connections appear only in the interactive route.
  • The standby's quota was checked against interactive peak, and any shared pool counted once.
  • The shed ladder has named levels, triggers and a slow way back up.
  • Shed responses carry a distinct code, lane and reason.
  • A drill has cut the quota and the dashboard showed the right lane shed.

Questions readers ask

Should shed requests get a 429 or a 503?
Use a status your clients do not already treat as their own fault, and add a machine-readable code that names the lane and the reason. RFC 6585 defines 429 as the user having sent too many requests in a given amount of time and allows a Retry-After header1; a shed caused by upstream capacity is a different condition, so give it a distinct error code even if you keep the status.
Does a per-key rate limit enforce lane priority?
No. A key limit caps what one key can send; it does not reserve provider capacity for another key. It becomes a lane control only when the lane ceilings together stay below the provider quota with interactive headroom left over.

From the team that builds it

Where this shows up in INFRO

INFRO is a control plane in front of the provider accounts you already hold. The capabilities below are the parts of it this article touches, each with the documentation that defines its behaviour.

  • Health-aware routing

    Available with a plan

    Named routes use priority or weighted targets over your own connections, with health checks, retries and failover.

    Read the documentation for Health-aware routing
  • Routing rules

    Available with a plan

    If-rules on a route send matching requests to their own targets, by metadata, feature, end user, key, project, prompt size, streaming, images, tools or time of day. Failure rules decide per failure kind whether to retry, move on or stop.

    Read the documentation for Routing rules
  • Automatic failover

    Available with a plan

    A route that times out or 5xxs before the response starts is retried elsewhere without your client seeing it.

    Read the documentation for Automatic failover
  • Concurrency limit per key

    Available with a plan

    The most requests a gateway key may have in flight at once, streams included; one more is refused until one finishes.

  • Request traces and payload logs

    Available with a plan

    Per-request timing detail, with prompt and completion logging off by default and a retention window you choose.

    Read the documentation for Request traces and payload logs
  • OpenTelemetry export

    Available with a plan

    Spans pushed to your own collector from the request path, with the endpoint and headers set in the console. Included on every plan.

    Read the documentation for OpenTelemetry export
  • Usage and cost per request

    Available with a plan

    Every request records its model, latency, token or unit counts, and estimated provider cost, queryable by project and key.

    Read the documentation for Usage and cost per request

INFRO accounts are self-serve. Provider usage is billed to you by your providers; INFRO charges a software subscription and takes 0% of model spend. See plans.

Sources

Numbered in order of first citation. Each was opened and checked against the claim on the date shown.

  1. 1.RFC 6585: Additional HTTP Status Codes (opens in a new tab) — IETF / RFC Editor. Accessed Nov 20, 2026.
  2. 2.Routes (opens in a new tab) — INFRO Docs. Accessed Nov 20, 2026.
  3. 3.Azure OpenAI in Microsoft Foundry Models quotas and limits (opens in a new tab) — Microsoft Learn. Accessed Nov 20, 2026.
  4. 4.Errors (opens in a new tab) — INFRO Docs. Accessed Nov 20, 2026.
  5. 5.Smart routing (opens in a new tab) — INFRO Docs. Accessed Nov 20, 2026.
  6. 6.Rate & spend limits (opens in a new tab) — INFRO Docs. Accessed Nov 20, 2026.
  7. 7.Failover & fallbacks (opens in a new tab) — INFRO Docs. Accessed Nov 20, 2026.
  8. 8.Manage traffic with spillover for provisioned deployments (opens in a new tab) — Microsoft Learn. Accessed Nov 20, 2026.
  9. 9.Generative AI semantic conventions (moved) (opens in a new tab) — OpenTelemetry. Accessed Nov 20, 2026.

About this article

Read on 2026-11-20: RFC 6585, the OpenTelemetry GenAI conventions landing page, Azure OpenAI quotas and limits, Azure's spillover guide, and INFRO's docs on limits, errors, fallbacks, routing and routes. Provider quota figures beyond the cited Azure rows, and any provider's internal queueing behaviour, were left out as unconfirmed.

Last checked against its sources on . The next review is due by . No separate named reviewer is recorded for this article.

How articles are written, sourced and corrected is set out in the editorial standards.