Why quota shortage is a priority problem, not a retry problem
A 429 storm has a simple cause: demand for tokens or requests per minute exceeds what your provider account will admit. RFC 6585 defines 429 as the user having sent too many requests in a given amount of time, and says only that the response MAY carry a Retry-After header. It does not say how the server identifies the user or counts requests.1 In practice that means the limit you hit may be shared with every other workload on the same account, and the response will not tell you which of your callers consumed it.
When that happens, someone's request gets served and someone's does not. If you have not decided who, the decision is made by arrival order: whichever caller retries fastest wins. Background agents and backfills usually retry fastest and hold the most requests in flight, so they win. The person waiting on a chat answer loses.
Circuit breakers and per-key rate limits do not fix this. A breaker answers "is the provider healthy?" and a throttled provider is healthy but full. A rate limit caps one caller but says nothing about relative importance. What is missing is an explicit rule for which work survives when capacity is short. This guide builds that rule in four parts: lanes, quota shares, a shed ladder and queue limits. Then it covers the case that breaks most designs, a failover target with less quota than the primary.
The three lanes
Assign every request to exactly one lane at the edge, before it reaches the gateway. Do not infer the lane from the model name or prompt length; a gateway sees a payload, not intent.
| Lane | Who is waiting | Deadline | On shortage |
|---|---|---|---|
| Interactive | A person, with a stream open or a spinner showing | Seconds | Degrade, then reject with a clear message |
| Background | A job, agent step or webhook handler | Minutes | Queue with a hard age limit, then defer |
| Batch | Nobody; backfills, evals, nightly summaries | Hours | Pause entirely; resume when quota recovers |
Three rules keep the model honest:
- The lane is a property of the caller's intent. A support agent's final answer is interactive; the same agent's overnight re-indexing is batch, even if both use the same model and prompt template.
- Lanes do not promote themselves. A background job that fails and retries stays in background. Retries must not climb into a higher lane.
- Work that can wait an hour belongs on a path built for waiting. If a provider offers a batch interface or a lower-rate background window, send batch-lane work there and take it off the synchronous quota altogether; see batch or synchronous for how to decide which workloads move.
Tag the lane on every request. INFRO routes can match a key in the request's metadata object, and the same attribution values arrive from metadata or X-INFRO-* headers.2 Using a lane key means the same value drives routing, appears on the trace and can be filtered later.
from openai import OpenAI
client = OpenAI(base_url="https://api.infro.io/v1", api_key=INFRO_KEY)
resp = client.chat.completions.create(
model="chat-interactive", # a route name, see the lane routes below
messages=[{"role": "user", "content": "Where is my order?"}],
metadata={"lane": "interactive", "feature": "support-chat"},
stream=True,
)Reserve quota per lane, in the provider's units
Provider limits are not one number. Azure OpenAI's quota page lists both requests per minute and tokens per minute per model and deployment type, and for some model versions the ratio between them differs: the page says versions of gpt-chat-latest from 2026-05-05 to 2026-06-24 use 10 RPM per 1,000 TPM, while 2026-08-06 and 2026-10-07 use 1 RPM per 1,000 TPM.3 On the newer ratio, a lane of short interactive requests can exhaust the request limit long before it uses its tokens. Your limiter has to track both, and admit a request only if both buckets can pay for it.
Split the quota into three kinds of share:
- Reserved share. Capacity only that lane may use. Interactive gets the largest reserve, because it is the lane that cannot wait.
- Borrowable share. Capacity a lane may use only when higher lanes are not using it. Batch borrows; it never reserves.
- Headroom. A slice nobody uses in steady state, so a burst of interactive traffic does not have to preempt work already running.
Admission uses a token estimate, not a measurement. INFRO exposes its own upper-bound estimate of prompt size as input_tokens for route conditions, alongside the request's max_tokens.2 Reserve prompt estimate plus max_tokens at admission, then return the difference when the response reports real usage. Without a max_tokens cap on background and batch calls, one long completion can consume a lane's whole minute.
def admit(lane, est_tokens, now):
b = buckets[lane] # token + request buckets for this lane
if b.try_take(tokens=est_tokens, requests=1, now=now):
return "ok"
# Borrow only if every higher lane is idle enough and shed level allows it.
if lane != "interactive" and shed_level < BORROW_CUTOFF[lane]:
if borrow_pool.try_take(tokens=est_tokens, requests=1, now=now,
leave_for=higher_lanes(lane)):
return "ok"
return "queue" if queue_has_room(lane) else "shed"A shed ladder with fixed order
Do not decide at 3 a.m. what to drop. Define levels, the signal that moves between them, and what each level cuts. Drive the level from signals you can read: the rate of provider 429s over a short window, queue wait at the 95th percentile, and any Retry-After value.
| Level | Trigger (examples) | Batch | Background | Interactive |
|---|---|---|---|---|
| 0 Normal | 429 rate near zero | Borrow idle capacity | Reserved share | Reserved share |
| 1 Pressure | Sustained provider 429s, or queue wait over half the deadline | Paused, held in its queue | Reserved share only, no borrowing | Unchanged |
| 2 Shortage | 429s continue after level 1 | Paused | New work queued; oldest past its age limit dropped | Lower max_tokens, route to a smaller model |
| 3 Emergency | Interactive wait breaches its deadline | Paused | Deferred; callers told to retry later | Admit only up to the reserved share; reject the rest with a clear error |
Two rules make the ladder safe:
- Move down fast, move up slowly. Raise the shed level on the first confirmed signal. Lower it only after a quiet window several times longer than the one that raised it. Otherwise the level flaps and the lanes oscillate.
- Degradation comes before rejection for interactive. A shorter answer from a smaller model is usually better for the user than an error. Test that the smaller model is acceptable first; see fallback equivalence.
Tell the caller what happened. INFRO's error model separates a limit INFRO enforces (rate_limit_exceeded) from a 429 your own provider returned, which carries the code provider_rate_limited.4 Your own shed responses should follow the same idea: a distinct code naming the lane and reason, so a client does not read "the platform is shedding background work" as "I personally sent too much". The RFC also notes that answering every request in an overload with a 429 itself consumes resources, and that dropping connections may be more appropriate under attack.1 At level 3, reject cheaply and early, before parsing large bodies.
Queue limits come from deadlines
A queue converts overload into latency. That is useful only while the wait is shorter than the caller's deadline. Past that, a queued request is wasted work: it still consumes quota when it finally runs, and nobody is waiting for the answer.
Size each queue from the deadline and the drain rate, not from a round number:
max_depth = admitted_rate_per_second × (deadline_seconds − p50_service_seconds)
max_age = deadline_seconds − p50_service_secondsRules for queues:
- Enforce age as well as depth. Check the age at dequeue time and drop expired entries without calling the provider.
- Under overload, serve the newest interactive request first. The oldest is the most likely to have been abandoned by the user. Keep first-in-first-out order for background, where fairness matters more than recency.
- Never let a lower lane queue ahead of a higher one. Dequeue by lane priority, then by order within the lane.
- Bound in-flight work too. Queued requests are not the only risk; streams already open hold quota for their whole duration. A per-key limit on concurrent requests, streams included, keeps a single lane from occupying every slot; INFRO offers a concurrency limit per gateway key.
Retries are where storms come from
Every retry is new demand on the quota that just refused you. Three settings matter:
- Background and batch retries must not use the interactive reserve. Charge retries to the original lane's bucket.
- Keep gateway retries off lower lanes. INFRO routes can retry the same target 0–3 times, with no backoff between retries.5 On a background or batch route, set retries to 0 and put the backoff in your queue worker, where you can honor Retry-After and add jitter.
- Use failure rules by kind. A route's failure rules decide per failure kind (rate limit, server error, timeout, connection, rejected key) whether to retry, move to the next target at once, or stop.5 For the batch lane, a rate-limit failure should stop. For interactive, it should move on.
A related caution: INFRO documents X-RateLimit-Remaining as a best-effort upper bound, because rate limiting is enforced at the edge location serving the request rather than from one global counter. It says to back off as the value nears zero but not to schedule against it.6 Treat any remaining-quota header the same way, as a hint for the shed level, not as admission control.
When failover lands on less quota
A degraded failover is where lanes pay for themselves. Failover itself is straightforward: INFRO moves a request to the next eligible connection when the first fails before the first token, including on a provider 429, and a provider 429 does not count against the connection's health.7 The trouble is what the next connection can absorb.
Three failure modes recur:
- The standby is smaller. Azure's quota tiers differ by an order of magnitude. In the Tier 1 table,
gpt-6-astraon Global Standard lists 250 RPM and 250,000 TPM, and 100 RPM and 100,000 TPM on Data Zone Standard; the Tier 2 row for the same model lists 1,000 RPM.3 If the primary is a Tier 2 subscription and the standby is a Tier 1, a failover sends full traffic into a quarter of the capacity. - The pool is shared. For Global Standard, deployments of the same model and version share one quota pool across all regions in a subscription, and Data Zone Standard shares one per data zone.3 A second region in the same subscription is not extra capacity for that model.
- Every lane fails over together. Without lane-aware routing, batch work lands on the standby along with everything else and consumes the capacity interactive needed.
The fix is to give lanes different target lists, so standby capacity is reserved for the lane that needs it. INFRO named routes accept priority or weighted targets over your own connections, and route rules can match on metadata.2 A layout:
| Route | Targets | Retries | Notes |
|---|---|---|---|
chat-interactive | Primary, then standby | 1 | A smaller-model rule at shed level 2 |
chat-background | Primary only | 0 | Never touches the standby |
chat-batch | Primary only, or a batch interface | 0 | Paused from your side at level 1 |
Note the limits of this. A request that names a model directly is tried across every eligible connection for that vendor, so lane routing only works if each lane calls its own route name.7 Also, INFRO does not add capacity: it states that your own provider accounts keep their model, token and request quotas and that it adds no shared pool of purchased capacity.6 Routes decide where requests go. The queue, the quota shares and the shed ladder in this guide live in your application or worker layer, and a per-key request ceiling is a coarse lane control, not a reservation. It helps only when the sum of lane ceilings stays below the provider's quota with interactive headroom left over.
Provider-side overflow features can be lane-aware too. Azure's spillover sends overage from a provisioned deployment to a standard deployment, and it can be switched on per request with the x-ms-spillover-deployment header. If the header is not set, spillover does not start. Spillover requests are billed at the standard deployment's token rates.8 Setting the header only on the interactive lane keeps background traffic from spilling onto pay-as-you-go capacity.
See which lane was shed
You cannot tune what you cannot see. Record, per lane: admitted, queued, shed and deferred counts, queue wait at p50 and p95, provider 429 rate, shed level over time, and the failover and spillover rate. Group by the lane tag you set at the edge.
INFRO records model, latency, token counts and estimated provider cost on every request, and traces carry the route and connection that served each one; spans can be sent to your own OpenTelemetry collector. OpenTelemetry's GenAI semantic conventions now live in their own repository, so check there for current attribute names before you standardize spans, and add lane and shed.level as your own attributes.9 Alert on the interactive lane's queue wait breaching half its deadline, not on the raw 429 count, which mostly measures background noise.
Rehearse it. Cut your effective limit in a staging environment and confirm that level 1 pauses batch within one window, that interactive wait stays under its deadline, and that nothing retried its way into a higher lane. Failover drills covers the provider-failure side of the same exercise.
Checklist before the next storm
- Every caller sets a lane; requests without one default to background, not interactive.
- Reserved and borrowable shares are written down per model, in both RPM and TPM.
max_tokensis capped on background and batch calls.- Each queue has a maximum age and depth derived from its deadline.
- Retries are charged to the original lane; background and batch routes retry 0 times at the gateway.
- Standby connections appear only in the interactive route.
- The standby's quota was checked against interactive peak, and any shared pool counted once.
- The shed ladder has named levels, triggers and a slow way back up.
- Shed responses carry a distinct code, lane and reason.
- A drill has cut the quota and the dashboard showed the right lane shed.
