Should you hedge LLM requests? Retry budgets and tail latency

What hedging is, and what it is not

A plain retry waits for an attempt to fail or time out, then sends another. A hedged request sends a second copy while the first is still running, keeps whichever answers first, and cancels the other. Failover is a third thing: it moves to another target after a failure, so only one attempt is live at a time.

The distinction matters for LLM traffic because the second copy is billable work. A duplicate chat completion consumes input tokens, starts generating output tokens, and counts against the same request-per-minute and token-per-minute limits as the original. Providers apply those limits per organization and per project, and a limit trips on whichever of requests or tokens runs out first.¹ A hedge is therefore a deliberate trade: extra spend and extra rate-limit pressure, in exchange for a shorter tail.

Whether that trade pays depends on one question: why is the tail slow?

Find out why p99 is slow before you hedge

Hedging only helps when the slow request is slow by bad luck, so that a fresh copy would probably be fast. For LLM calls, the tail has several causes, and only some of them are like that.

Split your latency histogram by time to first token and by total duration before you decide. If the p99 mass is in total duration for long answers, a hedge will double your cost and leave the p99 where it was. Our time to first token post covers how to measure the first-token component and set timeouts around it.

The cost side: duplicate token spend

Hedge cost has three parts: the duplicate's input tokens, whatever output it generated before you cancelled it, and any server-side tool fees it triggered.

Two cautions. First, do not assume that cancelling the HTTP connection stops generation or billing at your provider; I could not confirm that behaviour from the pages I read, so check your provider's documentation and reconcile against the usage your provider reports. Second, the hedge rate is not fixed. During a provider slowdown, many more requests cross the threshold, so the duplicate rate rises exactly when the provider is struggling. That is how a hedge becomes a retry storm, and it is why the budget below is not optional.

Set a retry budget, not an attempt count

"Retry up to three times" sounds safe per request. Across the fleet it is not: if the provider is fully down, every request makes four attempts and your load on it quadruples. A retry budget caps retries and hedges as a fraction of original traffic instead.

The mechanism is a token bucket. Every original request deposits a small amount (say 0.1 of a token); every retry or hedge withdraws one whole token; when the bucket is empty, you do not retry or hedge. With a ratio of 0.1, the worst case is about 10% extra load rather than a multiple of it.

Rules that keep the budget honest:

• One budget for retries and hedges together. Otherwise a retry loop and a hedge both spend your headroom without knowing about each other.
• One layer retries. Official OpenAI SDKs automatically retry eligible 429 and 503 responses, subject to their settings.¹ If your SDK, your service wrapper and your gateway each retry three times, the multiplication is nine or twenty-seven. Disable or bound all but one layer.
• A deadline for the whole operation, not per attempt. OpenAI's guidance notes that a per-attempt timeout is not necessarily a deadline for the entire operation.¹ Hand every attempt the remaining time, and stop when it is gone.
• Scope budgets per route or model, so one failing model cannot spend the headroom of healthy ones.

The code below combines the budget with a hedge that fires only on time to first token. The start coroutine must resolve when the first token arrives, not when the answer finishes.

import asyncio, random

class RetryBudget:
    def __init__(self, ratio=0.1, cap=50.0):
        self.ratio, self.cap, self.tokens = ratio, cap, cap

    def on_request(self):
        self.tokens = min(self.cap, self.tokens + self.ratio)

    def try_spend(self):
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

async def hedged_first_token(start, budget, hedge_after, deadline_s):
    """start(attempt) resolves once the first token has arrived."""
    budget.on_request()
    loop = asyncio.get_running_loop()
    end = loop.time() + deadline_s
    tasks = [asyncio.create_task(start(1))]
    hedged = False
    try:
        while tasks:
            timeout = end - loop.time()
            if not hedged:
                timeout = min(timeout, hedge_after)
            if timeout <= 0:
                raise TimeoutError("operation deadline reached")
            done, _ = await asyncio.wait(
                tasks, timeout=timeout, return_when=asyncio.FIRST_COMPLETED
            )
            for t in done:
                tasks.remove(t)
                if t.exception() is None:
                    return t.result()      # winner; the rest are cancelled below
            if not done and not hedged:
                hedged = True
                if budget.try_spend():
                    tasks.append(asyncio.create_task(start(2)))
        raise RuntimeError("all attempts failed")
    finally:
        for t in tasks:
            t.cancel()

Honour Retry-After, and never hedge a 429

A 429 is the provider telling you that you are over a limit. A duplicate request is the wrong response. OpenAI documents Retry-After as the minimum number of seconds to wait before retrying a temporary rate-limit error, when present, and says to treat it as a minimum and add a small random delay so clients do not retry in step.¹ The header itself is defined in HTTP semantics, in RFC 9110's section on response context fields.²

Several details matter in practice:

• It also appears on 503. OpenAI documents that Retry-After may be present on 429s from a temporary rate limit and on 503s from temporary model overload.¹
• Not every 429 is retryable. The same page says the header does not mean quota, billing or other errors that need user action can be fixed by retrying, and a hard spend limit also returns a 429.¹ Branch on the error code, not on the status alone.
• slow_down is about ramp rate. A 429 with that code can occur even within your request and token limits, because traffic rose too quickly; as a rule of thumb, past 1 million input tokens per minute, OpenAI suggests raising traffic by no more than 50% every 15 minutes.¹ Turning hedging on across a whole fleet at once is exactly such a ramp.
• If the delay is longer than you can afford, stop. OpenAI's guidance is to defer the request rather than retry sooner when a server delay exceeds the configured maximum.¹ Compare Retry-After with your remaining deadline and fail fast or move to another target.

import random

def retry_delay(attempt, retry_after, remaining, base=0.5, cap=20.0):
    if retry_after is not None:
        delay = retry_after + random.uniform(0, 0.25)  # never shorter than the server asked
    else:
        delay = random.uniform(0, min(cap, base * 2 ** attempt))  # full jitter
    return delay if delay < remaining else None  # None: give up or fail over

For the broader picture of limits and headers by provider, see provider rate limits compared, and for why throttling should not trip a breaker, circuit breakers for LLM providers.

Keep non-idempotent tool calls safe

HTTP semantics classify methods by whether repeating them has the same intended effect; the definitions are in RFC 9110's sections on idempotent methods and on POST.² A chat completion is a POST. Calling the model twice with the same prompt has no side effect beyond cost and a different answer, so hedging the model call is acceptable. Executing what the model asks for is where duplicates hurt.

Three rules:

1. Hedge the model call, never the tool execution. The tool runner is a separate step. It acts on exactly one response, the winner's, and never on both.
2. Give every side-effecting tool an idempotency key derived from stable identifiers, such as run id plus step index plus tool-call id, so a retried execution after a timeout cannot repeat the effect. The key lives at your tool or the downstream API; the model provider does not provide it for you.
3. Exclude requests with server-side side effects from hedging. If the provider runs tools itself (search, fetch, code sandboxes), a duplicate runs and bills them twice. See what server-side tools cost an agent.

Streaming adds one more rule. OpenAI's guidance is that HTTP error responses apply before the stream starts, that an error after streaming begins can arrive as a stream event, and that you should not automatically replay a request after consuming output.¹ So the hedge window ends at the first token. After that, you are in recovery territory, covered in streaming LLM failure modes.

Where a gateway fits, and what to measure

The pages I read for INFRO describe two relevant behaviours. Connection failover tries the same model on another of your own connections, and a route tries its targets in priority or weighted order, both only before the first token.³ Rules on a route can match on fields such as stream and has_tools, and failure rules decide per failure kind whether to retry, move on or stop.⁴ That gives you a place to keep retry policy out of application code, and to treat tool-carrying requests differently. Those pages do not describe hedged duplicate requests, so if you hedge, build it in a layer you own and keep it under the same budget. Per-request latency, estimated cost and traces help judge the result, and spans can be exported to your own collector. Note that OpenTelemetry's GenAI conventions have moved to a separate repository, so pin the version you instrument against.⁵

Whatever layer you use, record these per request: attempt number, whether a hedge fired, which attempt won, time to first token for each, and tokens spent by the loser. Then compare the p99 gain with the duplicate spend over a week of traffic.

• Latency split by first-token wait versus total duration
• Exactly one layer retries; SDK and gateway retries are bounded
• Retry budget covers retries and hedges together
• Retry-After treated as a minimum, with jitter and a deadline check
• 429 and quota errors never trigger a hedge
• Tool execution runs once, with idempotency keys
• Hedge rate, winner and wasted tokens are logged and alerted on

If the p99 gain is small or the wasted tokens are large, drop the hedge and fix the cause: shorter outputs, a faster tier, or a second target to fail over to.