What this comparison covers, and what it does not
The question is which agent and multimodal work can wait for a lower rate without a rewrite. One caveat first: OpenAI's pricing page and its agents announcement could not be read when this was written (both returned HTTP 403). So this article does not state rates, window lengths or eligibility rules for any newer background completion window. Everything about OpenAI below is what is documented for Flex and Batch, as summarised in INFRO's reads of OpenAI's pages. Treat a new window as a third column in the table below, and fill it from OpenAI's own page before you route anything to it.
The method does not depend on that column. Any slower window has three properties you must measure: how long it can take, what happens when it cannot serve you, and what a job costs once failures are included.
Flex, Batch and standard side by side
Flex bills at Batch rates for Responses or Chat Completions requests, in exchange for slower responses and occasional resource unavailability. It is in beta with limited model availability and returns 429 Resource Unavailable, without charge, when there is no capacity.1 Batch is a file-based API with a 24-hour window and a separate rate-limit pool.2
| Standard | Flex | Batch API | |
|---|---|---|---|
| Rate | List rate | Batch rate (half) | Half the synchronous rate |
| How you send it | Normal request | Normal request with service_tier: "flex" | JSONL file, then poll |
| Latency | Seconds | Slower; OpenAI suggests a 15-minute client timeout | Up to 24 hours, often sooner |
| When capacity is short | Normal 429 handling | 429 Resource Unavailable, not charged | Unfinished requests expire at 24 hours; completed ones are billed |
| Tool-calling loops | Yes | Yes, step by step | Not inside one request |
| Code change | None | A field and a longer timeout | A new submit, poll and match pipeline |
On GPT-6 Sol, Flex is $1.00 input and $5.00 output per million tokens against $2.00 and $10.00 standard, up to 272K input tokens.1 The rate card lists Batch and Flex at half the standard rate across models, for example GPT-6 Astra on Batch at $5 input and $25 output.3 These are OpenAI's list rates, which your own contract may change.
Latency: what "background" really means for an agent
An agent run is a chain of model calls. A window that adds minutes to each call adds minutes times the number of steps to the job. A 12-step run with a 2-minute worst-case step is a 24-minute job, and the tail is set by the slowest step, not the average one.
Batch is different in kind. Its window applies to a whole request, and OpenAI's Batch API completes within 24 hours, often sooner.2 That suits a stage that runs the same prompt over many items. It does not suit the loop, because each tool result has to come back before the next request exists. The practical split:
- Flex for the loop itself: each step is an ordinary request, so state, tools and the Responses API shape stay as they are.
- Batch for the fan-out stages around the loop: classify 20,000 documents, caption 5,000 images, grade last night's runs.
- Standard for any step a person is waiting on, and for the final step if a deadline is attached to it.
OpenAI also advises against running large ETL or batch jobs in Fast mode, which is a sign that each tier has an intended traffic shape.1 The same applies in the other direction.
Retries: where a slow lane turns into a bill
A refused Flex request is not charged, but the steps before it were. OpenAI suggests either backing off or retrying with service_tier: "auto" when finishing matters more than price.1 Which one you pick decides whether a job costs more than standard would have.
Batch has a different failure shape. Nothing times out in front of a user; requests expire or error individually, results may come back in any order, and you match them on custom_id.2 Expiry is normal operation, so the resubmission path has to exist before the first large run.
Three rules keep retries from eating the rate gap:
- Make steps idempotent or resumable. If a run is restarted from scratch after a refusal at step 8 of 12, you pay for 8 steps twice.
- Cap Flex attempts per step, then fall back to standard for the remaining steps rather than the whole job.
- Back off, don't hammer. A gateway route that retries the same target retries immediately, with no backoff,4 which is the wrong behaviour for a capacity refusal. Set the failure rule for that kind of failure explicitly.
Cost per finished job
Compare options by what a completed job costs, failures included:
expected cost = (1 − p) × slow_cost + p × (slow_cost_wasted + fallback_cost)
where p is the share of jobs that hit a refusal and fall back.
The lesson is about design more than price: resumability matters more than the headline rate. Whatever a newer background window lists, put its rate and refusal behaviour into the same formula.
An eligibility rule you can encode
Work is eligible for a slow lane only if all four hold:
- Nobody, human or live process, waits on the answer inside the window.
- Each step is idempotent, or the run can resume from its last completed step.
- A missed deadline is an acceptable outcome, or a standard-rate fallback is allowed by budget.
- The model and features you use are available on the slow lane (Flex has limited model availability).1
Multimodal work needs one more check: INFRO's async jobs API today covers video generation, upscale and lip-sync, while images and audio are synchronous.5 A long media render already has its own job lifecycle, so decide per modality whether a slow text window adds anything.
from dataclasses import dataclass
@dataclass
class Job:
user_waiting: bool
resumable: bool
deadline_minutes: int | None
standard_fallback_allowed: bool
model_on_flex: bool
def lane(job: Job) -> str:
if job.user_waiting or not job.model_on_flex:
return "standard"
if not job.resumable and not job.standard_fallback_allowed:
return "standard"
if job.deadline_minutes is not None and job.deadline_minutes < 30:
return "standard"
return "background"Routing only eligible work
Keep the decision in one place and let configuration carry it. In an INFRO route, a name your application sends as model resolves to your own provider connections, and up to 20 rules can send matching requests to their own targets, checked in order with the first match deciding.6 Rules can read metadata, the workflow and step attribution values, has_tools, input_tokens and the hour in UTC.6 So an application tags eligible work and a rule sends it to the background target:
{
"name": "Background agent work",
"match": "all",
"conditions": [
{ "field": "metadata", "key": "lane", "op": "in", "value": ["background"] }
],
"targets": [{ "connection_id": "conn_…", "model": "openai/gpt-6-sol" }],
"fallthrough": true
}The condition follows the shape in the routes docs; confirm exact operators there before you copy it. Two cautions. First, whether a request field such as service_tier reaches OpenAI is a question for the endpoint reference, so check it before relying on it; a design that works either way keeps background traffic on its own route and its own OpenAI project, where OpenAI lets you choose which service tiers each project may use.1 Second, any upstream failure before the response starts moves a request to the next target, and a route's failure rules can change that per kind of failure: retry, move on or stop.4 Set the rate-limit kind to move to the standard target, not to retry in place.
Traffic that goes through INFRO records the model, latency and estimated provider cost per request, so you can sum cost per finished job by trace. Costs are estimates against your own rates, not a bill; your providers invoice you directly. Batch files submitted straight to OpenAI do not pass through a gateway, so give them their own provider project to attribute them.2 OpenAI's Responses API through INFRO is covered by the Claude Code and Codex ingress, with routing and budget checks applied as on any other request.
A rollout order
- Pick one eligible workflow and tag it with
lane: backgroundin metadata. - Run it on Flex for a week alongside a standard baseline, recording refusals, step latency and finished-job cost.
- Add resume-on-standard before widening the share.
- Move fan-out stages to Batch only when a pipeline that matches
custom_idand resubmits expired items is in place. - When OpenAI's newer background windows are readable and available for your models, add them as a third lane with the same measurements and the same fallback.
