Batch or synchronous: which LLM workloads should move

Most teams send every model call synchronously because that is how the first prototype worked. A good share of production LLM work does not need an answer in seconds: nightly summaries, backfills over stored documents, evaluation runs, classification of yesterday's tickets. The major providers publish a second way to send that work, at half the list rate, in exchange for giving them up to a day to do it.

This guide sets out what the trade is, which workloads should take it, and how to run a batch without losing requests.

What the providers offer

OpenAI's Batch API processes a file of requests asynchronously at half the synchronous rate. Each batch completes within 24 hours, often sooner, and draws on a separate rate-limit pool that does not consume your standard per-model limits. One batch holds up to 50,000 requests in an input file of up to 200 MB. It works with Chat Completions, Responses, Embeddings, Completions, Moderations and Images¹.

Anthropic's Message Batches API takes up to 100,000 Messages requests or 256 MB per batch, whichever comes first, bills them at 50% of standard input and output, and says most batches finish in less than an hour. Results are available once every request has finished or after 24 hours, and are kept for 29 days².

Google Gemini lists a Batch tier on its pricing page: Gemini 3.8 Flash is $0.375 input and $1.875 output per 1M tokens on Batch, against $0.75 and $3.75 standard, through 2026-12-31³.

The per-token effect is exact: GPT-6 Sol is $1.00 input and $5.00 output per 1M on Batch against $2.00 and $10.00 standard⁴; Claude Sonnet 5.5 is $1 and $5 against $2 and $10, and Claude Opus 5.5 $2 and $10 against $4 and $20⁵.

Between synchronous and batch, OpenAI also offers Flex processing: set service_tier to "flex" on a normal request and it is priced at Batch API rates, with slower responses and occasional 429 Resource Unavailable errors that are not charged. OpenAI suggests it for evaluations, data enrichment and asynchronous workloads, and raises the client timeout in its examples to 15 minutes⁶.

Which workloads should move

The deciding question is who waits for the answer.

• A person is waiting. Chat, autocomplete, a support agent mid-conversation. Stay synchronous. Lower the cost with prompt caching and model choice, not with latency.
• A live process is waiting, but minutes are acceptable. A document pipeline that runs after upload, a report generated on request. A flex or low-priority tier fits, with a long timeout and a retry on unavailability.
• Nothing is waiting until later today. Evaluations, backfills, re-classifying a corpus after a taxonomy change, nightly digests, embedding a new data set. Batch.
• The work is multi-step with tool calls. An agent that calls a tool, reads the result and decides again cannot live inside one batch request. Run it synchronously, or restructure it so each step over many items is its own batch.

Two workload types benefit more than the headline rate suggests. Long reasoning jobs: Anthropic recommends batch processing for thinking budgets above 32k tokens, because such long-running requests can hit system timeouts and open-connection limits⁷. And jobs sharing a long prefix: Anthropic's caching multipliers stack with the batch rate, though cache hits inside a batch are best-effort, with typical hit rates of 30% to 98%².

Running a batch without losing requests

The failure modes are not the synchronous ones. Nothing times out in front of a user; instead, some requests simply do not come back.

• Order is not preserved. On both providers, results may not match the input order; match on the custom_id you set on each request¹².
• Expiry is normal. OpenAI marks requests unfinished at 24 hours as expired and charges only for completed requests¹. Anthropic returns each request as succeeded, errored, canceled or expired, and does not bill the last three².
• Errors are per request. A malformed request in the file fails on its own; the rest continue.

So every batch pipeline needs three things: a stable custom_id per unit of work, a store of which ids succeeded, and a resubmission step for the rest.

import json
import time
from openai import OpenAI

client = OpenAI()  # a key for your own OpenAI project

def submit(items: list[dict]) -> str:
    with open("batch.jsonl", "w", encoding="utf-8") as f:
        for item in items:
            f.write(json.dumps({
                "custom_id": item["id"],
                "method": "POST",
                "url": "/v1/chat/completions",
                "body": {
                    "model": "gpt-6-sol",
                    "messages": [
                        {"role": "system", "content": "Classify the ticket. Reply with one label."},
                        {"role": "user", "content": item["text"]},
                    ],
                    "max_tokens": 20,
                },
            }) + "\n")
    upload = client.files.create(file=open("batch.jsonl", "rb"), purpose="batch")
    batch = client.batches.create(
        input_file_id=upload.id,
        endpoint="/v1/chat/completions",
        completion_window="24h",
    )
    return batch.id

def collect(batch_id: str) -> tuple[dict[str, str], set[str]]:
    while True:
        batch = client.batches.retrieve(batch_id)
        if batch.status in ("completed", "expired", "cancelled", "failed"):
            break
        time.sleep(60)
    answers: dict[str, str] = {}
    if batch.output_file_id:
        for line in client.files.content(batch.output_file_id).text.splitlines():
            row = json.loads(line)
            if row["response"] and row["response"]["status_code"] == 200:
                body = row["response"]["body"]
                answers[row["custom_id"]] = body["choices"][0]["message"]["content"]
    failed: set[str] = set()
    if batch.error_file_id:
        for line in client.files.content(batch.error_file_id).text.splitlines():
            failed.add(json.loads(line)["custom_id"])
    return answers, failed

Anything neither answered nor listed as failed — for example, requests lost to an expired batch — is found by comparing the ids you submitted with the ids you got back. Resubmit those in the next batch.

The Anthropic equivalent reports each request's outcome directly:

import anthropic

client = anthropic.Anthropic()

def resubmit_ids(batch_id: str) -> list[str]:
    retry: list[str] = []
    for result in client.messages.batches.results(batch_id):
        kind = result.result.type
        if kind == "succeeded":
            store(result.custom_id, result.result.message)
        elif kind in ("errored", "expired", "canceled"):
            retry.append(result.custom_id)
    return retry

def store(custom_id: str, message) -> None:
    ...  # write to your own results table, keyed by custom_id

Cost accounting for batch traffic

A batch job's cost appears on your provider's invoice like any other usage, at the batch rate. Anthropic's cost report marks it with service_tier: batch when grouped by description. The trouble is attribution: batch jobs usually run from scripts, not from the service that tags its requests, so they arrive on the invoice with no team, no feature and no workflow.

Give batch work its own provider project, workspace or API key, and record each batch's purpose, owner and item count when you submit it. That turns a line item nobody recognises into one that can be booked.

Measuring the move

Before moving a workload, record three numbers on the synchronous path for a representative week: estimated provider cost, the number of units of work completed, and how long results took to be used downstream. After the move, measure the same three. The rate halves by definition; the questions worth answering are whether the unit count held (expired and errored requests that were never resubmitted quietly shrink it) and whether anything downstream was waiting on results it now gets hours later.

Watch for rate-limit side effects too. OpenAI's batch pool is separate from synchronous limits¹, so moving a large nightly job off the synchronous path can free headroom for user-facing traffic during the hours it used to run. That is often worth as much as the rate itself, and it only shows if you track throttling on the synchronous path before and after.

A checklist before moving a workload

• Nobody waits on the answer for at least an hour, or a flex tier with a long timeout fits.
• Each unit of work has a stable custom_id.
• Results are stored by id, and a resubmission step handles expired and errored ids.
• Requests set max_tokens, so a runaway answer cannot inflate the batch.
• The batch runs under its own provider project, workspace or key, with an owner recorded.
• The result quality has been compared with the synchronous path on a sample; the model is the same, but prompts built for chat sometimes assume a follow-up turn that a batch will never have.

Batch traffic and INFRO

INFRO's gateway serves synchronous requests; it has no batch endpoint, so a batch job goes from your code straight to your provider account with your own provider key and is billed there at the provider's batch rate. INFRO does not see it per request. Its asynchronous jobs are for media generation, not for batched text.

Where that shows up is reconciliation. Comparing INFRO's estimate with OpenAI's or Anthropic's reported cost, a batch job looks like reported spend with no matching requests; INFRO names batch jobs among the typical sources of traffic that bypasses it, and with usage access it lists the provider keys involved. Running batch work under its own provider key makes that finding immediately recognisable instead of an unexplained gap.