# Batch or synchronous: which LLM workloads should move

URL: https://infro.io/blog/batch-vs-synchronous-llm
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-09-29
Last modified: 2026-09-29
Last reviewed: 2026-09-29 (next review due 2026-10-29)
Topic: Cost and FinOps
Format: Guide

> Move a workload to a batch API when nothing waits on the answer for minutes to hours: evaluations, backfills, classification and enrichment of stored data, nightly summaries. OpenAI's Batch API and Anthropic's Message Batches both bill at 50% of the synchronous rate and finish within 24 hours, with results matched by custom_id because order is not preserved. Keep anything a user or a live process waits on synchronous, and handle expired and errored requests as a normal outcome.

Key takeaways:
- OpenAI's Batch API and Anthropic's Message Batches both bill at half the synchronous list rate and complete within a 24-hour window.
- The decision is about who waits. If no person or live process needs the answer for minutes to hours, batch is usually right.
- Results can come back in any order on both providers; always match them on custom_id.
- Expired, errored and canceled requests are part of normal operation. Write the resubmission path before the first large batch.
- Batch jobs submitted directly to a provider do not pass through a gateway, so keep them in their own provider project or workspace to attribute and reconcile them.

Most teams send every model call synchronously because that is how the first prototype worked. A good share of production LLM work does not need an answer in seconds: nightly summaries, backfills over stored documents, evaluation runs, classification of yesterday's tickets. The major providers publish a second way to send that work, at half the list rate, in exchange for giving them up to a day to do it.

This guide sets out what the trade is, which workloads should take it, and how to run a batch without losing requests.

## What the providers offer

**OpenAI's Batch API** processes a file of requests asynchronously at half the synchronous rate. Each batch completes within 24 hours, often sooner, and draws on a separate rate-limit pool that does not consume your standard per-model limits. One batch holds up to 50,000 requests in an input file of up to 200 MB. It works with Chat Completions, Responses, Embeddings, Completions, Moderations and Images[1].

**Anthropic's Message Batches API** takes up to 100,000 Messages requests or 256 MB per batch, whichever comes first, bills them at 50% of standard input and output, and says most batches finish in less than an hour. Results are available once every request has finished or after 24 hours, and are kept for 29 days[2].

**Google Gemini** lists a Batch tier on its pricing page: Gemini 3.8 Flash is $0.375 input and $1.875 output per 1M tokens on Batch, against $0.75 and $3.75 standard, through 2026-12-31[3].

The per-token effect is exact: GPT-6 Sol is $1.00 input and $5.00 output per 1M on Batch against $2.00 and $10.00 standard[4]; Claude Sonnet 5.5 is $1 and $5 against $2 and $10, and Claude Opus 5.5 $2 and $10 against $4 and $20[5].

[Diagram: Output list rates per 1M tokens, standard and batch, from each vendor's pricing page on 2026-09-29.]

Between synchronous and batch, OpenAI also offers **Flex processing**: set `service_tier` to `"flex"` on a normal request and it is priced at Batch API rates, with slower responses and occasional `429 Resource Unavailable` errors that are not charged. OpenAI suggests it for evaluations, data enrichment and asynchronous workloads, and raises the client timeout in its examples to 15 minutes[6].

[Diagram: OpenAI's Batch API and Anthropic's Message Batches side by side, as documented on 2026-09-29.]

## Which workloads should move

The deciding question is who waits for the answer.

- **A person is waiting.** Chat, autocomplete, a support agent mid-conversation. Stay synchronous. Lower the cost with [prompt caching](https://infro.io/blog/prompt-caching-break-even) and model choice, not with latency.
- **A live process is waiting, but minutes are acceptable.** A document pipeline that runs after upload, a report generated on request. A flex or low-priority tier fits, with a long timeout and a retry on unavailability.
- **Nothing is waiting until later today.** Evaluations, backfills, re-classifying a corpus after a taxonomy change, nightly digests, embedding a new data set. Batch.
- **The work is multi-step with tool calls.** An agent that calls a tool, reads the result and decides again cannot live inside one batch request. Run it synchronously, or restructure it so each step over many items is its own batch.

[Diagram: A decision on who waits for the answer. Only work nobody waits on for hours belongs in a batch.]

Two workload types benefit more than the headline rate suggests. Long reasoning jobs: Anthropic recommends batch processing for thinking budgets above 32k tokens, because such long-running requests can hit system timeouts and open-connection limits[7]. And jobs sharing a long prefix: Anthropic's caching multipliers stack with the batch rate, though cache hits inside a batch are best-effort, with typical hit rates of 30% to 98%[2].

## Running a batch without losing requests

The failure modes are not the synchronous ones. Nothing times out in front of a user; instead, some requests simply do not come back.

- **Order is not preserved.** On both providers, results may not match the input order; match on the `custom_id` you set on each request[1][2].
- **Expiry is normal.** OpenAI marks requests unfinished at 24 hours as expired and charges only for completed requests[1]. Anthropic returns each request as `succeeded`, `errored`, `canceled` or `expired`, and does not bill the last three[2].
- **Errors are per request.** A malformed request in the file fails on its own; the rest continue.

So every batch pipeline needs three things: a stable `custom_id` per unit of work, a store of which ids succeeded, and a resubmission step for the rest.

```python
import json
import time
from openai import OpenAI

client = OpenAI()  # a key for your own OpenAI project

def submit(items: list[dict]) -> str:
    with open("batch.jsonl", "w", encoding="utf-8") as f:
        for item in items:
            f.write(json.dumps({
                "custom_id": item["id"],
                "method": "POST",
                "url": "/v1/chat/completions",
                "body": {
                    "model": "gpt-6-sol",
                    "messages": [
                        {"role": "system", "content": "Classify the ticket. Reply with one label."},
                        {"role": "user", "content": item["text"]},
                    ],
                    "max_tokens": 20,
                },
            }) + "\n")
    upload = client.files.create(file=open("batch.jsonl", "rb"), purpose="batch")
    batch = client.batches.create(
        input_file_id=upload.id,
        endpoint="/v1/chat/completions",
        completion_window="24h",
    )
    return batch.id

def collect(batch_id: str) -> tuple[dict[str, str], set[str]]:
    while True:
        batch = client.batches.retrieve(batch_id)
        if batch.status in ("completed", "expired", "cancelled", "failed"):
            break
        time.sleep(60)
    answers: dict[str, str] = {}
    if batch.output_file_id:
        for line in client.files.content(batch.output_file_id).text.splitlines():
            row = json.loads(line)
            if row["response"] and row["response"]["status_code"] == 200:
                body = row["response"]["body"]
                answers[row["custom_id"]] = body["choices"][0]["message"]["content"]
    failed: set[str] = set()
    if batch.error_file_id:
        for line in client.files.content(batch.error_file_id).text.splitlines():
            failed.add(json.loads(line)["custom_id"])
    return answers, failed
```

Anything neither answered nor listed as failed — for example, requests lost to an expired batch — is found by comparing the ids you submitted with the ids you got back. Resubmit those in the next batch.

The Anthropic equivalent reports each request's outcome directly:

```python
import anthropic

client = anthropic.Anthropic()

def resubmit_ids(batch_id: str) -> list[str]:
    retry: list[str] = []
    for result in client.messages.batches.results(batch_id):
        kind = result.result.type
        if kind == "succeeded":
            store(result.custom_id, result.result.message)
        elif kind in ("errored", "expired", "canceled"):
            retry.append(result.custom_id)
    return retry

def store(custom_id: str, message) -> None:
    ...  # write to your own results table, keyed by custom_id
```

## Cost accounting for batch traffic

A batch job's cost appears on your provider's invoice like any other usage, at the batch rate. Anthropic's cost report marks it with `service_tier: batch` when grouped by description. The trouble is attribution: batch jobs usually run from scripts, not from the service that tags its requests, so they arrive on the invoice with no team, no feature and no workflow.

Give batch work its own provider project, workspace or API key, and record each batch's purpose, owner and item count when you submit it. That turns a line item nobody recognises into one that can be booked.

## Measuring the move

Before moving a workload, record three numbers on the synchronous path for a representative week: estimated provider cost, the number of units of work completed, and how long results took to be used downstream. After the move, measure the same three. The rate halves by definition; the questions worth answering are whether the unit count held (expired and errored requests that were never resubmitted quietly shrink it) and whether anything downstream was waiting on results it now gets hours later.

Watch for rate-limit side effects too. OpenAI's batch pool is separate from synchronous limits[1], so moving a large nightly job off the synchronous path can free headroom for user-facing traffic during the hours it used to run. That is often worth as much as the rate itself, and it only shows if you track throttling on the synchronous path before and after.

## A checklist before moving a workload

- [ ] Nobody waits on the answer for at least an hour, or a flex tier with a long timeout fits.
- [ ] Each unit of work has a stable `custom_id`.
- [ ] Results are stored by id, and a resubmission step handles expired and errored ids.
- [ ] Requests set `max_tokens`, so a runaway answer cannot inflate the batch.
- [ ] The batch runs under its own provider project, workspace or key, with an owner recorded.
- [ ] The result quality has been compared with the synchronous path on a sample; the model is the same, but prompts built for chat sometimes assume a follow-up turn that a batch will never have.

## Batch traffic and INFRO

INFRO's gateway serves synchronous requests; it has no batch endpoint, so a batch job goes from your code straight to your provider account with your own provider key and is billed there at the provider's batch rate. INFRO does not see it per request. Its asynchronous [jobs](https://infro.io/docs/api/jobs) are for media generation, not for batched text.

Where that shows up is [reconciliation](https://infro.io/docs/cost/reconciliation). Comparing INFRO's estimate with OpenAI's or Anthropic's reported cost, a batch job looks like reported spend with no matching requests; INFRO names batch jobs among the typical sources of traffic that bypasses it, and with usage access it lists the provider keys involved. Running batch work under its own provider key makes that finding immediately recognisable instead of an unexplained gap.

## Sources

1. Batch API — OpenAI. https://developers.openai.com/api/docs/guides/batch (accessed 2026-09-29)
2. Batch processing — Anthropic. https://platform.claude.com/docs/en/build-with-claude/batch-processing (accessed 2026-09-29)
3. Gemini Developer API pricing — Google. https://ai.google.dev/gemini-api/docs/pricing (accessed 2026-09-29)
4. Pricing — OpenAI. https://developers.openai.com/api/docs/pricing (accessed 2026-09-29)
5. Pricing — Anthropic. https://platform.claude.com/docs/en/about-claude/pricing (accessed 2026-09-29)
6. Flex processing — OpenAI. https://developers.openai.com/api/docs/guides/flex-processing (accessed 2026-09-29)
7. Extended thinking — Anthropic. https://platform.claude.com/docs/en/build-with-claude/extended-thinking (accessed 2026-09-29)