Long context in production: cost, latency and context rot

Context windows have grown from a few thousand tokens to a million on several current models, and the tempting conclusion is that context management is solved: send everything, let the model sort it out. It is not solved. It moved. The limit you hit is no longer the window. It is cost per turn, time to first token, and a model that quietly stops using what you sent.

What counts toward the window

Anthropic's documentation is precise about it: the system prompt, every message — including tool results, images and documents — and your tool definitions all count, and so does the output the model generates for the turn, including extended thinking¹. Cached prompt prefixes still occupy the window; caching changes what you pay for those tokens, not whether they count¹.

Two overflow behaviours matter for error handling. If the input alone exceeds the window, the API returns a 400 invalid_request_error ("prompt is too long") on every Claude model. On Claude 4.5 models and newer, a request whose input plus max_tokens exceeds the window is accepted, and generation stops with stop_reason: "model_context_window_exceeded" if it reaches the limit¹. The first is a request your code should never send; the second is an answer your code must recognise as incomplete.

Window sizes differ by model, and so do tokenizers. Anthropic lists a 1M-token context window for its current Fable, Mythos, Opus 5.x and Sonnet 5.x models, with up to 128k output tokens per request, and 200k for others such as Claude Sonnet 4.5¹. It also notes that Claude 4.7 and later models use a newer tokenizer that produces roughly 30 percent more tokens for the same text, and tells readers to recount prompts against the model they plan to use². A prompt budget measured on one model is not a budget on another.

Why long context costs more than it looks

A conversation is stateless on the wire. Every turn re-sends the whole history, so a session that keeps everything pays for turn 1's text forty times by turn 40. Total input grows roughly with the square of the number of turns. OWASP lists repeatedly exceeding the context window among the forms of unbounded consumption, alongside runaway pay-per-use cost³.

The second cost is price tiers. Some vendors charge a higher per-token rate once a single prompt crosses a length threshold, and the rate is chosen by the prompt's total length:

• Google prices Gemini 3.1 Pro input at $2.00 per 1M tokens for prompts up to 200k tokens and $4.00 above, and output at $12.00 and $18.00 respectively⁴.
• OpenAI publishes a separate long-context rate column for its GPT-6 models, above the standard rates⁵.
• Anthropic bills long-context requests on its 1M-window models at standard pricing, with no beta header needed¹.

A prompt that drifts from 190k to 210k tokens on a tiered model does not cost 10% more. It can cost roughly twice as much for input. If your workload sits near a threshold, put the threshold in your budget logic.

The third cost is time. The model has to process every input token before it produces the first output token, so time to first token grows with the prompt. Measure it for your own prompt sizes rather than assuming; see latency optimization.

Context rot: more tokens, worse use of them

Anthropic's documentation names the problem: as token count grows, accuracy and recall degrade, "a phenomenon known as context rot"¹. Its engineering guidance describes models as having an "attention budget" and recommends finding "the smallest set of high-signal tokens" for the task⁶.

The best-known study of the effect is "Lost in the Middle" (Liu et al., TACL). Across multi-document question answering and key-value retrieval, performance was often highest when the relevant information sat at the beginning or end of the input, and "significantly degrades" when the model had to use information in the middle — including for models built for long contexts⁷.

Two practical rules follow:

• Place what matters at the edges. Instructions and the most relevant documents at the start; the current question and any must-follow constraints at the end. Background goes in the middle, if it goes anywhere.
• Send less. A retrieval step that selects five good passages usually beats a prompt that includes fifty mediocre ones, on cost and often on accuracy. See where the tokens go in a RAG pipeline.

Budget the window in code

Treat the window as a budget with line items. Output is reserved first, because a response that runs out of room is a failed request. Then the stable prefix, then pinned notes, then the variable parts, with headroom for estimation error.

The budgeter below counts with the provider's token-counting endpoint, trims oldest turns first while keeping pinned messages, and compacts the trimmed turns into a summary note instead of discarding them. Anthropic's endpoint accepts the same inputs as a message — system prompt, tools, images, PDFs — returns an estimate, and is free to use within its own rate limits².

from dataclasses import dataclass

import anthropic

client = anthropic.Anthropic()


@dataclass
class Budget:
    window: int = 200_000
    reserve_output: int = 16_000
    headroom: float = 0.10       # tokenizer drift and estimation error
    compact_above: int = 60_000  # history tokens that trigger compaction

    @property
    def input_limit(self) -> int:
        return int((self.window - self.reserve_output) * (1 - self.headroom))


def count(model: str, system: str, tools: list, messages: list) -> int:
    return client.messages.count_tokens(
        model=model, system=system, tools=tools, messages=messages
    ).input_tokens


def summarize(model: str, turns: list) -> str:
    response = client.messages.create(
        model=model,
        max_tokens=2_000,
        system="Summarize this conversation for your own later use. Keep decisions, "
               "facts the user supplied, open questions and ids. Drop pleasantries.",
        messages=turns + [{"role": "user", "content": "Write the summary now."}],
    )
    return "".join(block.text for block in response.content if block.type == "text")


def fit(model: str, system: str, tools: list, pinned: list, history: list,
        budget: Budget) -> list:
    """Return messages that fit the budget: pinned notes, a summary, recent turns."""
    history_tokens = count(model, system, [], history) if history else 0
    if history_tokens > budget.compact_above and len(history) > 6:
        # Compact everything but the last six messages into one note.
        # (Keep the cut on a user turn, and keep tool_use / tool_result pairs together.)
        old, recent = history[:-6], history[-6:]
        note = summarize(model, old)
        history = [{"role": "user", "content": f"Notes from earlier in this session:\n{note}"},
                   {"role": "assistant", "content": "Understood."}] + recent

    messages = pinned + history
    while count(model, system, tools, messages) > budget.input_limit:
        if len(history) <= 2:
            raise ValueError("request cannot fit the context budget; split the task")
        history = history[2:]  # drop the oldest exchange
        messages = pinned + history
    return messages

The code makes the policy explicit, which is its main value. A few rules it encodes are worth keeping whatever you write:

• Compaction beats truncation. Dropping old turns loses decisions the user already made; summarising them into a note keeps them at a fraction of the tokens.
• Keep message structure valid. A tool result without its tool call, or a history that starts with an assistant turn, is a provider error. Trim on exchange boundaries.
• Refuse clearly. When a single input cannot fit after trimming, return an error that tells the user to split the task, rather than silently cutting a document in half.
• Count once per turn in production. The example counts several times for clarity; in a hot path, count the new material and add it to the previous response's reported usage.

Compaction, notes and fresh sessions

Provider features can do some of this for you. Anthropic offers server-side compaction, which summarises earlier parts of a conversation on the server so it can continue past the window limit (in beta for Claude 4.6 and later models), and context editing strategies that clear old tool results or thinking blocks¹. Its engineering guidance adds three patterns beyond compaction: structured notes the agent writes outside the window and reads back, sub-agents that work in a clean window and return a condensed result, and just-in-time retrieval that loads data by reference when needed instead of up front⁶.

Provider-side features are convenient and provider-specific. If you route one workload across several vendors, keep the budget logic in your own code so behaviour does not change when a request fails over to a different model.

Keep the stable part stable

Compaction and caching pull in opposite directions if you are careless. Provider caches match on an exact prefix; OpenAI's guide requires "the entire rendered prefix to match" and advises stable instructions and shared reference material first⁸. Put system prompt, tools and pinned notes first and never rewrite them turn to turn; compact only the history after them. Rewriting the summary on every turn resets the cache from that point on, so compact in steps — when a threshold is crossed — not continuously.

Long context in INFRO

INFRO routes requests to the provider accounts you connect and records each one with the token counts your provider reported — it does not re-tokenize — and an estimate of what your provider will bill for it (chat completions). Three behaviours matter for long prompts.

Output reservation under budgets. A blocking budget admits a request only if its worst-case estimated cost fits. When a chat request sets no max_tokens, the worst case assumes the model's maximum output — 32,768 tokens where the catalog lists no maximum — and INFRO sends that ceiling to the provider. Setting max_tokens to what the task needs reserves less, so more concurrent long-context requests fit under one budget (spend controls).

Long-context premiums are not modelled per request. The per-request estimate uses one rate pair per model, so a prompt past a vendor's long-context threshold is estimated at the standard rate. Reconciliation against your provider's reported cost itemises long-context pricing as an unmodelled premium, separately from the residual gap (reconciliation). If most of a workload's prompts sit above a threshold, consider entering the rate you actually pay under Prices so its estimates and budgets use it.

Growth is visible. prompt_tokens is recorded on every request and exported in the request CSV, and tagging calls with metadata.trace_id and step lets GET /v1/traces/{trace_id} show a session's calls in order (request traces). A session whose prompt tokens climb steadily turn after turn is a session with no compaction.

Checklist

• Every long-context call path counts tokens before sending, against the model it will actually use.
• Output room is reserved first; max_tokens is set to what the task needs.
• Responses that stop on a context limit (model_context_window_exceeded, finish_reason: "length") are treated as incomplete.
• System prompt, tools and pinned notes form a stable prefix; compaction only touches the history after it.
• History is compacted into notes at a threshold, on exchange boundaries, not truncated mid-turn.
• The most relevant material is placed at the start or end of the prompt.
• Budget logic knows each vendor's long-context price threshold for the models in use.
• Prompt-token growth per session is charted, and sessions without compaction are found.