# Long context in production: cost, latency and context rot

URL: https://infro.io/blog/long-context-window-management
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-09-29
Last modified: 2026-09-29
Last reviewed: 2026-09-29 (next review due 2026-10-29)
Topic: Agents and multimodal systems
Format: Guide

> Treat the context window as a budget, not a bucket: count tokens before sending, reserve room for output, keep a stable prefix for caching, put the material that matters at the start or end, and compact old turns and tool results into notes before the history grows unbounded. Long prompts cost more on every turn, can cross a vendor's long-context price tier, and recall degrades as context grows — so a larger window is headroom, not a reason to send more.

Key takeaways:
- Everything in a request counts toward the window — system prompt, tools, history, tool results, documents and the output, including reasoning.
- A multi-turn conversation re-sends its history every turn, so carrying everything makes total input grow with the square of the turn count.
- Some vendors bill a whole request at a higher rate once the prompt crosses a long-context threshold. Know where yours are.
- Recall degrades as context grows and is weakest in the middle. Place what matters at the start or end, and send less.
- Budget the window in code: count, reserve output, trim, compact into notes, and refuse clearly when nothing else fits.

Context windows have grown from a few thousand tokens to a million on several current models, and the tempting conclusion is that context management is solved: send everything, let the model sort it out. It is not solved. It moved. The limit you hit is no longer the window. It is cost per turn, time to first token, and a model that quietly stops using what you sent.

## What counts toward the window

Anthropic's documentation is precise about it: the system prompt, every message — including tool results, images and documents — and your tool definitions all count, and so does the output the model generates for the turn, including extended thinking[1]. Cached prompt prefixes still occupy the window; caching changes what you pay for those tokens, not whether they count[1].

Two overflow behaviours matter for error handling. If the input alone exceeds the window, the API returns a `400 invalid_request_error` ("prompt is too long") on every Claude model. On Claude 4.5 models and newer, a request whose input plus `max_tokens` exceeds the window is accepted, and generation stops with `stop_reason: "model_context_window_exceeded"` if it reaches the limit[1]. The first is a request your code should never send; the second is an answer your code must recognise as incomplete.

Window sizes differ by model, and so do tokenizers. Anthropic lists a 1M-token context window for its current Fable, Mythos, Opus 5.x and Sonnet 5.x models, with up to 128k output tokens per request, and 200k for others such as Claude Sonnet 4.5[1]. It also notes that Claude 4.7 and later models use a newer tokenizer that produces roughly 30 percent more tokens for the same text, and tells readers to recount prompts against the model they plan to use[2]. A prompt budget measured on one model is not a budget on another.

## Why long context costs more than it looks

A conversation is stateless on the wire. Every turn re-sends the whole history, so a session that keeps everything pays for turn 1's text forty times by turn 40. Total input grows roughly with the square of the number of turns. OWASP lists repeatedly exceeding the context window among the forms of unbounded consumption, alongside runaway pay-per-use cost[3].

> **Example: Forty turns, carried or compacted**
> A support session has a 5,000-token fixed prefix (system prompt and tools), and each turn adds 3,000 tokens of user message, tool results and reply.
> 
> Carrying everything, turn n sends 5,000 + 3,000 × (n − 1) tokens. Over 40 turns that is 2,540,000 input tokens, and turn 40 alone sends 122,000.
> 
> Compacting whenever the history passes 30,000 tokens — replacing it with a 2,000-token summary — the 40 turns send 801,000 input tokens, and the three compaction calls add 97,000 input and 6,000 output tokens. No turn sends more than 35,000.
> 
> At illustrative rates of $2 per million input tokens and $10 per million output tokens, the carried session's input costs about $5.08; the compacted one about $1.86 including the summaries. The replies cost the same in both.

The second cost is price tiers. Some vendors charge a higher per-token rate once a single prompt crosses a length threshold, and the rate is chosen by the prompt's total length:

[Diagram: Long-context pricing rules as published on each vendor's own page on 2026-09-29. Check the page for the models you actually call.]

- Google prices Gemini 3.1 Pro input at $2.00 per 1M tokens for prompts up to 200k tokens and $4.00 above, and output at $12.00 and $18.00 respectively[4].
- OpenAI publishes a separate long-context rate column for its GPT-6 models, above the standard rates[5].
- Anthropic bills long-context requests on its 1M-window models at standard pricing, with no beta header needed[1].

A prompt that drifts from 190k to 210k tokens on a tiered model does not cost 10% more. It can cost roughly twice as much for input. If your workload sits near a threshold, put the threshold in your budget logic.

The third cost is time. The model has to process every input token before it produces the first output token, so time to first token grows with the prompt. Measure it for your own prompt sizes rather than assuming; see [latency optimization](https://infro.io/blog/llm-latency-optimization).

## Context rot: more tokens, worse use of them

Anthropic's documentation names the problem: as token count grows, accuracy and recall degrade, "a phenomenon known as *context rot*"[1]. Its engineering guidance describes models as having an "attention budget" and recommends finding "the smallest set of high-signal tokens" for the task[6].

The best-known study of the effect is "Lost in the Middle" (Liu et al., TACL). Across multi-document question answering and key-value retrieval, performance was often highest when the relevant information sat at the beginning or end of the input, and "significantly degrades" when the model had to use information in the middle — including for models built for long contexts[7].

Two practical rules follow:

- **Place what matters at the edges.** Instructions and the most relevant documents at the start; the current question and any must-follow constraints at the end. Background goes in the middle, if it goes anywhere.
- **Send less.** A retrieval step that selects five good passages usually beats a prompt that includes fifty mediocre ones, on cost and often on accuracy. See [where the tokens go in a RAG pipeline](https://infro.io/blog/rag-token-accounting).

## Budget the window in code

Treat the window as a budget with line items. Output is reserved first, because a response that runs out of room is a failed request. Then the stable prefix, then pinned notes, then the variable parts, with headroom for estimation error.

[Diagram: An illustrative plan for a 200,000-token window. The headroom absorbs tokenizer differences and estimation error; it is not spare capacity to fill.]

The budgeter below counts with the provider's token-counting endpoint, trims oldest turns first while keeping pinned messages, and compacts the trimmed turns into a summary note instead of discarding them. Anthropic's endpoint accepts the same inputs as a message — system prompt, tools, images, PDFs — returns an estimate, and is free to use within its own rate limits[2].

```python
from dataclasses import dataclass

import anthropic

client = anthropic.Anthropic()


@dataclass
class Budget:
    window: int = 200_000
    reserve_output: int = 16_000
    headroom: float = 0.10       # tokenizer drift and estimation error
    compact_above: int = 60_000  # history tokens that trigger compaction

    @property
    def input_limit(self) -> int:
        return int((self.window - self.reserve_output) * (1 - self.headroom))


def count(model: str, system: str, tools: list, messages: list) -> int:
    return client.messages.count_tokens(
        model=model, system=system, tools=tools, messages=messages
    ).input_tokens


def summarize(model: str, turns: list) -> str:
    response = client.messages.create(
        model=model,
        max_tokens=2_000,
        system="Summarize this conversation for your own later use. Keep decisions, "
               "facts the user supplied, open questions and ids. Drop pleasantries.",
        messages=turns + [{"role": "user", "content": "Write the summary now."}],
    )
    return "".join(block.text for block in response.content if block.type == "text")


def fit(model: str, system: str, tools: list, pinned: list, history: list,
        budget: Budget) -> list:
    """Return messages that fit the budget: pinned notes, a summary, recent turns."""
    history_tokens = count(model, system, [], history) if history else 0
    if history_tokens > budget.compact_above and len(history) > 6:
        # Compact everything but the last six messages into one note.
        # (Keep the cut on a user turn, and keep tool_use / tool_result pairs together.)
        old, recent = history[:-6], history[-6:]
        note = summarize(model, old)
        history = [{"role": "user", "content": f"Notes from earlier in this session:\n{note}"},
                   {"role": "assistant", "content": "Understood."}] + recent

    messages = pinned + history
    while count(model, system, tools, messages) > budget.input_limit:
        if len(history) <= 2:
            raise ValueError("request cannot fit the context budget; split the task")
        history = history[2:]  # drop the oldest exchange
        messages = pinned + history
    return messages
```

The code makes the policy explicit, which is its main value. A few rules it encodes are worth keeping whatever you write:

- **Compaction beats truncation.** Dropping old turns loses decisions the user already made; summarising them into a note keeps them at a fraction of the tokens.
- **Keep message structure valid.** A tool result without its tool call, or a history that starts with an assistant turn, is a provider error. Trim on exchange boundaries.
- **Refuse clearly.** When a single input cannot fit after trimming, return an error that tells the user to split the task, rather than silently cutting a document in half.
- **Count once per turn in production.** The example counts several times for clarity; in a hot path, count the new material and add it to the previous response's reported usage.

## Compaction, notes and fresh sessions

[Diagram: The loop for long-running sessions: count, compare, compact, send with a stable prefix. When compaction stops freeing room, start a fresh session from the notes.]

Provider features can do some of this for you. Anthropic offers server-side compaction, which summarises earlier parts of a conversation on the server so it can continue past the window limit (in beta for Claude 4.6 and later models), and context editing strategies that clear old tool results or thinking blocks[1]. Its engineering guidance adds three patterns beyond compaction: structured notes the agent writes outside the window and reads back, sub-agents that work in a clean window and return a condensed result, and just-in-time retrieval that loads data by reference when needed instead of up front[6].

Provider-side features are convenient and provider-specific. If you route one workload across several vendors, keep the budget logic in your own code so behaviour does not change when a request fails over to a different model.

## Keep the stable part stable

Compaction and caching pull in opposite directions if you are careless. Provider caches match on an exact prefix; OpenAI's guide requires "the entire rendered prefix to match" and advises stable instructions and shared reference material first[8]. Put system prompt, tools and pinned notes first and never rewrite them turn to turn; compact only the history after them. Rewriting the summary on every turn resets the cache from that point on, so compact in steps — when a threshold is crossed — not continuously.

## Long context in INFRO

INFRO routes requests to the provider accounts you connect and records each one with the token counts your provider reported — it does not re-tokenize — and an estimate of what your provider will bill for it ([chat completions](https://infro.io/docs/api/chat-completions)). Three behaviours matter for long prompts.

**Output reservation under budgets.** A blocking budget admits a request only if its worst-case estimated cost fits. When a chat request sets no `max_tokens`, the worst case assumes the model's maximum output — 32,768 tokens where the catalog lists no maximum — and INFRO sends that ceiling to the provider. Setting `max_tokens` to what the task needs reserves less, so more concurrent long-context requests fit under one budget ([spend controls](https://infro.io/docs/org/spend-controls)).

**Long-context premiums are not modelled per request.** The per-request estimate uses one rate pair per model, so a prompt past a vendor's long-context threshold is estimated at the standard rate. Reconciliation against your provider's reported cost itemises long-context pricing as an *unmodelled premium*, separately from the residual gap ([reconciliation](https://infro.io/docs/cost/reconciliation)). If most of a workload's prompts sit above a threshold, consider entering the rate you actually pay under Prices so its estimates and budgets use it.

**Growth is visible.** `prompt_tokens` is recorded on every request and exported in the request CSV, and tagging calls with `metadata.trace_id` and `step` lets `GET /v1/traces/{trace_id}` show a session's calls in order ([request traces](https://infro.io/docs/observability/traces)). A session whose prompt tokens climb steadily turn after turn is a session with no compaction.

## Checklist

- [ ] Every long-context call path counts tokens before sending, against the model it will actually use.
- [ ] Output room is reserved first; `max_tokens` is set to what the task needs.
- [ ] Responses that stop on a context limit (`model_context_window_exceeded`, `finish_reason: "length"`) are treated as incomplete.
- [ ] System prompt, tools and pinned notes form a stable prefix; compaction only touches the history after it.
- [ ] History is compacted into notes at a threshold, on exchange boundaries, not truncated mid-turn.
- [ ] The most relevant material is placed at the start or end of the prompt.
- [ ] Budget logic knows each vendor's long-context price threshold for the models in use.
- [ ] Prompt-token growth per session is charted, and sessions without compaction are found.

## Questions

### Is it better to use a 1M-token context window or retrieval?

For a large, changing corpus, retrieval usually wins on cost and often on accuracy, because each request carries only relevant passages. A long window is worth it when the task genuinely needs the whole artefact at once, such as reviewing one long contract or a codebase slice, and even then it pays to put the most relevant parts at the edges.

## Sources

1. Context windows — Anthropic. https://platform.claude.com/docs/en/build-with-claude/context-windows (accessed 2026-09-29)
2. Token counting — Anthropic. https://platform.claude.com/docs/en/build-with-claude/token-counting (accessed 2026-09-29)
3. LLM10:2025 Unbounded Consumption — OWASP GenAI Security Project. https://genai.owasp.org/llmrisk/llm102025-unbounded-consumption/ (accessed 2026-09-29)
4. Gemini Developer API pricing — Google. https://ai.google.dev/gemini-api/docs/pricing (accessed 2026-09-29)
5. API pricing — OpenAI. https://developers.openai.com/api/docs/pricing (accessed 2026-09-29)
6. Effective context engineering for AI agents — Anthropic. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents (accessed 2026-09-29)
7. Lost in the Middle: How Language Models Use Long Contexts — Liu et al., Transactions of the Association for Computational Linguistics. https://arxiv.org/abs/2307.03172 (accessed 2026-09-29)
8. Prompt caching — OpenAI. https://developers.openai.com/api/docs/guides/prompt-caching (accessed 2026-09-29)