Skip to content

Cost and FinOpsMethodology

Do Agents API tool updates or time awareness break the cache?

Prompt caches match on an exact prefix, so anything that changes bytes early in a long session's context can turn cheap reads into full-price writes. This is a replay method to measure that for a tool-list edit or an opt-in time-awareness setting, and to price it per session before you switch either on.

By INFRO Editorial

Published

8 min read

Glowing token streams being compressed into a precise efficient AI infrastructure pipeline

In short

Neither change can be ruled safe from the documentation read for this article. Provider caches match an exact prefix 1, so any change that alters bytes ahead of cached content should cost one re-write per change, and a per-turn change costs one on every turn. The size is measurable with a paired replay and a three-term formula.

Key takeaways

  • A provider prompt cache matches an exact prefix, so a change that alters bytes ahead of cached content forces those tokens to be written again.
  • The extra input cost of one change is roughly the affected prefix tokens times the input rate times (write multiplier minus read multiplier).
  • A change made once costs one re-write per session; a setting that alters the prefix every turn costs a re-write on every turn.
  • Measure with a paired replay: same sessions with and without the change, comparing cached_tokens and cache_write_tokens per turn.
  • The Agents API documentation for time awareness was not read for this article, so treat where it puts its content as something your test must establish.

What you can and cannot know before you test

Two changes are on the table for a long-running Agents API session: editing the tool list while the session is live, and turning on the opt-in time-awareness setting part-way through. Both are worth testing for the same reason. Provider caches match on the exact token prefix of a request, and only the portion identical to a recent request, starting from the very beginning, can hit the cache 1. If a change alters bytes ahead of content that was being read from cache, that content stops being a read.

What the sources read for this article establish is narrower than the title. They establish the prefix rule, the usage fields that show cache behaviour, and that a tool-list update takes effect on the next model call for function tools and MCP servers, and by the next turn for built-in tools 2. They do not establish where an Agents API session places tool definitions relative to history, or what the time-awareness option adds to a request and where. OpenAI's pricing page and main site pages could not be read, and the changelog text that loaded was navigation only. So this article gives you a way to measure the effect, not a verdict.

Why a change early in the prefix is expensive

Order matters. INFRO's guidance for cache-friendly requests is system prompt first, then tool definitions, then append-only history, then the newest message, because tool definitions are part of the prompt prefix 3. Anything that changes the prefix invalidates the cache from that point on 1.

If tools sit ahead of a long history in the rendered context, a tool edit changes bytes before all of that history. The tokens that were read at the cached rate on the previous turn have to be processed again. On GPT-5.6 and later, a written token is billed at the write rate instead of the input rate: 1.25x for the 30-minute lifetime, 2.5x for 12 hours, with reads at 0.1x (0.05x on GPT-6.1 Sol and Terra) 4. So the turn after the change pays a write on tokens that would have been reads.

That reasoning is an inference from the prefix rule, not something observed in an Agents API session. If the session keeps tools in a different position, or caches history separately, the loss is smaller. The replay below tells you which.

Three variations to expect:

  • Tool appended at the end of the tool list. Content before the new tool should still match. Whether history after it matches is the open question.
  • Tool removed or reordered. Bytes change at the first difference, so everything after that point is affected.
  • A description edited. Same as any other byte change; it is as costly as the position it sits at, not as small as the edit.

Time awareness: the same test with a different failure shape

The time-awareness option is a different risk because of how often it can change, not how much. A tool edit happens once. A clock-like value that is refreshed each turn would change every turn. OpenAI's caching guidance, as summarised in INFRO's article, says the cached prefix must be byte-identical and that timestamps, request ids and per-user data belong after the last breakpoint 4. If the option places a changing value inside the cached region, every turn pays a re-write; if it appends the value to the newest message, the cache is untouched.

The feature's documentation was not read for this article, so I cannot tell you which of those it does. Treat three questions as unknown until your replay answers them:

  1. Does it add tokens at all, and how many per turn?
  2. Do those tokens sit before or after the cached region?
  3. Does the added content change between turns, or only once when you enable it?

Question 3 decides the cost shape: a one-time change costs one re-write per session, and a per-turn change costs one on every turn after it is enabled.

The cost formula

For one change event at turn k, let P be the number of tokens from the first changed byte to the end of the cached prefix at that turn. Let p be your input rate per token, w the write multiplier and r the read multiplier. Without the change those tokens would have been read; with it they are written again:

extra = P × p × (w − r)

For a setting that changes the prefix every turn, sum that over each affected turn with that turn's P. If a request creates no cache write, as when you have set explicit mode and placed no breakpoint 4, the missed tokens bill at the plain input rate and w becomes 1. Use the rates on your own contract, not list rates, if you have negotiated them.

Note what the formula leaves out: output tokens, which are billed the same either way, and latency, which also rises on a cold prefix. It also assumes the new prefix is cached again afterwards. If the next turn comes after the entry's lifetime has lapsed, you would have paid the write anyway, and the change costs nothing extra. That is why the control group below matters.

The paired replay

Run this on a replay set, not on live traffic. Pick 30 to 100 representative sessions, weighted toward long ones. Pin the model snapshot, hold compaction settings fixed, and keep turn spacing inside the cache lifetime. Compaction rewrites accumulated context, so letting it vary would confuse the result 2.

  1. Control arm. Replay each session with no tool edit and without time awareness. Record per turn: input_tokens, cached_tokens, cache_write_tokens, output tokens.
  2. Tool-edit arm. Same sessions, with one change made at a chosen turn (for example the 10th): append a tool, then in a separate run remove a tool, then in another edit a description. Make each change in its own run.
  3. Time-awareness arm. Same sessions, with the option enabled at the same turn, and a second run with it enabled from turn one.
  4. Compare the same turn across arms. The effect shows as the turn after the change, where cached_tokens falls and cache_write_tokens rises relative to control.
  5. Repeat on a second day and at a different time of day, since cache behaviour is not guaranteed on every request: a lifetime is a minimum for read pricing, not a promise of a hit 4.

The usage fields to read are the ones OpenAI's response usage separates: input_tokens_details.cached_tokens for billed reads (excluding tokens charged as writes) and cache_write_tokens for writes, with a split by lifetime 4. Do not infer anything from prompt token counts.

extra_cost.py
def turn_cost(u, p, w=1.25, r=0.10):
    """u: usage for one turn. p: input rate per token.
    Assumes input_tokens includes cached and written tokens;
    confirm that against your own responses first."""
    cached = u["cached_tokens"]
    written = u["cache_write_tokens"]
    plain = u["input_tokens"] - cached - written
    return plain * p + cached * p * r + written * p * w

def session_extra(treatment, control, p, **kw):
    t = sum(turn_cost(u, p, **kw) for u in treatment)
    c = sum(turn_cost(u, p, **kw) for u in control)
    return t - c

def hit_rate(turns):
    total = sum(u["input_tokens"] for u in turns)
    return sum(u["cached_tokens"] for u in turns) / total if total else 0.0

Report the median and a high percentile of session_extra per arm, and the hit rate per turn index so the dip is visible. Then reconcile the batch against the spend your provider reports for the same window. Any gap is cost your per-turn accounting is not attributing, a problem the compaction guide also warns about 2.

Reading the result and deciding

Turn the numbers into a decision with the extra cost per session next to the value of the change.

Result in the replayWhat it meansAction
Cached tokens unchanged after the editThe edit lands outside the cached regionEnable; re-check after any model change
One-turn dip, then recoveryOne re-write per change, as the formula predictsMultiply by changes per session and session count
Dip on every turn after enablingThe setting changes the prefix each turnDo not enable on long sessions until the content moves after the cached region, or price the sum over turns
Dip with a longer-than-lifetime gap in control tooCold start, not the changeSubtract it; the control arm already contains it
how to read the treatment arm against control.

Two operational rules follow. Batch tool changes: ten edits spread over a session cost ten re-writes, one deployment that makes them together costs one. And make the change at a boundary where the prefix is short, such as session start, rather than turn 40 when P is largest.

Watching it after you ship

A replay predicts; production confirms. The same two fields should be on a dashboard by session and by turn index, with an alert when the hit rate for a cohort drops after a release. If your agent traffic passes through a gateway, INFRO records the estimated provider cost per request and can show a lost prompt cache as a traffic change with an estimated weekly cost change, using request metadata rather than answers. That is a check on a rollout, not a substitute for the paired replay, and it only sees requests that go through it. A failover to a different connection also resets the cache for a prefix 1, so separate failover turns from change turns when you read a dip.

Questions readers ask

Does adding a tool at the end of the list keep the earlier prefix cached?
By the prefix rule, tokens before the first changed byte can still match, so a tool appended after existing ones should leave the system prompt and earlier tools intact 1. Whether everything after the tool list (the history) still hits depends on how the session assembles its context, which the paired replay in this article checks.
Can I tell from the bill whether a change broke the cache?
Not reliably. Read cached_tokens and cache_write_tokens in each response's usage rather than judging from prompt token counts 4. A drop in cached tokens with a matching rise in written tokens on the turn after the change is the signature.

From the team that builds it

Where this shows up in INFRO

INFRO is a control plane in front of the provider accounts you already hold. The capabilities below are the parts of it this article touches, each with the documentation that defines its behaviour.

  • Usage and cost per request

    Available with a plan

    Every request records its model, latency, token or unit counts, and estimated provider cost, queryable by project and key.

    Read the documentation for Usage and cost per request
  • Traffic changes

    Available with a plan

    Changes in your traffic that come with no error: a model answered by a new snapshot, longer answers, a lost prompt cache, more fallbacks, answers cut off by the length limit, slower responses or a new error mix. Each finding names the likely cause and the estimated weekly change in provider cost. INFRO sees request metadata, not answers, so it does not judge answer quality.

    Read the documentation for Traffic changes
  • OpenTelemetry export

    Available with a plan

    Spans pushed to your own collector from the request path, with the endpoint and headers set in the console. Included on every plan.

    Read the documentation for OpenTelemetry export

INFRO accounts are self-serve. Provider usage is billed to you by your providers; INFRO charges a software subscription and takes 0% of model spend. See plans.

Sources

Numbered in order of first citation. Each was opened and checked against the claim on the date shown.

  1. 1.Prompt caching — Features (opens in a new tab) — INFRO Docs. Accessed Nov 11, 2026.
  2. 2.Agents API background compaction and auto review: what to re-measure (opens in a new tab) — INFRO. Accessed Nov 11, 2026.
  3. 3.Tool calling — Features (opens in a new tab) — INFRO Docs. Accessed Nov 11, 2026.
  4. 4.OpenAI's 12-hour prompt cache: when the 2.5x write pays off (opens in a new tab) — INFRO. Accessed Nov 11, 2026.

About this article

Read on 2026-11-11: INFRO's prompt caching and tool calling docs and two INFRO articles that summarize OpenAI's prompt caching guide and changelog entries. OpenAI's pricing page and its main site pages returned HTTP 403, and the Agents API changelog text that loaded showed only navigation, so the time-awareness feature's mechanics, billing and prefix placement are unconfirmed and left out; no OpenAI rate is quoted except as an illustrative figure.

Last checked against its sources on . The next review is due by . No separate named reviewer is recorded for this article.

How articles are written, sourced and corrected is set out in the editorial standards.