What you can and cannot know before you test
Two changes are on the table for a long-running Agents API session: editing the tool list while the session is live, and turning on the opt-in time-awareness setting part-way through. Both are worth testing for the same reason. Provider caches match on the exact token prefix of a request, and only the portion identical to a recent request, starting from the very beginning, can hit the cache 1. If a change alters bytes ahead of content that was being read from cache, that content stops being a read.
What the sources read for this article establish is narrower than the title. They establish the prefix rule, the usage fields that show cache behaviour, and that a tool-list update takes effect on the next model call for function tools and MCP servers, and by the next turn for built-in tools 2. They do not establish where an Agents API session places tool definitions relative to history, or what the time-awareness option adds to a request and where. OpenAI's pricing page and main site pages could not be read, and the changelog text that loaded was navigation only. So this article gives you a way to measure the effect, not a verdict.
Why a change early in the prefix is expensive
Order matters. INFRO's guidance for cache-friendly requests is system prompt first, then tool definitions, then append-only history, then the newest message, because tool definitions are part of the prompt prefix 3. Anything that changes the prefix invalidates the cache from that point on 1.
If tools sit ahead of a long history in the rendered context, a tool edit changes bytes before all of that history. The tokens that were read at the cached rate on the previous turn have to be processed again. On GPT-5.6 and later, a written token is billed at the write rate instead of the input rate: 1.25x for the 30-minute lifetime, 2.5x for 12 hours, with reads at 0.1x (0.05x on GPT-6.1 Sol and Terra) 4. So the turn after the change pays a write on tokens that would have been reads.
That reasoning is an inference from the prefix rule, not something observed in an Agents API session. If the session keeps tools in a different position, or caches history separately, the loss is smaller. The replay below tells you which.
Three variations to expect:
- Tool appended at the end of the tool list. Content before the new tool should still match. Whether history after it matches is the open question.
- Tool removed or reordered. Bytes change at the first difference, so everything after that point is affected.
- A description edited. Same as any other byte change; it is as costly as the position it sits at, not as small as the edit.
Time awareness: the same test with a different failure shape
The time-awareness option is a different risk because of how often it can change, not how much. A tool edit happens once. A clock-like value that is refreshed each turn would change every turn. OpenAI's caching guidance, as summarised in INFRO's article, says the cached prefix must be byte-identical and that timestamps, request ids and per-user data belong after the last breakpoint 4. If the option places a changing value inside the cached region, every turn pays a re-write; if it appends the value to the newest message, the cache is untouched.
The feature's documentation was not read for this article, so I cannot tell you which of those it does. Treat three questions as unknown until your replay answers them:
- Does it add tokens at all, and how many per turn?
- Do those tokens sit before or after the cached region?
- Does the added content change between turns, or only once when you enable it?
Question 3 decides the cost shape: a one-time change costs one re-write per session, and a per-turn change costs one on every turn after it is enabled.
The cost formula
For one change event at turn k, let P be the number of tokens from the first changed byte to the end of the cached prefix at that turn. Let p be your input rate per token, w the write multiplier and r the read multiplier. Without the change those tokens would have been read; with it they are written again:
extra = P × p × (w − r)
For a setting that changes the prefix every turn, sum that over each affected turn with that turn's P. If a request creates no cache write, as when you have set explicit mode and placed no breakpoint 4, the missed tokens bill at the plain input rate and w becomes 1. Use the rates on your own contract, not list rates, if you have negotiated them.
Note what the formula leaves out: output tokens, which are billed the same either way, and latency, which also rises on a cold prefix. It also assumes the new prefix is cached again afterwards. If the next turn comes after the entry's lifetime has lapsed, you would have paid the write anyway, and the change costs nothing extra. That is why the control group below matters.
The paired replay
Run this on a replay set, not on live traffic. Pick 30 to 100 representative sessions, weighted toward long ones. Pin the model snapshot, hold compaction settings fixed, and keep turn spacing inside the cache lifetime. Compaction rewrites accumulated context, so letting it vary would confuse the result 2.
- Control arm. Replay each session with no tool edit and without time awareness. Record per turn:
input_tokens,cached_tokens,cache_write_tokens, output tokens. - Tool-edit arm. Same sessions, with one change made at a chosen turn (for example the 10th): append a tool, then in a separate run remove a tool, then in another edit a description. Make each change in its own run.
- Time-awareness arm. Same sessions, with the option enabled at the same turn, and a second run with it enabled from turn one.
- Compare the same turn across arms. The effect shows as the turn after the change, where
cached_tokensfalls andcache_write_tokensrises relative to control. - Repeat on a second day and at a different time of day, since cache behaviour is not guaranteed on every request: a lifetime is a minimum for read pricing, not a promise of a hit 4.
The usage fields to read are the ones OpenAI's response usage separates: input_tokens_details.cached_tokens for billed reads (excluding tokens charged as writes) and cache_write_tokens for writes, with a split by lifetime 4. Do not infer anything from prompt token counts.
def turn_cost(u, p, w=1.25, r=0.10):
"""u: usage for one turn. p: input rate per token.
Assumes input_tokens includes cached and written tokens;
confirm that against your own responses first."""
cached = u["cached_tokens"]
written = u["cache_write_tokens"]
plain = u["input_tokens"] - cached - written
return plain * p + cached * p * r + written * p * w
def session_extra(treatment, control, p, **kw):
t = sum(turn_cost(u, p, **kw) for u in treatment)
c = sum(turn_cost(u, p, **kw) for u in control)
return t - c
def hit_rate(turns):
total = sum(u["input_tokens"] for u in turns)
return sum(u["cached_tokens"] for u in turns) / total if total else 0.0Report the median and a high percentile of session_extra per arm, and the hit rate per turn index so the dip is visible. Then reconcile the batch against the spend your provider reports for the same window. Any gap is cost your per-turn accounting is not attributing, a problem the compaction guide also warns about 2.
Reading the result and deciding
Turn the numbers into a decision with the extra cost per session next to the value of the change.
| Result in the replay | What it means | Action |
|---|---|---|
| Cached tokens unchanged after the edit | The edit lands outside the cached region | Enable; re-check after any model change |
| One-turn dip, then recovery | One re-write per change, as the formula predicts | Multiply by changes per session and session count |
| Dip on every turn after enabling | The setting changes the prefix each turn | Do not enable on long sessions until the content moves after the cached region, or price the sum over turns |
| Dip with a longer-than-lifetime gap in control too | Cold start, not the change | Subtract it; the control arm already contains it |
Two operational rules follow. Batch tool changes: ten edits spread over a session cost ten re-writes, one deployment that makes them together costs one. And make the change at a boundary where the prefix is short, such as session start, rather than turn 40 when P is largest.
Watching it after you ship
A replay predicts; production confirms. The same two fields should be on a dashboard by session and by turn index, with an alert when the hit rate for a cohort drops after a release. If your agent traffic passes through a gateway, INFRO records the estimated provider cost per request and can show a lost prompt cache as a traffic change with an estimated weekly cost change, using request metadata rather than answers. That is a check on a rollout, not a substitute for the paired replay, and it only sees requests that go through it. A failover to a different connection also resets the cache for a prefix 1, so separate failover turns from change turns when you read a dip.
