# Do Agents API tool updates or time awareness break the cache?

URL: https://infro.io/blog/do-agents-api-tool-updates-or-time-awareness-break-the-cache
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-11-11
Last modified: 2026-11-11
Last reviewed: 2026-11-11 (next review due 2026-12-11)
Topic: Cost and FinOps
Format: Methodology

> Neither change can be ruled safe from the documentation read for this article. Provider caches match an exact prefix [^infro-prompt-caching-docs], so any change that alters bytes ahead of cached content should cost one re-write per change, and a per-turn change costs one on every turn. The size is measurable with a paired replay and a three-term formula.

Key takeaways:
- A provider prompt cache matches an exact prefix, so a change that alters bytes ahead of cached content forces those tokens to be written again.
- The extra input cost of one change is roughly the affected prefix tokens times the input rate times (write multiplier minus read multiplier).
- A change made once costs one re-write per session; a setting that alters the prefix every turn costs a re-write on every turn.
- Measure with a paired replay: same sessions with and without the change, comparing cached_tokens and cache_write_tokens per turn.
- The Agents API documentation for time awareness was not read for this article, so treat where it puts its content as something your test must establish.

## What you can and cannot know before you test

Two changes are on the table for a long-running Agents API session: editing the tool list while the session is live, and turning on the opt-in time-awareness setting part-way through. Both are worth testing for the same reason. Provider caches match on the exact token prefix of a request, and only the portion identical to a recent request, starting from the very beginning, can hit the cache [1]. If a change alters bytes ahead of content that was being read from cache, that content stops being a read.

What the sources read for this article establish is narrower than the title. They establish the prefix rule, the usage fields that show cache behaviour, and that a tool-list update takes effect on the next model call for function tools and MCP servers, and by the next turn for built-in tools [2]. They do not establish where an Agents API session places tool definitions relative to history, or what the time-awareness option adds to a request and where. OpenAI's pricing page and main site pages could not be read, and the changelog text that loaded was navigation only. So this article gives you a way to measure the effect, not a verdict.

## Why a change early in the prefix is expensive

Order matters. INFRO's guidance for cache-friendly requests is system prompt first, then tool definitions, then append-only history, then the newest message, because tool definitions are part of the prompt prefix [3]. Anything that changes the prefix invalidates the cache from that point on [1].

If tools sit ahead of a long history in the rendered context, a tool edit changes bytes before all of that history. The tokens that were read at the cached rate on the previous turn have to be processed again. On GPT-5.6 and later, a written token is billed at the write rate instead of the input rate: 1.25x for the 30-minute lifetime, 2.5x for 12 hours, with reads at 0.1x (0.05x on GPT-6.1 Sol and Terra) [4]. So the turn after the change pays a write on tokens that would have been reads.

That reasoning is an inference from the prefix rule, not something observed in an Agents API session. If the session keeps tools in a different position, or caches history separately, the loss is smaller. The replay below tells you which.

Three variations to expect:

- **Tool appended at the end of the tool list.** Content before the new tool should still match. Whether history after it matches is the open question.
- **Tool removed or reordered.** Bytes change at the first difference, so everything after that point is affected.
- **A description edited.** Same as any other byte change; it is as costly as the position it sits at, not as small as the edit.

## Time awareness: the same test with a different failure shape

The time-awareness option is a different risk because of how often it can change, not how much. A tool edit happens once. A clock-like value that is refreshed each turn would change every turn. OpenAI's caching guidance, as summarised in INFRO's article, says the cached prefix must be byte-identical and that timestamps, request ids and per-user data belong after the last breakpoint [4]. If the option places a changing value inside the cached region, every turn pays a re-write; if it appends the value to the newest message, the cache is untouched.

The feature's documentation was not read for this article, so I cannot tell you which of those it does. Treat three questions as unknown until your replay answers them:

1. Does it add tokens at all, and how many per turn?
2. Do those tokens sit before or after the cached region?
3. Does the added content change between turns, or only once when you enable it?

Question 3 decides the cost shape: a one-time change costs one re-write per session, and a per-turn change costs one on every turn after it is enabled.

## The cost formula

For one change event at turn *k*, let *P* be the number of tokens from the first changed byte to the end of the cached prefix at that turn. Let *p* be your input rate per token, *w* the write multiplier and *r* the read multiplier. Without the change those tokens would have been read; with it they are written again:

`extra = P × p × (w − r)`

For a setting that changes the prefix every turn, sum that over each affected turn with that turn's *P*. If a request creates no cache write, as when you have set explicit mode and placed no breakpoint [4], the missed tokens bill at the plain input rate and *w* becomes 1. Use the rates on your own contract, not list rates, if you have negotiated them.

> **Example: Illustrative: one tool edit on a long session**
> These figures are invented to show the arithmetic. Suppose the prefix at the turn after the edit is 60,000 tokens, your input rate is $2.00 per million tokens, writes are 1.25x and reads 0.1x.
> 
> - Normal turn, all cached: 60,000 × $2.00/M × 0.1 = $0.012.
> - Turn after a tool edit that invalidates everything: 60,000 × $2.00/M × 1.25 = $0.150.
> - Extra: $0.138, which equals 60,000 × $2.00/M × (1.25 − 0.1).
> 
> If only the first 8,000 tokens survive, *P* is 52,000 and the extra is $0.120. At 10,000 sessions a month with one edit each, the first case is about $1,380 a month. Your prefix sizes, rates and session counts will differ.

Note what the formula leaves out: output tokens, which are billed the same either way, and latency, which also rises on a cold prefix. It also assumes the new prefix is cached again afterwards. If the next turn comes after the entry's lifetime has lapsed, you would have paid the write anyway, and the change costs nothing extra. That is why the control group below matters.

## The paired replay

Run this on a replay set, not on live traffic. Pick 30 to 100 representative sessions, weighted toward long ones. Pin the model snapshot, hold compaction settings fixed, and keep turn spacing inside the cache lifetime. Compaction rewrites accumulated context, so letting it vary would confuse the result [2].

1. **Control arm.** Replay each session with no tool edit and without time awareness. Record per turn: `input_tokens`, `cached_tokens`, `cache_write_tokens`, output tokens.
2. **Tool-edit arm.** Same sessions, with one change made at a chosen turn (for example the 10th): append a tool, then in a separate run remove a tool, then in another edit a description. Make each change in its own run.
3. **Time-awareness arm.** Same sessions, with the option enabled at the same turn, and a second run with it enabled from turn one.
4. **Compare the same turn across arms.** The effect shows as the turn after the change, where `cached_tokens` falls and `cache_write_tokens` rises relative to control.
5. **Repeat on a second day** and at a different time of day, since cache behaviour is not guaranteed on every request: a lifetime is a minimum for read pricing, not a promise of a hit [4].

The usage fields to read are the ones OpenAI's response usage separates: `input_tokens_details.cached_tokens` for billed reads (excluding tokens charged as writes) and `cache_write_tokens` for writes, with a split by lifetime [4]. Do not infer anything from prompt token counts.

```python
def turn_cost(u, p, w=1.25, r=0.10):
    """u: usage for one turn. p: input rate per token.
    Assumes input_tokens includes cached and written tokens;
    confirm that against your own responses first."""
    cached = u["cached_tokens"]
    written = u["cache_write_tokens"]
    plain = u["input_tokens"] - cached - written
    return plain * p + cached * p * r + written * p * w

def session_extra(treatment, control, p, **kw):
    t = sum(turn_cost(u, p, **kw) for u in treatment)
    c = sum(turn_cost(u, p, **kw) for u in control)
    return t - c

def hit_rate(turns):
    total = sum(u["input_tokens"] for u in turns)
    return sum(u["cached_tokens"] for u in turns) / total if total else 0.0
```

Report the median and a high percentile of `session_extra` per arm, and the hit rate per turn index so the dip is visible. Then reconcile the batch against the spend your provider reports for the same window. Any gap is cost your per-turn accounting is not attributing, a problem the compaction guide also warns about [2].

## Reading the result and deciding

Turn the numbers into a decision with the extra cost per session next to the value of the change.

| Result in the replay | What it means | Action |
| --- | --- | --- |
| Cached tokens unchanged after the edit | The edit lands outside the cached region | Enable; re-check after any model change |
| One-turn dip, then recovery | One re-write per change, as the formula predicts | Multiply by changes per session and session count |
| Dip on every turn after enabling | The setting changes the prefix each turn | Do not enable on long sessions until the content moves after the cached region, or price the sum over turns |
| Dip with a longer-than-lifetime gap in control too | Cold start, not the change | Subtract it; the control arm already contains it |

Table: how to read the treatment arm against control.

Two operational rules follow. Batch tool changes: ten edits spread over a session cost ten re-writes, one deployment that makes them together costs one. And make the change at a boundary where the prefix is short, such as session start, rather than turn 40 when *P* is largest.

## Watching it after you ship

A replay predicts; production confirms. The same two fields should be on a dashboard by session and by turn index, with an alert when the hit rate for a cohort drops after a release. If your agent traffic passes through a gateway, INFRO records the estimated provider cost per request and can show a lost prompt cache as a traffic change with an estimated weekly cost change, using request metadata rather than answers. That is a check on a rollout, not a substitute for the paired replay, and it only sees requests that go through it. A failover to a different connection also resets the cache for a prefix [1], so separate failover turns from change turns when you read a dip.

## Questions

### Does adding a tool at the end of the list keep the earlier prefix cached?

By the prefix rule, tokens before the first changed byte can still match, so a tool appended after existing ones should leave the system prompt and earlier tools intact [1]. Whether everything after the tool list (the history) still hits depends on how the session assembles its context, which the paired replay in this article checks.

### Can I tell from the bill whether a change broke the cache?

Not reliably. Read cached_tokens and cache_write_tokens in each response's usage rather than judging from prompt token counts [4]. A drop in cached tokens with a matching rise in written tokens on the turn after the change is the signature.

## Sources

1. Prompt caching — Features — INFRO Docs. https://infro.io/docs/features/prompt-caching (accessed 2026-11-11)
2. Agents API background compaction and auto review: what to re-measure — INFRO. https://infro.io/blog/agents-api-background-compaction-and-auto-review-what-to-re-measure (accessed 2026-11-11)
3. Tool calling — Features — INFRO Docs. https://infro.io/docs/features/tool-calling (accessed 2026-11-11)
4. OpenAI's 12-hour prompt cache: when the 2.5x write pays off — INFRO. https://infro.io/blog/openai-12-hour-prompt-cache (accessed 2026-11-11)
