The FinOps Foundation defines a cost anomaly as spending that is different, usually higher, than normal historical or expected spend, and anomaly management as detecting, alerting on and managing those events quickly enough to limit the damage1. For LLM traffic the definition holds, but the causes are specific. Spend rises because something about the requests changed, and most of the time it is one of five things.
This analysis describes each pattern, the signal it leaves first, and a detection method that does not drown you in false alarms.
Why a fixed threshold fails
"Alert when daily spend exceeds $500" breaks in both directions. A project that normally spends $40 a day can triple and never cross it. A project that normally spends $480 crosses it every busy Monday. Weekends are quieter than weekdays, so a Saturday that looks like a Tuesday is itself unusual.
A better baseline is each scope's own usual day: the median of the same weekday over the last few weeks. A better yardstick is that scope's typical day-to-day variation, measured robustly. NIST's handbook defines the median absolute deviation as the median of each value's distance from the median, and notes it is even less affected by extremes in the tail than other measures of spread2. That matters here: with a standard deviation, last week's incident widens the band that should catch next week's.
from statistics import median
def usual_day(history: list[float]) -> float:
"""history: the same weekday's spend over the last four weeks."""
return median(history)
def spread(history: list[float], floor: float) -> float:
m = median(history)
mad = median(abs(x - m) for x in history) * 1.4826 # scaled to match a std. dev. on normal data
return max(mad, 0.10 * m, floor)
def score(projected_today: float, history: list[float], floor: float = 10.0) -> float:
return (projected_today - usual_day(history)) / spread(history, floor)
# A Tuesday projected at $412 against Tuesdays of $90, $96, $101 and $94:
print(round(score(412.0, [90.0, 96.0, 101.0, 94.0]), 1))Three guards keep the score useful: project today's spend to the end of the day instead of comparing a partial day with a whole one; do not score until enough of the day has passed for the projection to mean something; and ignore changes too small to matter in dollars, however large they are in percent.
Estimated provider cost, USD
- Wednesday$94
- Thursday$101
- Friday$88
- Saturday$41
- Sunday$38
- Monday$97
- Tuesday (projected)$412
- 40% · Warning at 4 spreads
- 80% · Critical at 8 spreads
- 100% · Scale shown to 10 spreads
1. A runaway loop or retry storm
An agent that never reaches its stop condition, a tool that keeps returning an error the model retries, or a client that retries on every timeout all produce the same shape: many more model calls per unit of work. Total request volume may not look extreme if only one workflow is affected, which is why this is easy to miss at the organization level.
First signal: calls per run for one workflow, or requests per minute on one key. Confirmation: many calls sharing one trace or run id. Fix: cap steps, tokens and spend per run, and give each key its own spend limit so one caller cannot consume a whole budget. See bounding agent loops.
2. Longer answers after a snapshot change
Providers regularly point a model alias at a new snapshot. Your code did not change, your prompts did not change, but answers get longer, and output tokens are the most expensive tokens on the request.
First signal: output tokens per request on one model rise. Confirmation: the model id the provider names in its response changed on the same day. Fix: pin a dated snapshot where the provider offers one, set max_tokens deliberately, and re-run your evaluation on the new snapshot before accepting it.
3. A lost prompt cache
A cache hit bills the prefix at a fraction of the input rate. A one-line edit near the top of a system prompt, a reordered tool list, or a timestamp added to the first message makes every prefix new, and the input bill jumps with no change in volume. On Anthropic even changing the thinking configuration between requests invalidates cache breakpoints3.
First signal: the cached share of prompt tokens on one model or route drops. Confirmation: a prompt or tool-definition change deployed that day. Fix: keep stable content first and volatile content last. The economics are in prompt caching break-even.
4. Silent fallback to a dearer model
Failover protects availability. When a primary starts timing out for a slice of requests, those requests quietly move to the next target in the chain. If that target is a larger model, cost per request rises while error rates stay flat.
First signal: the share of requests served by a fallback on one route rises. Confirmation: failed attempts on the primary recorded against the recovered requests. Fix: treat a rising fallback share as an incident on the primary, and put a same-model standby on another account ahead of a different model in the chain.
5. Spend your gateway never saw
The four patterns above are visible in your own request records. The fifth is not: traffic that reaches your provider account through another path. A key copied into a notebook, a service calling the provider directly, or a batch job submitted outside the gateway all bill the same account.
First signal: the provider's reported cost for a day exceeds your estimate for it, in the same direction day after day. Confirmation: usage grouped by API key shows a key your gateway never used. Both OpenAI's usage API and Anthropic's usage report can be grouped by API key45. Fix: rotate the key, move the caller behind the gateway, and give the provider account only the keys you route. See reconciling estimated spend with provider invoices.
| Anomaly | First signal | Scope that shows it | What confirms it |
|---|---|---|---|
| Runaway loop or retry storm | Requests per run climb | One workflow or key | Many calls under one trace id |
| Longer answers after a snapshot change | Output tokens per request up | One model on one connection | A new served model id |
| Lost prompt cache | Cached share of input falls | One model or route | A prompt or tool edit that day |
| Silent fallback to a dearer model | Fallback share up | One route | Failed attempts on the primary |
| Spend the gateway did not see | Reported cost above estimate | One provider account | Usage on a key the gateway never used |
Where to look first
When an alert fires on an organization or project, the useful next question is which model, key, workflow or end customer contributed most of the difference. An alert that names its top contributors turns a dashboard hunt into a single click. Document each resolved anomaly and its cause; the FinOps Foundation's framework makes documenting anomalies and their resolutions part of the process, not an afterthought1.
Triage in the first hour
When a spend alert fires, the order of questions matters more than the tooling. A short runbook keeps the first hour focused:
- Is it real? Check whether the projection rests on enough of the day and enough priced requests. A day with many unpriced requests is a lower bound, not an anomaly.
- Where is it? Narrow from organization to project, then to model, key, workflow and end customer. One contributor usually explains most of the difference.
- Which of the five? Compare that contributor's calls per run, output tokens per request, cached share and fallback share with its usual day. The metric that moved names the pattern.
- Is it still growing? A loop or a leaked key keeps spending while you investigate; a snapshot change or a lost cache costs at a steady higher rate. Contain the first kind immediately with a key limit or a budget; the second can wait for a proper fix.
- What changed? Deploys, prompt edits, provider announcements and routing changes on the same day. Most anomalies have a change behind them within a few hours of the first signal.
- Write it down. The cause, the signal that found it, the time to detection and what would have caught it sooner.
The last step is the one that improves detection. A pattern you have documented twice is a pattern worth a dedicated alert.
Detecting these in INFRO
INFRO's spend alerts score each scope — the organization, and each project, key, provider connection and model, plus workflows and end customers from the Team plan — against its usual day: the median of the same weekday over the last four weeks, with distance measured in the scope's median absolute deviation, never less than 10% of the usual day. A spend.anomaly is a warning at 4 spreads above the usual day and critical at 8. Changes smaller than the larger of $10 and 3% of the organization's average day are never reported, a scope is learning until it has 14 days of history, nothing is scored before a quarter of the UTC day has passed, and a day with more than 20% unpriced requests is not scored. When an organization or project fires, the finding names its three largest contributors.
Patterns 2, 3 and 4 are traffic changes: traffic.model_changed when a model on one connection is answered by a model id the provider had not reported in the previous 14 days, and traffic.regression when output tokens per request, the cached share of prompt tokens, the fallback share, length-limited answers, p95 latency or an error code's share move far from the usual day. Each finding states the estimated weekly change in provider cost. Pattern 5 needs reconciliation on the Business plan, which raises cost.drift when the estimate leans the same way by more than 5% on five of seven days and, with usage access, names traffic that bypassed INFRO.
Alerts describe spend; budgets enforce it. Anomaly detection for latency and error rates beyond these signals is on INFRO's roadmap, not available today.