Cut your LLM API bill by 80% without rewriting your app

Most oversized LLM bills have the same root cause: every request in the product — the two-line summary, the yes/no classification, the 40-step agent run — goes to the same flagship model someone picked during the prototype. That was the right call in week one, when you had no evals, no traffic data, and quality was the only risk. Eighteen months later the default is still there, and you're paying frontier prices for work a $0.28-per-million-token model handles fine. The fix is not a rewrite. It's six changes, most of them config-level, applied in order — ending with a worked before/after model for a 500M-token-a-month product.

1. Measure per-feature token spend

You can't tier traffic you can't see. Most teams know their monthly invoice and roughly nothing else — not which feature burns the tokens, not the input/output split, not the p95 prompt size. Fix that before touching model choice.

Tag every LLM call with the feature that generated it and log four numbers: model, input tokens, output tokens, latency. A week of data is enough. If your gateway or provider exposes usage analytics, this is mostly free; if not, it's an afternoon of middleware.

You'll almost certainly find two things. First, 60–80% of tokens come from two or three commodity features — classification, extraction, summarization — that never needed a frontier model. Second, output tokens dominate the bill despite being a fraction of the volume, because output typically costs 4–8x input. That asymmetry drives most of what follows.

2. Right-size the model per task

Model choice is a per-task decision, not a per-product one. Split your traffic into three tiers:

• Cheap tier: classification, routing, extraction, autocomplete, title generation. High volume, low ambiguity, easy to eval.
• Mid tier: RAG answers, drafting, summarization with nuance, most tool-calling.
• Frontier tier: multi-step agents, hard code generation, anything where a quality failure costs you a customer.

The demotion test is simple: run a sample of a feature's real traffic through a model one tier down and compare outputs — with an eval script if you have one, by eye if you don't. If you can't reliably tell the difference, you're overpaying. The converse holds too: some workloads genuinely need frontier models, so leave them there. The goal is matching cost to difficulty, not minimizing cost everywhere.

In code, tiering is a dictionary, not an architecture change:

from openai import OpenAI

# One client. Only the base URL changed.
client = OpenAI(base_url="https://api.infro.io/v1", api_key=INFRO_KEY)

TIERS = {
    "triage":     "deepseek/deepseek-v3.2",     # ~$0.28 / $0.42 per 1M
    "rag_answer": "zai/glm-4.6",              # ~$0.60 / $2.20
    "agent_step": "moonshot/kimi-k2",         # ~$0.60 / $2.50
    "escalation": "anthropic/claude-opus-5",  # $5 / $25 — earn it
}

def complete(feature: str, messages):
    return client.chat.completions.create(
        model=TIERS[feature],
        messages=messages,
        metadata={"feature": feature},  # tags spend for step 1
    )

3. Swap commodity tasks to open-weight models

This is where the big money is. The open-weight frontier — DeepSeek, Qwen, Kimi, GLM, MiniMax — has closed most of the gap with Western flagships on the workloads that dominate production traffic, at 5–20x lower prices. Verify that on your own traffic rather than taking it from a blog post, but public leaderboards and our own testing both support it for classification, extraction, summarization, RAG answering, and a growing share of agentic work.

Prices are public list at the time of writing; see the model catalog for current numbers. Note the output-token gap: GPT-5.1 output costs roughly 24x DeepSeek V3.2 output. For a summarization feature that is nearly pure output spend, that ratio is your savings ceiling.

4. Prompt and context hygiene

Cheaper models cut the rate; hygiene cuts the volume. Four habits pay for themselves within a billing cycle:

• Trim conversation history. Keep a sliding window of recent turns plus a running summary, not the full transcript. A 30-turn chat resent on every request is a quadratic-ish token bill.
• Cap RAG context. The most common waste we see: pipelines shipping 20 retrieved chunks when a reranker would show the answer lives in the top 3–5.
• Cache your system prompt. Most providers discount cached input heavily — often 75–90% off. If your system prompt plus tool definitions run 3,000 tokens across a million requests a month, caching that prefix is real money for zero product change.
• Constrain output. Set max_tokens, ask for terse JSON instead of prose, and kill the "Certainly! Here is..." preamble with a one-line instruction. Output is your expensive direction; act like it.

5. Batch and go async where latency allows

A surprising share of LLM traffic has no human waiting on it: nightly digests, data enrichment, embedding backfills, eval runs, moderation sweeps. Move those to batch or async endpoints. Batch APIs typically run about 50% off interactive pricing, and queue-tolerant jobs let a router pick cheaper capacity that would be too slow for interactive use. Rule of thumb: if a job runs on a cron, it should not pay real-time prices.

6. Route through a gateway

The last step makes the first five cheap to keep doing. A gateway that speaks the OpenAI API puts every model behind one base URL, so "try the tier-down model" becomes a string change instead of a new SDK, a new billing account, and new failure modes. It also covers what you'd otherwise build yourself: automatic failover when a provider degrades, unified billing across a dozen vendors, and per-feature usage analytics — step 1, handled.

Routing helps the rate, too. INFRO sends each request to the most cost-efficient provider that's currently reliable for that model, and buys capacity at wholesale, so you generally pay below list price. Setup is the base-URL swap in the snippet above; the quickstart covers the rest, and the pricing section has per-model numbers.

A worked example: 500M tokens a month

The token mix below is modeled on a composite of workloads we've seen — roughly 500M tokens a month (410M in, 90M out), everything on flagship models by default. Your split will differ, but the shape is typical.

At list prices that's $1,750 down to $415 a month — a 76% cut, and steps 2–4 did almost all of it. Batch pricing on the digest job (roughly half off), prompt caching on the remaining Opus traffic, and below-list gateway pricing take the realistic landing zone to about $350, or roughly 80% off. No feature was cut, nothing was rewritten, and the hardest traffic still runs on a frontier model.

What to do this week

1. Add feature tags to your LLM calls and let a week of data accumulate.
2. Take your biggest commodity feature and eval it on DeepSeek V3.2 or GLM-4.6 against 200 real examples.
3. Trim your RAG context and cache your system prompt — the two fastest wins.
4. Move anything that runs on a cron to batch.

The short version: put a gateway in front of your traffic and the rest of this playbook becomes configuration. INFRO puts 120+ models behind one OpenAI-compatible endpoint, with smart routing, automatic failover, and pricing at or below list. The quickstart is a base-URL change — enough to run the step 2 evals this afternoon. Measure first, though; the bill you cut should be one you understand.