The INFRO blog
Field guides for production AI
How to route, observe, pay for and govern AI traffic across the providers you already use — written by the team building INFRO. Every article shows its sources, when it was published, when it last changed and when it is next reviewed.
95 articles · 6 topics · Last updated · Editorial standards
Featured
Architecture & migrationCornerstone guide
What is an LLM gateway? A production engineer's guide
A working engineer's walkthrough of the LLM gateway pattern: the problems it absorbs, what happens to a request inside one, what it should never do, and how to test one before it sits in your request path.
GuideUpdated 10 min read
Cost & FinOps
Reconcile your AI bill: estimate vs. provider invoice
Your gateway's cost figure is a model of a price list, not a measurement of a charge. The two are rarely compared, and the gap is where an unmodelled rate hides for a quarter.
AnalysisUpdated 16 min read
Routing & reliability
LLM routing strategies for reliable production AI
Routing decides which model and which provider account serve each request. This guide separates the two decisions, walks through five strategies and their failure modes, and shows how to test a policy before it carries production traffic.
GuideUpdated 10 min read
Routing & reliability
Automatic LLM failover: a production implementation guide
Failover is more than retrying a 500. This guide covers which failures to move, how many times, where streaming makes recovery impossible, when a different model is acceptable, and what operators need to see.
GuideUpdated 9 min read
Topics
- Routing and reliabilityRouting policy, failover, retries, latency and incident response for AI traffic that has to keep working when a provider does not.14 articles
- Cost and FinOpsEstimating, attributing, forecasting and controlling provider spend: unit economics, budgets, caching and the practice of AI FinOps.22 articles
- Security and governanceCredentials, data paths, model access policy, audit and the evidence a security or procurement review actually asks for.13 articles
- Evaluation and modelsChoosing models with evidence: evaluation harnesses, cost per successful task, and what changes when you host or switch a model family.14 articles
- Architecture and migrationWhere a gateway sits, what it should and should not abstract, build-versus-buy, and moving an existing application without a rewrite.18 articles
- Agents and multimodal systemsThe infrastructure under agent loops, retrieval pipelines and image, video and audio workloads, where one request becomes many.14 articles
Start here: operating LLMs in production
Five guides in the order the problems usually arrive: what the layer is, how requests choose a provider, what happens when one fails, how to see what happened, and what it all costs.
- 1What is an LLM gateway? A production engineer's guide10 min read
- 2LLM routing strategies for reliable production AI10 min read
- 3Automatic LLM failover: a production implementation guide9 min read
- 4LLM observability: what to measure in production11 min read
- 5The production LLM cost optimization playbook11 min read
Latest
Agents & multimodal
AI image and video production for agencies
Generative tools multiply inside an agency faster than anyone can track them. This guide covers whose provider contract a job runs on, how to price a job before it runs, how to compare takes across models, how review should work, and what to do when a vendor retires the model a campaign was built on.
Guide11 min read
Cost & FinOps
AI gateway pricing models: markup, fees, requests, subscription
Two gateways with similar features can charge on entirely different things: what you buy through them, what your models cost, how many requests you send, or nothing at all. The base of the fee decides how it grows.
Comparison10 min read
Architecture & migration
AI model deprecation calendar: OpenAI, Anthropic, Google
One dated table of the model retirements OpenAI, Anthropic and Google have announced between now and mid-2027, with each vendor's listed replacement and a link to the page it came from, plus the provider-hosted copies that retire on a different day. Reviewed monthly.
Guide9 min read
Cost & FinOps
Does an AI spend cap hold under 500 concurrent requests?
A spend cap that checks a running total and then serves the request will be passed by a burst. Here is why, how a reservation-based cap is built instead, and what a sandbox test of one did and did not show.
Methodology12 min read
Cost & FinOps
The AI spend report leadership actually reads
Most AI cost dashboards are built for the engineer who configured the gateway. A CFO or CTO needs four answers on one page, once a week, with the reason behind each. Here is what to put in the report, which period to use, who gets which alert, and how to give finance budget rights without engineering access.
Guide12 min read
Architecture & migration
Classifiers in the LLM request path: route, guard, monitor
A classifier in front of a model call can pick the model, refuse an injection or order the context; one behind it can score the answer. Each one adds a call that can be slow, wrong or down. Here is how to place them, budget them and decide what happens when they fail.
Guide11 min read
All articles
Newest first. Each topic page lists the same articles by subject.
- AI image and video production for agenciesAgents and multimodal systemsGuide · 11 min
- AI gateway pricing models: markup, fees, requests, subscriptionCost and FinOpsComparison · 10 min
- AI model deprecation calendar: OpenAI, Anthropic, GoogleArchitecture and migrationGuide · 9 min
- Does an AI spend cap hold under 500 concurrent requests?Cost and FinOpsMethodology · 12 min
- The AI spend report leadership actually readsCost and FinOpsGuide · 12 min
- Classifiers in the LLM request path: route, guard, monitorArchitecture and migrationGuide · 11 min
- Decision models compared: OpenAI, Jev, Clef and moreEvaluation and modelsComparison · 12 min
- Estimate LLM cost before the request, not after the billCost and FinOpsGuide · 12 min
- AI gateway keys: environments, limits, rotation, off switchSecurity and governanceGuide · 12 min
- INFRO Studio: image, video and audio on your own providersAgents and multimodal systemsGuide · 12 min
- LLM guardrails at the gateway: checks before and afterSecurity and governanceGuide · 13 min
- LLM response caching: when exact-match pays, when it's wrongCost and FinOpsGuide · 13 min
- LLM routing rules: if-rules, failure rules, key spreadingRouting and reliabilityGuide · 13 min
- LLM spend limits compared: who caps spend on your own keysCost and FinOpsComparison · 12 min
- Manage your AI gateway from Claude Code or Cursor via MCPArchitecture and migrationGuide · 10 min
- MCP gateway: one governed entry point for agent toolsAgents and multimodal systemsGuide · 12 min
- OpenAI's October 23 shutdowns: find every call before it failsArchitecture and migrationGuide · 11 min
- September 3, 2026: provider incidents and failoverRouting and reliabilityAnalysis · 11 min
- Who owns your AI gateway? The 2026 acquisitionsArchitecture and migrationAnalysis · 12 min
- AI updates, September–October 2026: models, prices, shutdownsEvaluation and modelsAnalysis · 8 min
- How Jev cuts the cost of AI decisions: the arithmeticCost and FinOpsAnalysis · 9 min
- Jev for LLM guardrails and model routingSecurity and governanceGuide · 7 min
- Jev use cases: where typed AI decisions fit in a companyArchitecture and migrationGuide · 8 min
- What is Jev? TypeSafe's System One model, explainedEvaluation and modelsGuide · 11 min
- Image models compared: GPT Image 2, Nano Banana, FLUX.2Agents and multimodal systemsComparison · 7 min
- Showback and chargeback for AI spendCost and FinOpsGuide · 7 min
- What security reviewers ask about AI gatewaysSecurity and governanceChecklist · 7 min
- AI video models compared: Veo 3.1, Kling, Wan and Sora's exitAgents and multimodal systemsComparison · 8 min
- Claude Fable 5.1, Opus 5.5, Sonnet 5.5, Haiku 4.5 comparedEvaluation and modelsComparison · 8 min
- Async media jobs: webhooks, idempotency and retriesAgents and multimodal systemsGuide · 9 min
- Batch or synchronous: which LLM workloads should moveCost and FinOpsGuide · 7 min
- Bounding AI agent loops: step, time, token and spend limitsAgents and multimodal systemsGuide · 10 min
- BYOK gateway architecture: sealing, scoping and rotating keysSecurity and governanceGuide · 9 min
- Claude Fable 5.1 vs GPT-6 Astra: rates, caching and fitEvaluation and modelsComparison · 7 min
- Coding models compared: Opus 5.5, Kimi, Qwen3-Coder, MiniMaxEvaluation and modelsComparison · 8 min
- Cost per successful task: the LLM metric that mattersCost and FinOpsGuide · 7 min
- Frontier models compared: GPT-6, Fable 5.1, Gemini, GrokEvaluation and modelsComparison · 8 min
- GPT-6 Astra, Sol or Luna: choosing a tier per taskEvaluation and modelsComparison · 7 min
- Circuit breakers for LLM providers: throttling is not failureRouting and reliabilityAnalysis · 9 min
- Failover drills: test LLM provider failure before it happensRouting and reliabilityMethodology · 10 min
- A threat model for an LLM gateway, mapped to OWASPSecurity and governanceAnalysis · 8 min
- A prompt and payload logging policy for LLM trafficSecurity and governanceChecklist · 7 min
- Provider rate limits compared: headers, tiers and 429sRouting and reliabilityComparison · 10 min
- Five LLM spend anomalies and the signal each leavesCost and FinOpsAnalysis · 8 min
- Long context in production: cost, latency and context rotAgents and multimodal systemsGuide · 9 min
- MCP and tool calling in production: failure modes and fixesAgents and multimodal systemsGuide · 10 min
- Measuring time to first token and setting timeoutsRouting and reliabilityMethodology · 9 min
- From direct OpenAI calls to a gateway: a canary runbookArchitecture and migrationGuide · 9 min
- When a provider retires a model: a migration playbookEvaluation and modelsGuide · 8 min
- Azure OpenAI and Bedrock behind one gateway: ids and quotasArchitecture and migrationGuide · 8 min
- Multi-region AI traffic: latency, quotas and data locationArchitecture and migrationGuide · 8 min
- Negotiated rates vs list price: why AI cost estimates driftCost and FinOpsGuide · 8 min
- Same open-weight model, different hosts: a consistency testEvaluation and modelsMethodology · 8 min
- Prompt caching across providers: when it paysCost and FinOpsAnalysis · 7 min
- Prompt injection in production: defense in depthSecurity and governanceGuide · 8 min
- Least privilege for provider admin and usage keysSecurity and governanceGuide · 7 min
- Where the tokens go in a RAG pipelineAgents and multimodal systemsAnalysis · 8 min
- Reasoning tokens: the hidden cost of thinking modelsCost and FinOpsGuide · 7 min
- Reconciling estimated LLM spend with provider invoicesCost and FinOpsGuide · 8 min
- Self-hosted models behind an OpenAI-compatible endpointArchitecture and migrationGuide · 8 min
- Fast, low-cost models compared: Gemini Flash, GPT-6 LunaEvaluation and modelsComparison · 7 min
- Streaming LLM failures: interrupted streams and recoveryRouting and reliabilityGuide · 9 min
- Structured outputs across providers: JSON schema reliabilityEvaluation and modelsGuide · 9 min
- Tracing an AI agent run across many model callsAgents and multimodal systemsGuide · 9 min
- Attributing AI cost to customers, features and teamsCost and FinOpsGuide · 13 min
- Reconcile your AI bill: estimate vs. provider invoiceCost and FinOpsAnalysis · 16 min
- Tokens are not the unit you are billed inCost and FinOpsAnalysis · 15 min
- LLM routing strategies for reliable production AIRouting and reliabilityGuide · 10 min
- Automatic LLM failover: a production implementation guideRouting and reliabilityGuide · 9 min
- How to migrate to an OpenAI-compatible multi-model APIArchitecture and migrationBrief · 2 min
- LLM observability: what to measure in productionRouting and reliabilityGuide · 11 min
- The production LLM cost optimization playbookCost and FinOpsGuide · 11 min
- LLM FinOps: control AI spend without slowing product teamsCost and FinOpsBrief · 2 min
- How to evaluate LLMs on real production workloadsEvaluation and modelsGuide · 9 min
- The AI agent infrastructure stack for production systemsAgents and multimodal systemsBrief · 2 min
- Production RAG infrastructure: beyond the vector databaseAgents and multimodal systemsBrief · 2 min
- One multimodal AI API for text, image, video, and audioAgents and multimodal systemsBrief · 2 min
- AI infrastructure for startups: what to build and what to buyArchitecture and migrationBrief · 2 min
- What is an LLM gateway? A production engineer's guideArchitecture and migrationGuide · 10 min
- Enterprise AI gateway checklist: security, control, and scaleSecurity and governanceChecklist · 2 min
- Build vs buy an LLM gateway: the real engineering tradeoffArchitecture and migrationBrief · 1 min
- How to avoid LLM vendor lock-in without limiting your productArchitecture and migrationBrief · 1 min
- LLM latency optimization: improve time to first useful answerRouting and reliabilityBrief · 1 min
- How to secure LLM API keys in productionSecurity and governanceBrief · 1 min
- LLM spend controls: budgets, alerts, and hard limitsCost and FinOpsBrief · 1 min
- Chinese open-weight models in production: hosting and complianceEvaluation and modelsGuide · 9 min
- The production LLM checklist: controls to verify before launchRouting and reliabilityChecklist · 2 min
- Model access governance for fast-moving AI teamsSecurity and governanceBrief · 1 min
- Multi-model API architecture: design one stable AI boundaryArchitecture and migrationBrief · 1 min
- LLM incident response: diagnose failures across providersRouting and reliabilityBrief · 1 min
- Quality-aware LLM routing: optimize cost without blind downgradesRouting and reliabilityBrief · 1 min
- AI API pricing explained: tokens, media units, and real costCost and FinOpsBrief · 1 min
- LLM gateway security architecture: one enforceable boundarySecurity and governanceBrief · 1 min
- From one model to a multi-model AI platform: a staged planArchitecture and migrationBrief · 1 min
- OpenRouter alternatives in 2026: how AI gateway models differArchitecture and migrationComparison · 8 min