Guides
Running LLM APIs in production
Each guide solves one problem completely without INFRO first, with code in TypeScript and Python that is run against the provider's documented responses on every build. A short section at the end shows the same thing through INFRO.
- How to handle OpenAI 429 rate-limit errors in productionAn OpenAI
429means one of six things, and only two of them, a rate limit andslow_down, are fixed by waiting: readerror.code, retry those afterretry-afteror with exponential backoff and jitter, and stop at once on the four billing and quota codes.Updated Oct 24, 2026 · facts checked Oct 24, 2026 - Anthropic 529 "overloaded" errors: what they mean and how to fail overA
529 overloaded_errormeans Anthropic's API is temporarily overloaded across all users, not that you hit your own limit: retry once after a short, jittered wait, then send the request to another provider or a tested model.Updated Oct 24, 2026 · facts checked Oct 24, 2026 - What to do when the OpenAI API is down: a runbookConfirm it from your own error codes first, because a status page updates after your users notice, and separate a real outage (5xx,
503 server_is_overloaded, timeouts) from a 429 that is your own spend or quota limit.Updated Oct 24, 2026 · facts checked Oct 24, 2026 - How to set a hard spending limit on the OpenAI API (per project, key or user)OpenAI's only hard spending limit is monthly, per organization or per project: open Project settings → Limits → Edit spend limit, turn on Enforce a hard limit, and requests over it fail with
429 project_spend_limit_exceeded.Updated Oct 24, 2026 · facts checked Oct 24, 2026 - How to track LLM cost per customer in a SaaS appRecord every model call with your own customer id, the model you asked for and the provider's token counts, price it at your own rates (cached input at the cached rate, reasoning tokens as output), and store "no rate" as unpriced rather than $0.Updated Oct 24, 2026 · facts checked Oct 24, 2026
- LLM failover patterns: retry vs fallback vs circuit breakerRetry the same provider only for a brief, transient error and only inside the caller's deadline; fall back to another provider or a tested model when the provider stays unavailable or rate limited; and put a circuit breaker in front of each provider so one that keeps failing stops receiving traffic until a probe shows it has recovered.Updated Oct 24, 2026 · facts checked Oct 24, 2026
- What is a BYOK AI gateway (and when you don't need one)A BYOK AI gateway sits between your applications and AI providers and runs every request on provider accounts you own, with keys you give it, so your providers bill you directly at your own rates while the gateway adds routing, failover, budgets and cost tracking.Updated Oct 24, 2026 · facts checked Oct 24, 2026