Use this as a launch review, not a reading exercise: each item should end with a link to the evidence — an evaluation run, a load-test result, a policy, a dashboard — or an explicit decision that it does not apply. Items are grouped by the team that usually owns them.
Quality and product behavior
A demo proves the happy path. Production needs to know how often the model is wrong, what happens when it is, and whether a release made it worse.
- A versioned evaluation set drawn from real and difficult traffic, with acceptance thresholds agreed before the run.
- A regression run for every prompt, model or retrieval change, blocking release when a threshold is missed.
- Output contracts (JSON schemas, tool arguments) validated on every response, with a defined behaviour on failure.
- Grounding and citation checks where answers depend on sources.
- A human escalation path for high-impact decisions, and product copy that communicates uncertainty.
Reliability and rollback
Provider quotas, timeouts and streaming interruptions are ordinary events at production volume. Decide the behaviour for each before it happens.
- Client and request deadlines set explicitly, not inherited from an SDK default.
- Retryable errors classified; deterministic request errors are not retried.
- Same-model failover to another provider account tested before any cross-model fallback is allowed.
- Failure injection run for a timeout, a 429, a 5xx and malformed output, with the observed behaviour recorded.
- A configuration-level rollback for model, prompt and routing changes.
- A named incident owner and communication channel.
Security and data
Model access is a credential and a data path at the same time. Both need an owner.
- Every key has an owner, an environment, a model allowlist and a spend ceiling.
- Provider credentials live in one place; applications do not hold raw provider keys.
- A written decision on prompt and output retention, including who may read captured payloads.
- DPA and subprocessor list reviewed where your customers require it.
- Irreversible agent actions require explicit authority, separate from permission to call a model.
Observability and economics
You should be able to answer "what happened to this request" and "where did this week's spend go" without exporting anything.
- Every request traceable by id to its model, provider, attempts, latency, token counts and estimated cost.
- Alerts on error rate, fallback rate, malformed output and latency, each with an owner.
- A cost-per-successful-task baseline for each important workload.
- Soft budget alerts and hard ceilings scoped narrowly enough that one workload cannot starve another.
- Spend attributed by project or feature, reviewed weekly with the model and prompt changes that moved it.