DocsObservability
Experiments
Replay a dataset across models, routes, prompt versions and settings through the gateway, judge every answer and compare, behind an estimate and a cap.
An experiment runs every item of a dataset through up to six variants and judges every answer with up to five evaluators. A variant is a model (a model, an alias or a route), an optional managed prompt version applied like the prompt extension (its messages first, the item's after, its variables filled from the item's metadata), and params (temperature, top_p, max_tokens, seed).
Estimate first
With dry_run: true the answer is the estimate: the request count (items × variants, plus one judge call per evaluator on every answer), the estimated provider cost per variant, per evaluator and in total, and how much of your monthly request allowance it uses. A model with no known rate is listed in unpriced and the total is null, never $0.
curl -X POST "https://api.infro.io/manage/v1/experiments" \
-H "Authorization: Bearer $INFRO_MANAGEMENT_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"dataset": "support-golden",
"variants": [
{"label": "current", "model": "anthropic/claude-sonnet-5", "prompt": {"slug": "support-reply", "version": 3}},
{"label": "candidate", "model": "anthropic/claude-sonnet-5", "prompt": {"slug": "support-reply", "version": 4}},
{"label": "cheaper", "model": "openai/gpt-5.4-mini", "prompt": {"slug": "support-reply", "version": 4}}
],
"evaluators": ["helpfulness"],
"dry_run": true
}'{
"object": "experiment_estimate",
"items": 200,
"requests": { "calls": 600, "judge_calls": 600, "total": 1200 },
"variants": [
{ "label": "current", "model": "anthropic/claude-sonnet-5", "calls": 200, "estimated_cost": 1.84 },
{ "label": "candidate", "model": "anthropic/claude-sonnet-5", "calls": 200, "estimated_cost": 1.91 },
{ "label": "cheaper", "model": "openai/gpt-5.4-mini", "calls": 200, "estimated_cost": 0.22 }
],
"evaluators": [{ "slug": "helpfulness", "model": "openai/gpt-5.4", "calls": 600, "estimated_cost": 2.1 }],
"estimated_cost": 6.07,
"unpriced": [],
"allowance": { "used": 41210, "included": 500000, "after": 42410 }
}To start it, send the same body without dry_run and with spend_cap in US dollars, at least the estimate (400 spend_cap_below_estimate otherwise). An experiment with an unpriced model cannot start (400 experiment_unpriced): its spend could not be held to the cap.
How it runs
- In the background, at most four calls at once, each a request through the gateway on your own providers: model access, guardrails, the environment you chose (
environment, defaultproduction) and your organization's and your own spend limits apply. It runs as the person who started it (a management token runs as the member who created it), metered withpurpose: experimentandexperiment_id, and counts against your request allowance. - It stops when its spend reaches the cap (
stop_reason: spend_cap; it takes only the calls that fit the room left, so it passes the cap by at most one call's estimation error), when a spend limit refuses a call (budget), or when a person stops it (person). - It resumes after a restart. A call that was in flight when the Worker stopped is closed as an error and never repeated, so it is never paid twice.
- Its answers are kept with the experiment (clipped to 20,000 characters) and deleted with it. Its judges' verdicts are kept with the experiment too, not written to your scores, so a test never moves a production figure.
Compare
GET /v1/experiments/{id} (or /manage/v1/…, or MCP get_experiment) returns summary, one entry per variant: each evaluator's mean score, pass rate for pass-or-fail judges and category counts, the error rate, p50 and p95 latency, and the estimated provider cost per item and in total (cost_is_partial when some calls were unpriced). GET …/results lists the items with every variant's answer and scores side by side. In the console, Promote moves a prompt label to the version a variant ran, with the same rules as moving it on the prompt's page.
| Endpoint | Access | What it does |
|---|---|---|
GET /v1/experiments, GET /v1/experiments/{id} | A key with history access | List and compare. |
GET /v1/experiments/{id}/results | History and payload access | Items side by side. |
POST /manage/v1/experiments | quality:write | Estimate (dry_run) or start; MCP start_experiment returns the estimate until called with a spend_cap. |
POST /manage/v1/experiments/{id}/stop | quality:write | Stop after the calls in flight. |