Skip to content

DocsObservability

Experiments

Replay a dataset across models, routes, prompt versions and settings through the gateway, judge every answer and compare, behind an estimate and a cap.


An experiment runs every item of a dataset through up to six variants and judges every answer with up to five evaluators. A variant is a model (a model, an alias or a route), an optional managed prompt version applied like the prompt extension (its messages first, the item's after, its variables filled from the item's metadata), and params (temperature, top_p, max_tokens, seed).

Estimate first

With dry_run: true the answer is the estimate: the request count (items × variants, plus one judge call per evaluator on every answer), the estimated provider cost per variant, per evaluator and in total, and how much of your monthly request allowance it uses. A model with no known rate is listed in unpriced and the total is null, never $0.

Estimate a three-variant experiment
curl -X POST "https://api.infro.io/manage/v1/experiments" \
  -H "Authorization: Bearer $INFRO_MANAGEMENT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "dataset": "support-golden",
    "variants": [
      {"label": "current", "model": "anthropic/claude-sonnet-5", "prompt": {"slug": "support-reply", "version": 3}},
      {"label": "candidate", "model": "anthropic/claude-sonnet-5", "prompt": {"slug": "support-reply", "version": 4}},
      {"label": "cheaper", "model": "openai/gpt-5.4-mini", "prompt": {"slug": "support-reply", "version": 4}}
    ],
    "evaluators": ["helpfulness"],
    "dry_run": true
  }'
The estimate
{
  "object": "experiment_estimate",
  "items": 200,
  "requests": { "calls": 600, "judge_calls": 600, "total": 1200 },
  "variants": [
    { "label": "current", "model": "anthropic/claude-sonnet-5", "calls": 200, "estimated_cost": 1.84 },
    { "label": "candidate", "model": "anthropic/claude-sonnet-5", "calls": 200, "estimated_cost": 1.91 },
    { "label": "cheaper", "model": "openai/gpt-5.4-mini", "calls": 200, "estimated_cost": 0.22 }
  ],
  "evaluators": [{ "slug": "helpfulness", "model": "openai/gpt-5.4", "calls": 600, "estimated_cost": 2.1 }],
  "estimated_cost": 6.07,
  "unpriced": [],
  "allowance": { "used": 41210, "included": 500000, "after": 42410 }
}

To start it, send the same body without dry_run and with spend_cap in US dollars, at least the estimate (400 spend_cap_below_estimate otherwise). An experiment with an unpriced model cannot start (400 experiment_unpriced): its spend could not be held to the cap.

How it runs

  • In the background, at most four calls at once, each a request through the gateway on your own providers: model access, guardrails, the environment you chose (environment, default production) and your organization's and your own spend limits apply. It runs as the person who started it (a management token runs as the member who created it), metered with purpose: experiment and experiment_id, and counts against your request allowance.
  • It stops when its spend reaches the cap (stop_reason: spend_cap; it takes only the calls that fit the room left, so it passes the cap by at most one call's estimation error), when a spend limit refuses a call (budget), or when a person stops it (person).
  • It resumes after a restart. A call that was in flight when the Worker stopped is closed as an error and never repeated, so it is never paid twice.
  • Its answers are kept with the experiment (clipped to 20,000 characters) and deleted with it. Its judges' verdicts are kept with the experiment too, not written to your scores, so a test never moves a production figure.

Compare

GET /v1/experiments/{id} (or /manage/v1/…, or MCP get_experiment) returns summary, one entry per variant: each evaluator's mean score, pass rate for pass-or-fail judges and category counts, the error rate, p50 and p95 latency, and the estimated provider cost per item and in total (cost_is_partial when some calls were unpriced). GET …/results lists the items with every variant's answer and scores side by side. In the console, Promote moves a prompt label to the version a variant ran, with the same rules as moving it on the prompt's page.

EndpointAccessWhat it does
GET /v1/experiments, GET /v1/experiments/{id}A key with history accessList and compare.
GET /v1/experiments/{id}/resultsHistory and payload accessItems side by side.
POST /manage/v1/experimentsquality:writeEstimate (dry_run) or start; MCP start_experiment returns the estimate until called with a spend_cap.
POST /manage/v1/experiments/{id}/stopquality:writeStop after the calls in flight.
↑ ↓ navigate↵ open/ anywhere in docs