# Experiments

> Replay a dataset across models, routes, prompt versions and settings through the gateway, judge every answer and compare, behind an estimate and a cap.

Source: https://infro.io/docs/observability/experiments

An experiment runs every item of a [dataset](https://infro.io/docs/observability/datasets) through up to six variants and judges every answer with up to five [evaluators](https://infro.io/docs/observability/evaluators). A variant is a `model` (a model, an alias or a route), an optional managed `prompt` version applied like the `prompt` extension (its messages first, the item's after, its variables filled from the item's `metadata`), and `params` (`temperature`, `top_p`, `max_tokens`, `seed`).

## Estimate first

With `dry_run: true` the answer is the estimate: the request count (items × variants, plus one judge call per evaluator on every answer), the estimated provider cost per variant, per evaluator and in total, and how much of your monthly request allowance it uses. A model with no known rate is listed in `unpriced` and the total is `null`, never $0.

```bash
curl -X POST "https://api.infro.io/manage/v1/experiments" \
  -H "Authorization: Bearer $INFRO_MANAGEMENT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "dataset": "support-golden",
    "variants": [
      {"label": "current", "model": "anthropic/claude-sonnet-5", "prompt": {"slug": "support-reply", "version": 3}},
      {"label": "candidate", "model": "anthropic/claude-sonnet-5", "prompt": {"slug": "support-reply", "version": 4}},
      {"label": "cheaper", "model": "openai/gpt-5.4-mini", "prompt": {"slug": "support-reply", "version": 4}}
    ],
    "evaluators": ["helpfulness"],
    "dry_run": true
  }'
```

```json
{
  "object": "experiment_estimate",
  "items": 200,
  "requests": { "calls": 600, "judge_calls": 600, "total": 1200 },
  "variants": [
    { "label": "current", "model": "anthropic/claude-sonnet-5", "calls": 200, "estimated_cost": 1.84 },
    { "label": "candidate", "model": "anthropic/claude-sonnet-5", "calls": 200, "estimated_cost": 1.91 },
    { "label": "cheaper", "model": "openai/gpt-5.4-mini", "calls": 200, "estimated_cost": 0.22 }
  ],
  "evaluators": [{ "slug": "helpfulness", "model": "openai/gpt-5.4", "calls": 600, "estimated_cost": 2.1 }],
  "estimated_cost": 6.07,
  "unpriced": [],
  "allowance": { "used": 41210, "included": 500000, "after": 42410 }
}
```

To start it, send the same body without `dry_run` and with `spend_cap` in US dollars, at least the estimate (`400` `spend_cap_below_estimate` otherwise). An experiment with an unpriced model cannot start (`400` `experiment_unpriced`): its spend could not be held to the cap.

## How it runs

- In the background, at most four calls at once, each a request through the gateway on your own providers: model access, guardrails, the environment you chose (`environment`, default `production`) and your organization's and your own spend limits apply. It runs as the person who started it (a management token runs as the member who created it), metered with `purpose: experiment` and `experiment_id`, and counts against your request allowance.
- It stops when its spend reaches the cap (`stop_reason: spend_cap`; it takes only the calls that fit the room left, so it passes the cap by at most one call's estimation error), when a spend limit refuses a call (`budget`), or when a person stops it (`person`).
- It resumes after a restart. A call that was in flight when the Worker stopped is closed as an error and never repeated, so it is never paid twice.
- Its answers are kept with the experiment (clipped to 20,000 characters) and deleted with it. Its judges' verdicts are kept with the experiment too, not written to your scores, so a test never moves a production figure.

## Compare

`GET /v1/experiments/{id}` (or `/manage/v1/…`, or MCP `get_experiment`) returns `summary`, one entry per variant: each evaluator's mean score, pass rate for pass-or-fail judges and category counts, the error rate, p50 and p95 latency, and the estimated provider cost per item and in total (`cost_is_partial` when some calls were unpriced). `GET …/results` lists the items with every variant's answer and scores side by side. In the console, Promote moves a prompt label to the version a variant ran, with the same rules as moving it on the prompt's page.

| Endpoint | Access | What it does |
| --- | --- | --- |
| `GET /v1/experiments`, `GET /v1/experiments/{id}` | A key with history access | List and compare. |
| `GET /v1/experiments/{id}/results` | History and payload access | Items side by side. |
| `POST /manage/v1/experiments` | `quality:write` | Estimate (`dry_run`) or start; MCP `start_experiment` returns the estimate until called with a `spend_cap`. |
| `POST /manage/v1/experiments/{id}/stop` | `quality:write` | Stop after the calls in flight. |
