# Evaluators

> LLM-as-judge on any chat model, on your own provider account: score a sample of live traffic or past requests by your rubric, with a monthly spend limit.

Source: https://infro.io/docs/observability/evaluators

An evaluator is a rubric that a chat model, the judge, applies to answers. The judge runs on one of your own provider connections, through the gateway, and each verdict is written as a [score](https://infro.io/docs/observability/scores) named after the evaluator on the request it judged, so it appears in the request log, on [runs](https://infro.io/docs/observability/runs) and in a prompt version's figures beside your own scores and your end users' feedback.

## The rubric

A rubric is text with placeholders: `{{input}}` is the last user message of the judged request, `{{output}}` its answer, `{{expected}}` a dataset item's expected output (in experiments), and `{{metadata.<key>}}` a value from the request's `metadata`. A rubric must include `{{output}}`. Each value is clipped to 12,000 characters and filled in once, so text in a request can never add a placeholder. Instead of text, an evaluator can use a [managed prompt](https://infro.io/docs/features/prompts) at a label as its rubric, so the judge has versions and labels of its own.

`output` says what the judge returns: `{"type": "numeric"}` (a score from 0 to 1), `{"type": "boolean"}` (pass or fail, stored as 1 or 0) or `{"type": "category", "values": ["resolved", "escalated", "wrong"]}`. INFRO adds an instruction to answer with one JSON object, `{"score": …, "reason": "…"}`, asks for `response_format: json_object` at temperature 0 with at most 400 output tokens, and reads the answer strictly. The reason is kept as the score's comment under the same rule as other comments: only while your organization's request logging is on.

## Live traffic

`sampling.rate` above 0 judges that share of served chat completions (including `/v1/messages` and `/v1/responses`), filtered by `projects`, `models` (the model that served the request), `prompt` (`{"slug": …, "version": …}`), `workflows` and `environments`. An evaluator limited to a project judges that project's requests only. Judging starts after the response has been sent and is never waited for: it never slows, changes or fails the judged request. A response served from INFRO's response cache, and a console connection test, are not judged.

The judge call runs as the key that made the judged request: its model access, guardrails, environment and spend limits apply, and a blocking budget refuses it. Live sampling needs `monthly_spend_limit`, in US dollars of estimated provider cost per calendar month (UTC). When the evaluator's judge calls reach it, live judging pauses until the next month and the evaluator records when (`this_month.limit_reached_at`); judge calls already in flight still finish.

```bash
curl -X POST "https://api.infro.io/manage/v1/evaluators" \
  -H "Authorization: Bearer $INFRO_MANAGEMENT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "slug": "helpfulness",
    "model": "openai/gpt-5.4",
    "rubric": "Does the answer address what was asked, accurately and completely?\n\nQuestion: {{input}}\n\nAnswer: {{output}}",
    "output": {"type": "numeric"},
    "sampling": {"rate": 0.05, "models": ["anthropic/claude-sonnet-5"]},
    "monthly_spend_limit": 25
  }'
```

## Run on past requests

`POST /manage/v1/evaluators/{ref}/runs` with `request_ids` (up to 1,000; in the console, the requests of a period) judges logged requests in the background. Only requests whose prompt and answer were logged can be judged; the rest are counted as `not_logged`. With `dry_run: true` it returns the request count and the estimated provider cost; to start it, `spend_cap` in US dollars is required and must be at least the estimate. The run stops when its spend reaches the cap, and can be stopped by a person.

## What a judge call is

- A request on your own provider, billed to you by that provider, metered with `purpose: judge` and `parent_request_id` set to the judged request. It counts against your plan's monthly request allowance.
- Never judged itself: judge calls run no evaluator and no eval monitor.
- A failure (an answer that is not the format asked for, a provider error, a refusal, a timeout after 30 seconds) is counted on the evaluator with the last reason. It is never written as a score.

## Reading evaluators

`GET /v1/evaluators` (a key with history access; a key scoped to a project sees that project's evaluators), `GET /manage/v1/evaluators` (`quality:read`), and the MCP tool `list_evaluators` return each evaluator with `this_month`: judged calls, failures, skipped samples, the estimated provider cost and the last failure.
