DocsObservability
Evaluators
LLM-as-judge on any chat model, on your own provider account: score a sample of live traffic or past requests by your rubric, with a monthly spend limit.
An evaluator is a rubric that a chat model, the judge, applies to answers. The judge runs on one of your own provider connections, through the gateway, and each verdict is written as a score named after the evaluator on the request it judged, so it appears in the request log, on runs and in a prompt version's figures beside your own scores and your end users' feedback.
The rubric
A rubric is text with placeholders: {{input}} is the last user message of the judged request, {{output}} its answer, {{expected}} a dataset item's expected output (in experiments), and {{metadata.<key>}} a value from the request's metadata. A rubric must include {{output}}. Each value is clipped to 12,000 characters and filled in once, so text in a request can never add a placeholder. Instead of text, an evaluator can use a managed prompt at a label as its rubric, so the judge has versions and labels of its own.
output says what the judge returns: {"type": "numeric"} (a score from 0 to 1), {"type": "boolean"} (pass or fail, stored as 1 or 0) or {"type": "category", "values": ["resolved", "escalated", "wrong"]}. INFRO adds an instruction to answer with one JSON object, {"score": …, "reason": "…"}, asks for response_format: json_object at temperature 0 with at most 400 output tokens, and reads the answer strictly. The reason is kept as the score's comment under the same rule as other comments: only while your organization's request logging is on.
Live traffic
sampling.rate above 0 judges that share of served chat completions (including /v1/messages and /v1/responses), filtered by projects, models (the model that served the request), prompt ({"slug": …, "version": …}), workflows and environments. An evaluator limited to a project judges that project's requests only. Judging starts after the response has been sent and is never waited for: it never slows, changes or fails the judged request. A response served from INFRO's response cache, and a console connection test, are not judged.
The judge call runs as the key that made the judged request: its model access, guardrails, environment and spend limits apply, and a blocking budget refuses it. Live sampling needs monthly_spend_limit, in US dollars of estimated provider cost per calendar month (UTC). When the evaluator's judge calls reach it, live judging pauses until the next month and the evaluator records when (this_month.limit_reached_at); judge calls already in flight still finish.
curl -X POST "https://api.infro.io/manage/v1/evaluators" \
-H "Authorization: Bearer $INFRO_MANAGEMENT_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"slug": "helpfulness",
"model": "openai/gpt-5.4",
"rubric": "Does the answer address what was asked, accurately and completely?\n\nQuestion: {{input}}\n\nAnswer: {{output}}",
"output": {"type": "numeric"},
"sampling": {"rate": 0.05, "models": ["anthropic/claude-sonnet-5"]},
"monthly_spend_limit": 25
}'Run on past requests
POST /manage/v1/evaluators/{ref}/runs with request_ids (up to 1,000; in the console, the requests of a period) judges logged requests in the background. Only requests whose prompt and answer were logged can be judged; the rest are counted as not_logged. With dry_run: true it returns the request count and the estimated provider cost; to start it, spend_cap in US dollars is required and must be at least the estimate. The run stops when its spend reaches the cap, and can be stopped by a person.
What a judge call is
- A request on your own provider, billed to you by that provider, metered with
purpose: judgeandparent_request_idset to the judged request. It counts against your plan's monthly request allowance. - Never judged itself: judge calls run no evaluator and no eval monitor.
- A failure (an answer that is not the format asked for, a provider error, a refusal, a timeout after 30 seconds) is counted on the evaluator with the last reason. It is never written as a score.
Reading evaluators
GET /v1/evaluators (a key with history access; a key scoped to a project sees that project's evaluators), GET /manage/v1/evaluators (quality:read), and the MCP tool list_evaluators return each evaluator with this_month: judged calls, failures, skipped samples, the estimated provider cost and the last failure.