Skip to content

DocsObservability

Evaluators

LLM-as-judge on any chat model, on your own provider account: score a sample of live traffic or past requests by your rubric, with a monthly spend limit.


An evaluator is a rubric that a chat model, the judge, applies to answers. The judge runs on one of your own provider connections, through the gateway, and each verdict is written as a score named after the evaluator on the request it judged, so it appears in the request log, on runs and in a prompt version's figures beside your own scores and your end users' feedback.

The rubric

A rubric is text with placeholders: {{input}} is the last user message of the judged request, {{output}} its answer, {{expected}} a dataset item's expected output (in experiments), and {{metadata.<key>}} a value from the request's metadata. A rubric must include {{output}}. Each value is clipped to 12,000 characters and filled in once, so text in a request can never add a placeholder. Instead of text, an evaluator can use a managed prompt at a label as its rubric, so the judge has versions and labels of its own.

output says what the judge returns: {"type": "numeric"} (a score from 0 to 1), {"type": "boolean"} (pass or fail, stored as 1 or 0) or {"type": "category", "values": ["resolved", "escalated", "wrong"]}. INFRO adds an instruction to answer with one JSON object, {"score": …, "reason": "…"}, asks for response_format: json_object at temperature 0 with at most 400 output tokens, and reads the answer strictly. The reason is kept as the score's comment under the same rule as other comments: only while your organization's request logging is on.

Live traffic

sampling.rate above 0 judges that share of served chat completions (including /v1/messages and /v1/responses), filtered by projects, models (the model that served the request), prompt ({"slug": …, "version": …}), workflows and environments. An evaluator limited to a project judges that project's requests only. Judging starts after the response has been sent and is never waited for: it never slows, changes or fails the judged request. A response served from INFRO's response cache, and a console connection test, are not judged.

The judge call runs as the key that made the judged request: its model access, guardrails, environment and spend limits apply, and a blocking budget refuses it. Live sampling needs monthly_spend_limit, in US dollars of estimated provider cost per calendar month (UTC). When the evaluator's judge calls reach it, live judging pauses until the next month and the evaluator records when (this_month.limit_reached_at); judge calls already in flight still finish.

Judge 5% of one model's answers, at most $25 a month
curl -X POST "https://api.infro.io/manage/v1/evaluators" \
  -H "Authorization: Bearer $INFRO_MANAGEMENT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "slug": "helpfulness",
    "model": "openai/gpt-5.4",
    "rubric": "Does the answer address what was asked, accurately and completely?\n\nQuestion: {{input}}\n\nAnswer: {{output}}",
    "output": {"type": "numeric"},
    "sampling": {"rate": 0.05, "models": ["anthropic/claude-sonnet-5"]},
    "monthly_spend_limit": 25
  }'

Run on past requests

POST /manage/v1/evaluators/{ref}/runs with request_ids (up to 1,000; in the console, the requests of a period) judges logged requests in the background. Only requests whose prompt and answer were logged can be judged; the rest are counted as not_logged. With dry_run: true it returns the request count and the estimated provider cost; to start it, spend_cap in US dollars is required and must be at least the estimate. The run stops when its spend reaches the cap, and can be stopped by a person.

What a judge call is

  • A request on your own provider, billed to you by that provider, metered with purpose: judge and parent_request_id set to the judged request. It counts against your plan's monthly request allowance.
  • Never judged itself: judge calls run no evaluator and no eval monitor.
  • A failure (an answer that is not the format asked for, a provider error, a refusal, a timeout after 30 seconds) is counted on the evaluator with the last reason. It is never written as a score.

Reading evaluators

GET /v1/evaluators (a key with history access; a key scoped to a project sees that project's evaluators), GET /manage/v1/evaluators (quality:read), and the MCP tool list_evaluators return each evaluator with this_month: judged calls, failures, skipped samples, the estimated provider cost and the last failure.

↑ ↓ navigate↵ open/ anywhere in docs