See each run, hear from each reader, version each prompt.
The gateway your requests already pass through records them as runs with their estimated provider cost, takes scores and end-user feedback on them, and fills managed prompts before its own checks. One integration, on your own provider accounts.
INFRO accounts are self-serve. The catalog, the docs, and the API contract are public, the gateway is deployed, and anything beyond v1 is marked Later. Plans and trial
Runs
An agent turn, a retrieval answer or a document pipeline is several model calls. Send one trace id with each of them and INFRO lists them as one run, with what the run cost your provider, so the question becomes which run and which step.
One row per run
Requests that share a trace id and a workflow are one run: its requests, errors, duration, models, estimated provider cost, scores and prompt versions.
Find the run that matters
Filter by workflow, model, end user or errors, and sort by cost or duration over a window of up to 31 days.
Then every step
Open a run in the console, or call the trace endpoint, for its requests in order with the cost of each step.
One header on each call.
A request without a trace id belongs to no run. Runs are grouped by workflow and trace id together, because trace ids are yours and two workflows can reuse one.
A run covers the model calls that went through INFRO, and says so in every answer. The full contract is on Runs.
# Send the same trace id on every call of one run
curl https://api.infro.io/v1/chat/completions \
-H "Authorization: Bearer $INFRO_API_KEY" \
-H "X-INFRO-Trace-Id: run_42" \
-H "Content-Type: application/json" \
-d '{"model": "anthropic/claude-sonnet-5", "metadata": {"workflow": "support-answer"},
"messages": [{"role": "user", "content": "Where is my order?"}]}'
# Then list the costliest failing runs of that workflow
curl "https://api.infro.io/v1/runs?workflow=support-answer&sort=costliest&has_errors=true" \
-H "Authorization: Bearer $INFRO_API_KEY"Scores and feedback
Scores from your code, from eval monitors, from your team and from the people reading your answers land on the same request, run and prompt version. Scores are not gateway requests and do not count against your plan's request allowance.
Feedback from the people reading the answer
Every successful response carries a signed token. The page showing the answer sends it with a thumbs up or down, with no API key in the browser.
Scores from your code
A number, a yes or no, or a label, on a request or on a whole run, with an optional comment.
From eval monitors and your team
Eval monitors score a share of served answers, and members score a request from the console. Every score is on the request, its run and its prompt version.
The token is the only authority.
The feedback endpoint needs no API key and accepts any origin. A token names one request, can rate that request and nothing else, and expires after seven days.
Relaying the token from your own server, limits and retention are on Scores and feedback.
// The answer's response carried X-INFRO-Feedback-Token.
const feedbackToken = res.headers.get("X-INFRO-Feedback-Token");
// When the person clicks thumbs up: no API key in the browser.
await fetch("https://api.infro.io/v1/feedback", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ token: feedbackToken, value: true }),
});Prompt management
Keep prompts as numbered versions with labels, and name one on a chat completion. The version runs through the same allowlist, guardrails and budget as every request, because the gateway fills it before them, and the version is recorded on the request.
Versions that never change
A version holds messages with variables, and can carry a default model and settings. A change is a new version.
Filled before the checks
The gateway fills the version before routing, model allowlists, guardrails and the spend check, so each one reads the final messages.
Production moved by a person
Labels such as staging move from code. The protected production label is moved by an owner or admin in the console, and every move is in the audit log.
Each version's record
Requests, error rate, p50 and p95 latency, estimated provider cost per request, mean score and feedback share, for the window you choose.
Roll back by moving the label
Point production at the earlier version. Requests use it within five seconds, with no deploy.
A field or a header
Name the prompt in the request body with its variables, or in a header for a client that cannot add a field.
Name the prompt; the gateway does the rest.
The response says which version answered. Prompts are resolved on chat completions, and on Messages and Responses requests from coding agents. A prompt your application fetches and sends to a provider directly is not checked by the gateway, and the answer says so.
Labels, variables, protected labels and the management API are on Prompt management.
curl https://api.infro.io/v1/chat/completions \
-H "Authorization: Bearer $INFRO_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "openai/gpt-5",
"prompt": {
"id": "support-reply",
"label": "production",
"variables": { "customer_name": "Ada" }
},
"messages": [{ "role": "user", "content": "Where is my order?" }]
}'Evaluators
Write a rubric, choose any chat model you have connected, and INFRO has it judge answers through the gateway: a sample of live traffic, or past requests on demand. Every verdict lands as a score on the request it judged.
Your rubric, any chat model
A rubric with the question, the answer and, in experiments, the expected output. The judge returns a number, a pass or fail, or one of your categories.
Live, within a monthly limit
Judge a share of served answers, filtered by project, model, prompt version or workflow. Live judging pauses for the month when the evaluator's spend limit is reached.
Or past requests, on demand
Judge up to 1,000 logged requests in the background, with the request count and estimated provider cost first and a spend cap it stops at.
A judge call is a request like any other.
It runs on your provider account and is billed to you by that provider. A live judge call runs as the key that made the judged request, so that key's model access, guardrails and budgets apply, and it starts after the answer is sent, never slowing or changing it. Every judge call counts against your plan's request allowance.
Placeholders, output types and failures are on Evaluators.
# Judge 5% of one model's answers, at most $25 a month
curl -X POST "https://api.infro.io/manage/v1/evaluators" \
-H "Authorization: Bearer $INFRO_MANAGEMENT_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"slug": "helpfulness",
"model": "openai/gpt-5.4",
"rubric": "Does the answer address what was asked, accurately and completely?\n\nQuestion: {{input}}\n\nAnswer: {{output}}",
"output": {"type": "numeric"},
"sampling": {"rate": 0.05, "models": ["anthropic/claude-sonnet-5"]},
"monthly_spend_limit": 25
}'Datasets and experiments
Build a dataset from the requests you already logged, replay it through the gateway across models, prompt versions and settings, and compare what each one scored and cost, behind an estimate and a cap.
From your traffic in one click
Add a logged request, or every request the current filters show (up to 1,000), with its answer as the expected output if you choose.
Or uploaded
JSONL or CSV of up to 5 MB, or items sent from code. A problem names the line or row.
Kept until you delete it
Items you chose to keep, not tied to your request-log retention, and deleted with the dataset, its project or your organization.
Up to six variants
A model or route, a prompt version and settings per variant. Every answer is judged by up to five evaluators and compared on scores, error rate, latency and estimated cost per item.
Estimate first, then a cap
The request count, the estimated provider cost and the share of your request allowance before anything runs. A spend cap at least the estimate is required, and the run stops at it.
Through the same checks
Every call is a gateway request on your own providers: model access, guardrails, environment and spend limits apply.
Nothing runs until you set the cap.
A model with no known rate is listed as unpriced and the experiment cannot start, because its spend could not be held to the cap. Verdicts from an experiment's judges stay with the experiment, so a test never moves a production score.
Variants, stop reasons and the comparison are on Experiments; building and uploading items on Datasets.
# Estimate first: nothing runs with dry_run
curl -X POST "https://api.infro.io/manage/v1/experiments" \
-H "Authorization: Bearer $INFRO_MANAGEMENT_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"dataset": "support-golden",
"variants": [
{"label": "current", "model": "anthropic/claude-sonnet-5", "prompt": {"slug": "support-reply", "version": 3}},
{"label": "candidate", "model": "anthropic/claude-sonnet-5", "prompt": {"slug": "support-reply", "version": 4}},
{"label": "cheaper", "model": "openai/gpt-5.4-mini", "prompt": {"slug": "support-reply", "version": 4}}
],
"evaluators": ["helpfulness"],
"dry_run": true
}'Rollback rules
A rule watches one label of one prompt. When the version it points at scores below your limit, the rule moves the label back to the previous version, or only alerts, the same audited move a person makes in the console.
Three signals
The mean of a score, the pass rate of a pass-or-fail score, or your end users' thumbs-down share, checked every five minutes over your window once enough answers are scored.
Back, or only an alert
The rule moves the label back to the previous version or the one you name, or only alerts. Either way the move is in the audit log and sent to your alert channels.
Protected labels, authorized by a person
A rule that moves a protected label is saved by an owner or admin in the console. A management token never writes one.
Each rule reads as a sentence.
A rule fires once for a version, then stays quiet until somebody moves the label again. A person who moves the label while the check runs wins. Rules read scores, which do not count against your request allowance, and call no model.
Metrics, windows, protected labels and the management API are on Prompt rollback rules.
# Roll staging back when users dislike a version
curl https://api.infro.io/manage/v1/prompts/support-reply/rules \
-H "Authorization: Bearer $INFRO_MANAGEMENT_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"label": "staging",
"metric": "thumbs_down_share_above",
"threshold": 0.3,
"min_samples": 100,
"window_minutes": 60,
"action": "rollback",
"target_version": "previous"
}'Alongside your tools
Every request can also be pushed as an OpenTelemetry span to a collector you choose, from the request path, with the endpoint and headers set in the console. Runs and scores stay in INFRO; your traces stay where your team already reads them.
They run together. Langfuse is an open-source observability and evaluation platform, with annotation queues for human review and experiments that run your own code, retrieval and tools included, which INFRO does not have. INFRO sits in the request path: it routes, fails over and refuses spend before your provider is called, and its judges and experiments run on that same path. A tested recipe for sending INFRO's spans to Langfuse is not published yet.
FAQ
Can INFRO manage prompts?
Yes, on every plan. Save a prompt as numbered versions with labels such as production and staging, and name it on a chat completion with a prompt field or the X-INFRO-Prompt header. The gateway fills in the version before routing, model allowlists, guardrails and the spend check, so every check reads the final messages, and the version is recorded on the request. A version never changes; rolling back is moving the label to an earlier version, which requests use within five seconds and without a deploy. Moving the protected production label is done by an owner or admin in the console.
How do I collect user feedback on AI answers?
Every successful response to a gateway key carries an X-INFRO-Feedback-Token header: a token signed by INFRO that names one request and is valid for seven days. Hand it to the page that shows the answer, and when the person clicks thumbs up or down, post it to POST /v1/feedback. That endpoint needs no API key, so your key never reaches the browser, and the token can rate its own request and nothing else. The feedback is shown on the request, its run and the prompt version that answered, and scores do not count against your request allowance.
Can INFRO judge the quality of AI answers?
Yes, with evaluators, on every plan. An evaluator is a rubric that a chat model you choose applies to answers, on one of your own provider connections through the gateway. It judges a sample of live traffic, held to a monthly spend limit you set, or up to 1,000 logged requests on demand, with an estimate first and a spend cap. Each verdict is a score on the request it judged, beside your own scores and your end users' feedback. Judging starts after the answer is sent, so it never slows or changes it. Each judge call is a request on your provider, billed to you by that provider and counted against your plan's request allowance.
What is a run in INFRO?
The requests of one workflow that carried the same trace id: an agent turn, a retrieval answer or a document pipeline. Send the trace id in the X-INFRO-Trace-Id header or metadata.trace_id, and the workflow in metadata, and GET /v1/runs and the console list one row per run with its requests, errors, duration, models, estimated provider cost, scores and prompt versions. A run covers the model calls that went through INFRO; your vector database, retrieval and own compute are not in its figures.
Can my application fetch a prompt and call the provider itself?
Yes: GET /v1/prompts returns a version's messages and settings with any gateway key that may use the prompt. A prompt used that way is not enforced, because INFRO never sees the call: no guardrail, allowlist or spend limit applies and no request records the version, and the answer says so. Naming the prompt on a chat completion through INFRO is what applies the checks.
How do I test a prompt or model change before it ships?
Run an experiment. Build a dataset from logged requests in one click, or upload one as JSONL or CSV, then replay it through the gateway across up to six variants: a model or route, a prompt version, and settings. Your evaluators judge every answer. The estimate comes first: the request count and the estimated provider cost per variant. To start, you set a spend cap at least that estimate, and the run stops if its spend reaches it. The comparison shows each variant's scores, error rate, p50 and p95 latency and estimated cost per item, and Promote in the console moves a prompt label to the version a variant ran.
Can a prompt roll back by itself when quality drops?
Yes, with a rollback rule on a label such as production. Every five minutes the rule reads the scores of the answers served from the version that label points at, over a window you choose: the mean of a score, the pass rate of a pass-or-fail score, or your end users' thumbs-down share. When the figure crosses your limit with at least the minimum number of scored answers, the rule moves the label back to the previous version, or only alerts. Every move is in the audit log and sent to your alert channels. A rule that moves a protected label is saved by an owner or admin in the console.
Every other question about INFRO is on the FAQ.
Get started
Quality and control in one record.
The request that was routed, priced and checked is the request that was scored, on every plan.