DocsFeatures
Prompt rollback rules
Move a prompt label back to the previous version, or alert, when its version scores below a limit: a mean score, a pass rate or end-user thumbs down.
A rule watches one label of one managed prompt, such as production. Every five minutes INFRO reads the scores of the answers served from the version that label points at, over the rule's window. When the figure crosses the rule's limit with at least the rule's number of scored answers, the rule fires: it moves the label back to the previous version (or the version you named), or only alerts. A move is the same label move a person makes in the console: requests naming the label receive the version it moved to within 5 seconds, and the audit log records it with the actor rollback rule prr_….
What a rule watches
metric | Reads | Fires when |
|---|---|---|
thumbs_down_share_above | Feedback from your end users (POST /v1/feedback, true/false). score_name optional: empty reads every feedback name. | The share of answers rated down rises above threshold, a share between 0 and 1. |
mean_below | A numeric score of the name in score_name, from your code, your judges or eval monitors. | The mean falls below threshold, on your score's own scale. |
pass_rate_below | A true/false score of the name in score_name. | The share that passed falls below threshold, between 0 and 1. |
Figures are per answer: a request's scores of that name are averaged first, so one answer rated ten times counts once. Only answers served from the label's current version, in the last window_minutes (15 to 1,440), count, and only scores still inside your plan's history window. A rule decides on at least min_samples scored answers, 20 or more; with fewer it waits.
With compare_to_previous, the rule fires only when the current version is also worse than the previous version was over its own last window of the same length (the window ending at the last request that version served). When the previous version has fewer than min_samples scored answers there, the rule cannot show the new version is worse, and does not fire.
When a rule fires
action: rollbackmoves the label totarget_version, or withprevious(or null) to the version before the one it points at. When there is nothing to go back to (the label is at version 1, or already at the target), the rule alerts instead.action: alert_onlymoves nothing.- Either way your alert channels receive
prompt.rolled_backorprompt.quality_dropped, the organization's owners and admins receive one email, and the prompt's page shows it on its timeline with the figures, for example: _Rolled back production v4 → v3: thumbs-down share 38% over 120 answers in the last 1 hour._ - A rule fires once for a version. After firing it stays quiet until somebody moves the label again; saving the rule wakes it too. A label is moved by at most one rule in one check.
- A person who moves the label while the check runs wins: the rule's move is dropped and the rule reads the label again in five minutes.
Protected labels
A rule on a protected label (production, from creation) moves it without a person, so the person authorizes that in advance: adding, changing, switching off or deleting such a rule needs an owner or admin in the console, who confirms it is them. A management token never writes one (403 prompt_label_protected). A rule saved while its label was not protected is not authorized to move it if the label is protected later: it alerts instead and says so, until an owner or admin saves it again.
In the console
Each prompt's page lists its rules as sentences, with where each stands: watching, quiet after firing, cannot move the label (and why), or waiting for an owner's confirmation. Owners, admins and developers add rules on unprotected labels; every role that reads prompts sees them. Every change is in the audit log.
Management API
| Operation | Scope | What it does |
|---|---|---|
GET /prompts/{slug}/rules | prompts:read | The prompt's rules, each with status, status_text and sentence, and events: the last 50 times a rule fired, newest first. |
POST /prompts/{slug}/rules | prompts:write | label, metric, score_name, threshold, min_samples (default 20), window_minutes (default 60), compare_to_previous, action (default rollback), target_version (default previous), enabled. |
PATCH /prompts/{slug}/rules/{id} | prompts:write | Any of the same fields but label. |
DELETE /prompts/{slug}/rules/{id} | prompts:write | The record of what it did stays on the prompt. |
curl https://api.infro.io/manage/v1/prompts/support-reply/rules \
-H "Authorization: Bearer $INFRO_MANAGEMENT_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"label": "staging",
"metric": "thumbs_down_share_above",
"threshold": 0.3,
"min_samples": 100,
"window_minutes": 60,
"action": "rollback",
"target_version": "previous"
}'{
"id": "prr_7k2m9q4w8x1c5v3b",
"object": "prompt_rollback_rule",
"prompt": "support-reply",
"label": "staging",
"metric": "thumbs_down_share_above",
"score_name": null,
"threshold": 0.3,
"min_samples": 100,
"window_minutes": 60,
"compare_to_previous": false,
"action": "rollback",
"target_version": null,
"enabled": true,
"authorized_protected": false,
"status": "watching",
"status_text": "Watching v4.",
"sentence": "When the thumbs-down share of the version staging points at rises above 30% over the last 1 hour (at least 100 answers), move staging back to the previous version.",
"last_fired_at": null
}Limits: 20 rules per prompt and 200 per organization (400 prompt_rule_limit_reached); a target_version the prompt does not have is 400 prompt_version_not_found. Rules are on every plan. They read scores, which do not count against your request allowance, and they call no model.