Skip to content

DocsFeatures

Prompt rollback rules

Move a prompt label back to the previous version, or alert, when its version scores below a limit: a mean score, a pass rate or end-user thumbs down.


A rule watches one label of one managed prompt, such as production. Every five minutes INFRO reads the scores of the answers served from the version that label points at, over the rule's window. When the figure crosses the rule's limit with at least the rule's number of scored answers, the rule fires: it moves the label back to the previous version (or the version you named), or only alerts. A move is the same label move a person makes in the console: requests naming the label receive the version it moved to within 5 seconds, and the audit log records it with the actor rollback rule prr_….

What a rule watches

metricReadsFires when
thumbs_down_share_aboveFeedback from your end users (POST /v1/feedback, true/false). score_name optional: empty reads every feedback name.The share of answers rated down rises above threshold, a share between 0 and 1.
mean_belowA numeric score of the name in score_name, from your code, your judges or eval monitors.The mean falls below threshold, on your score's own scale.
pass_rate_belowA true/false score of the name in score_name.The share that passed falls below threshold, between 0 and 1.

Figures are per answer: a request's scores of that name are averaged first, so one answer rated ten times counts once. Only answers served from the label's current version, in the last window_minutes (15 to 1,440), count, and only scores still inside your plan's history window. A rule decides on at least min_samples scored answers, 20 or more; with fewer it waits.

With compare_to_previous, the rule fires only when the current version is also worse than the previous version was over its own last window of the same length (the window ending at the last request that version served). When the previous version has fewer than min_samples scored answers there, the rule cannot show the new version is worse, and does not fire.

When a rule fires

  • action: rollback moves the label to target_version, or with previous (or null) to the version before the one it points at. When there is nothing to go back to (the label is at version 1, or already at the target), the rule alerts instead.
  • action: alert_only moves nothing.
  • Either way your alert channels receive prompt.rolled_back or prompt.quality_dropped, the organization's owners and admins receive one email, and the prompt's page shows it on its timeline with the figures, for example: _Rolled back production v4 → v3: thumbs-down share 38% over 120 answers in the last 1 hour._
  • A rule fires once for a version. After firing it stays quiet until somebody moves the label again; saving the rule wakes it too. A label is moved by at most one rule in one check.
  • A person who moves the label while the check runs wins: the rule's move is dropped and the rule reads the label again in five minutes.

Protected labels

A rule on a protected label (production, from creation) moves it without a person, so the person authorizes that in advance: adding, changing, switching off or deleting such a rule needs an owner or admin in the console, who confirms it is them. A management token never writes one (403 prompt_label_protected). A rule saved while its label was not protected is not authorized to move it if the label is protected later: it alerts instead and says so, until an owner or admin saves it again.

In the console

Each prompt's page lists its rules as sentences, with where each stands: watching, quiet after firing, cannot move the label (and why), or waiting for an owner's confirmation. Owners, admins and developers add rules on unprotected labels; every role that reads prompts sees them. Every change is in the audit log.

Management API

OperationScopeWhat it does
GET /prompts/{slug}/rulesprompts:readThe prompt's rules, each with status, status_text and sentence, and events: the last 50 times a rule fired, newest first.
POST /prompts/{slug}/rulesprompts:writelabel, metric, score_name, threshold, min_samples (default 20), window_minutes (default 60), compare_to_previous, action (default rollback), target_version (default previous), enabled.
PATCH /prompts/{slug}/rules/{id}prompts:writeAny of the same fields but label.
DELETE /prompts/{slug}/rules/{id}prompts:writeThe record of what it did stays on the prompt.
Roll staging back when users dislike a version
curl https://api.infro.io/manage/v1/prompts/support-reply/rules \
  -H "Authorization: Bearer $INFRO_MANAGEMENT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "label": "staging",
    "metric": "thumbs_down_share_above",
    "threshold": 0.3,
    "min_samples": 100,
    "window_minutes": 60,
    "action": "rollback",
    "target_version": "previous"
  }'
Response
{
  "id": "prr_7k2m9q4w8x1c5v3b",
  "object": "prompt_rollback_rule",
  "prompt": "support-reply",
  "label": "staging",
  "metric": "thumbs_down_share_above",
  "score_name": null,
  "threshold": 0.3,
  "min_samples": 100,
  "window_minutes": 60,
  "compare_to_previous": false,
  "action": "rollback",
  "target_version": null,
  "enabled": true,
  "authorized_protected": false,
  "status": "watching",
  "status_text": "Watching v4.",
  "sentence": "When the thumbs-down share of the version staging points at rises above 30% over the last 1 hour (at least 100 answers), move staging back to the previous version.",
  "last_fired_at": null
}

Limits: 20 rules per prompt and 200 per organization (400 prompt_rule_limit_reached); a target_version the prompt does not have is 400 prompt_version_not_found. Rules are on every plan. They read scores, which do not count against your request allowance, and they call no model.

↑ ↓ navigate↵ open/ anywhere in docs