Skip to content

DocsObservability

Alerts

Spend anomalies, budget forecasts, cost-estimate drift, unpriced traffic and failing connections, sent to email, Slack, Teams, PagerDuty or a webhook.


INFRO is in private early access. Running these examples requires an early-access account, an active trial or subscription entitlement, and your own provider connection. Self-serve purchasing is not generally open. See Early access.

INFRO watches the estimated provider cost of every request it routes and tells you when something changes: a project spending far more than it usually does, a budget that will be exceeded before it resets, cost estimates that have drifted from what your provider reports, requests running without a price, and provider connections that are failing. Every figure is an estimate of what your own provider will bill you — INFRO charges nothing per request.

Findings appear in the console under Alerts on every plan. Where they are sent is up to you: a notification rule routes an event to email, a Slack channel, a Microsoft Teams channel, a PagerDuty service, or a signed webhook endpoint.

Events

EventRaised whenSeverity
spend.anomalyA scope is on course to spend far more than its usual day, or (organization only) far less. See How an anomaly is scored.warning or critical; a drop is info
spend.forecast_over_budgetAt the current rate a budget or key limit will be exceeded by at least 5% before it resets, with at least three days of a monthly period left. Sent once per budget per period.critical for a limit that blocks, warning for an alert-only one
cost.driftOn at least five of the last seven days, INFRO's estimate for a provider leaned the same way, by more than 5%, from the cost that provider's own API reported. Requires cost reconciliation.warning
budget.threshold_crossedA budget passed one of its alert thresholds. Also emailed to the billing contact, as before.critical when a blocking budget reaches 100%, otherwise warning
connection.failingA provider connection is failing, or repeated failures took it out of rotation.critical when out of rotation, otherwise warning
unpriced.trafficFor more than six hours, at least 5% of successful requests (and at least 20) had no price. Every spend figure is a lower bound until they are priced.warning
traffic.model_changedA model on one of your provider connections is now answered by a model id the provider had not reported in the previous two weeks, usually a new snapshot behind the same name. See Traffic changes.warning; critical from an estimated $500 a week
traffic.regressionA traffic metric moved far from its usual day with no error: output tokens per request, prompt-cache share, fallback share, answers cut off by the length limit, p95 latency or time to first token, or one error code's share. figures.metric names which. See Traffic changes.By estimated weekly impact: info below $50, warning from $50, critical from $500

MCP gateway events

Two events from the MCP gateway, routed by the same notification rules to email, Slack or a signed webhook:

EventRaised whenSeverity
mcp.tool_changedA sync found that an MCP server changed the definition of an approved tool. The tool is refused until an admin approves the new definition. Sent once per tool and new definition.warning
mcp.server_failingOver 15 minutes, at least 20% of an MCP server's calls failed with an upstream error or a timeout, with at least 20 calls. Sent at most once per server per hour.warning

How an anomaly is scored

Each scope's spend is rolled up per UTC day. Today's spend so far is projected to the end of the UTC day and compared with that scope's usual day: the median of the same weekday over the last four weeks (while fewer than three of that weekday are available, blended 60/40 with the median of all recent days). The difference is measured in units of the scope's typical day-to-day spread — the median absolute deviation, scaled, never taken as less than 10% of the usual day or less than the floor below.

  • Warning at 4 spreads above the usual day, critical at 8.
  • Changes smaller than max($10, 3% of the organization's average day) are never reported, however large in relative terms.
  • A scope is learning until it has 14 days of history with priced traffic on at least 7 of them. The console shows how far along it is.
  • Nothing is scored before a quarter of the UTC day has elapsed, because an early projection is noise.
  • A day where more than 20% of requests are unpriced is not scored at all: it is a lower bound, and a lower bound cannot be called anomalous.
  • A sharp drop is reported for the whole organization only, as info — it is most often an outage in your own application.
  • A finding raised as a warning that becomes critical later the same day is updated and sent once more. Nothing is sent a third time.

Scopes: the organization, each project, key, provider connection and model on every plan; each workflow and end customer (from the workflow and user_id request metadata) from the Team plan, for your 200 highest-spending end customers. When the organization or a project fires, the finding names the three models, keys, workflows or end customers that contributed most, and those are not alerted separately the same day.

Notification channels and rules

A channel is where alerts go: an email list (verified members of the organization or its billing email), a Slack incoming webhook, a Microsoft Teams workflow, a PagerDuty service, or one of your webhook endpoints. A rule sends an event, at or above a severity, optionally for one scope only, to a channel. Email channels are on every plan; Slack, Teams, PagerDuty and webhook channels from the Startup plan. Disabling or deleting a channel works on every plan.

With no rule for an event: critical spend anomalies, budget forecasts and cost drift go to the organization's billing contact. Budget crossings and failing connections keep the emails they already send. Everything else is recorded in the console until a rule routes it.

  • A Slack channel only accepts a URL beginning https://hooks.slack.com/services/. The URL is stored encrypted and never shown again; the console shows its last four characters.
  • A Microsoft Teams channel takes the URL of a Teams workflow built from the template that posts to a channel when a webhook request is received (hosts under logic.azure.com or api.powerplatform.com), or of an older connector (webhook.office.com). Each finding arrives as an Adaptive Card with an Open in INFRO button. The URL is stored encrypted and never shown again.
  • A PagerDuty channel takes the Integration Key of an Events API v2 integration on the service to alert, and the region your PagerDuty account is hosted in (US or EU). Each finding is sent as a trigger event whose dedup_key is the finding's id, so a finding that is sent again updates the same PagerDuty alert. The key is stored encrypted and never shown again.
  • Slack, Teams and PagerDuty messages leave out end-customer ids unless the rule allows them, because an end-customer id can be personal data. Email, webhooks and the console always include them.
  • Each channel can send up to five test messages an hour from the console.
  • Failed sends are retried after 1 minute, 5 minutes, 30 minutes, 2 hours and 12 hours. After the last retry the channel is marked failing and the billing contact is emailed once.
  • Redirects are never followed, and addresses that are not publicly routable are refused.

Webhook payload

Alert events use the same envelope and the same INFRO-Signature scheme as job events. id is stable across retries of one send; an upgrade from warning to critical is a new event with a new id.

json
{
  "id": "evt_Q2M7K4ZP9W1TXB3F",
  "type": "spend.anomaly",
  "created_at": 1790241600,
  "organization_id": "org_…",
  "data": {
    "id": "calr_7FJ2N6Q0VH3KD9MB",
    "object": "alert",
    "type": "spend.anomaly",
    "severity": "warning",
    "status": "open",
    "scope": { "kind": "project", "id": "proj_…", "label": "Project support-bot" },
    "title": "Unusual spend: project support-bot on course for $412 today",
    "sentence": "Project support-bot is on course for $412 today; its usual Tuesday is $96 (±$18). Largest increases: +$248 from model openai/gpt-5.6-sol. Estimated provider cost, 4% of requests unpriced.",
    "facts": [{ "label": "Projected today", "value": "$412" }],
    "figures": {
      "day": "2026-09-22",
      "direction": "up",
      "projected_micros": "412000000",
      "expected_micros": "96000000",
      "spread_micros": "18000000",
      "score": 17.6,
      "unpriced_share": 0.04
    },
    "contributors": [
      { "scope_kind": "model", "scope_id": "openai/gpt-5.6-sol", "label": "Model openai/gpt-5.6-sol", "delta_micros": "248000000" }
    ],
    "likely_cause": null,
    "series": [],
    "deltas": [],
    "href": "/requests?range=24h&project=proj_…",
    "muted": false,
    "notifications_sent": 1,
    "acknowledged_at": null,
    "created_at": "2026-09-22T12:12:03.000Z",
    "updated_at": "2026-09-22T12:12:03.000Z"
  }
}

Money in figures and contributors is micro-USD as a decimal string. sentence states the arithmetic in words and is safe to show as-is. likely_cause, series (daily {day, value} readings of the metric, null for a day with no reading) and deltas (a new snapshot's before and after) are filled in on traffic changes; on every other event they are null or empty.

Read findings from the API

GET /v1/alerts lists findings, newest first, and GET /v1/alerts/{id} returns one. The key needs history access: an organization-wide key reads every finding, a project-scoped key reads only findings on its project, and a key with no project reads only findings on itself. Filters: status (open, acknowledged, resolved), type, since (ISO 8601). Page with cursor, using the response's next_cursor. Channels and rules are managed in the console.

bash
curl "https://api.infro.io/v1/alerts?status=open&type=spend.anomaly" \
  -H "Authorization: Bearer $INFRO_API_KEY"

Budgets and unpriced traffic

Anomalies and forecasts describe spend; budgets enforce it. Under a blocking budget, requests that could reach an unpriced model are refused unless you allow them; allowed unpriced traffic is not governed by the budget. Price your models under Prices so both the budget and the detector see the whole of your spend.

↑ ↓ navigate↵ open/ anywhere in docs