# Alerts

> Spend anomalies, budget forecasts, cost-estimate drift, unpriced traffic and failing connections, sent to email, Slack, Teams, PagerDuty or a webhook.

Source: https://infro.io/docs/observability/alerts

> **Note:** INFRO is in private early access. Running these examples requires an early-access account, an active trial or subscription entitlement, and your own provider connection. Self-serve purchasing is not generally open. See [Early access](https://infro.io/early-access).

INFRO watches the estimated provider cost of every request it routes and tells you when something changes: a project spending far more than it usually does, a budget that will be exceeded before it resets, cost estimates that have drifted from what your provider reports, requests running without a price, and provider connections that are failing. Every figure is an **estimate of what your own provider will bill you** — INFRO charges nothing per request.

Findings appear in the console under **Alerts** on every plan. Where they are sent is up to you: a notification rule routes an event to email, a Slack channel, a Microsoft Teams channel, a PagerDuty service, or a signed webhook endpoint.

## Events

| Event | Raised when | Severity |
| --- | --- | --- |
| `spend.anomaly` | A scope is on course to spend far more than its usual day, or (organization only) far less. See [How an anomaly is scored](#how-an-anomaly-is-scored). | `warning` or `critical`; a drop is `info` |
| `spend.forecast_over_budget` | At the current rate a budget or key limit will be exceeded by at least 5% before it resets, with at least three days of a monthly period left. Sent once per budget per period. | `critical` for a limit that blocks, `warning` for an alert-only one |
| `cost.drift` | On at least five of the last seven days, INFRO's estimate for a provider leaned the same way, by more than 5%, from the cost that provider's own API reported. Requires [cost reconciliation](https://infro.io/docs/org/billing). | `warning` |
| `budget.threshold_crossed` | A budget passed one of its alert thresholds. Also emailed to the billing contact, as before. | `critical` when a blocking budget reaches 100%, otherwise `warning` |
| `connection.failing` | A provider connection is failing, or repeated failures took it out of rotation. | `critical` when out of rotation, otherwise `warning` |
| `unpriced.traffic` | For more than six hours, at least 5% of successful requests (and at least 20) had no price. Every spend figure is a lower bound until they are priced. | `warning` |
| `traffic.model_changed` | A model on one of your provider connections is now answered by a model id the provider had not reported in the previous two weeks, usually a new snapshot behind the same name. See [Traffic changes](https://infro.io/docs/observability/traffic-changes). | `warning`; `critical` from an estimated $500 a week |
| `traffic.regression` | A traffic metric moved far from its usual day with no error: output tokens per request, prompt-cache share, fallback share, answers cut off by the length limit, p95 latency or time to first token, or one error code's share. `figures.metric` names which. See [Traffic changes](https://infro.io/docs/observability/traffic-changes). | By estimated weekly impact: `info` below $50, `warning` from $50, `critical` from $500 |

### MCP gateway events

Two events from the [MCP gateway](https://infro.io/docs/mcp-gateway), routed by the same notification rules to email, Slack or a signed webhook:

| Event | Raised when | Severity |
| --- | --- | --- |
| `mcp.tool_changed` | A sync found that an MCP server changed the definition of an approved tool. The tool is refused until an admin approves the new definition. Sent once per tool and new definition. | `warning` |
| `mcp.server_failing` | Over 15 minutes, at least 20% of an MCP server's calls failed with an upstream error or a timeout, with at least 20 calls. Sent at most once per server per hour. | `warning` |

## How an anomaly is scored

Each scope's spend is rolled up per UTC day. Today's spend so far is projected to the end of the UTC day and compared with that scope's **usual day**: the median of the same weekday over the last four weeks (while fewer than three of that weekday are available, blended 60/40 with the median of all recent days). The difference is measured in units of the scope's typical day-to-day spread — the median absolute deviation, scaled, never taken as less than 10% of the usual day or less than the floor below.

- **Warning** at 4 spreads above the usual day, **critical** at 8.
- Changes smaller than **max($10, 3% of the organization's average day)** are never reported, however large in relative terms.
- A scope is **learning** until it has 14 days of history with priced traffic on at least 7 of them. The console shows how far along it is.
- Nothing is scored before a quarter of the UTC day has elapsed, because an early projection is noise.
- A day where more than 20% of requests are unpriced is not scored at all: it is a lower bound, and a lower bound cannot be called anomalous.
- A sharp **drop** is reported for the whole organization only, as `info` — it is most often an outage in your own application.
- A finding raised as a warning that becomes critical later the same day is updated and sent once more. Nothing is sent a third time.

Scopes: the organization, each project, key, provider connection and model on every plan; each workflow and end customer (from the `workflow` and `user_id` request metadata) from the Team plan, for your 200 highest-spending end customers. When the organization or a project fires, the finding names the three models, keys, workflows or end customers that contributed most, and those are not alerted separately the same day.

## Notification channels and rules

A **channel** is where alerts go: an email list (verified members of the organization or its billing email), a Slack incoming webhook, a Microsoft Teams workflow, a PagerDuty service, or one of your [webhook endpoints](https://infro.io/docs/api/webhooks). A **rule** sends an event, at or above a severity, optionally for one scope only, to a channel. Email channels are on every plan; Slack, Teams, PagerDuty and webhook channels from the Startup plan. Disabling or deleting a channel works on every plan.

With no rule for an event: critical spend anomalies, budget forecasts and cost drift go to the organization's billing contact. Budget crossings and failing connections keep the emails they already send. Everything else is recorded in the console until a rule routes it.

- A Slack channel only accepts a URL beginning `https://hooks.slack.com/services/`. The URL is stored encrypted and never shown again; the console shows its last four characters.
- A Microsoft Teams channel takes the URL of a Teams workflow built from the template that posts to a channel when a webhook request is received (hosts under `logic.azure.com` or `api.powerplatform.com`), or of an older connector (`webhook.office.com`). Each finding arrives as an Adaptive Card with an **Open in INFRO** button. The URL is stored encrypted and never shown again.
- A PagerDuty channel takes the Integration Key of an Events API v2 integration on the service to alert, and the region your PagerDuty account is hosted in (US or EU). Each finding is sent as a `trigger` event whose `dedup_key` is the finding's id, so a finding that is sent again updates the same PagerDuty alert. The key is stored encrypted and never shown again.
- Slack, Teams and PagerDuty messages leave out end-customer ids unless the rule allows them, because an end-customer id can be personal data. Email, webhooks and the console always include them.
- Each channel can send up to five test messages an hour from the console.
- Failed sends are retried after 1 minute, 5 minutes, 30 minutes, 2 hours and 12 hours. After the last retry the channel is marked failing and the billing contact is emailed once.
- Redirects are never followed, and addresses that are not publicly routable are refused.

## Webhook payload

Alert events use the same envelope and the same `INFRO-Signature` scheme as [job events](https://infro.io/docs/api/webhooks). `id` is stable across retries of one send; an upgrade from warning to critical is a new event with a new `id`.

```json
{
  "id": "evt_Q2M7K4ZP9W1TXB3F",
  "type": "spend.anomaly",
  "created_at": 1790241600,
  "organization_id": "org_…",
  "data": {
    "id": "calr_7FJ2N6Q0VH3KD9MB",
    "object": "alert",
    "type": "spend.anomaly",
    "severity": "warning",
    "status": "open",
    "scope": { "kind": "project", "id": "proj_…", "label": "Project support-bot" },
    "title": "Unusual spend: project support-bot on course for $412 today",
    "sentence": "Project support-bot is on course for $412 today; its usual Tuesday is $96 (±$18). Largest increases: +$248 from model openai/gpt-5.6-sol. Estimated provider cost, 4% of requests unpriced.",
    "facts": [{ "label": "Projected today", "value": "$412" }],
    "figures": {
      "day": "2026-09-22",
      "direction": "up",
      "projected_micros": "412000000",
      "expected_micros": "96000000",
      "spread_micros": "18000000",
      "score": 17.6,
      "unpriced_share": 0.04
    },
    "contributors": [
      { "scope_kind": "model", "scope_id": "openai/gpt-5.6-sol", "label": "Model openai/gpt-5.6-sol", "delta_micros": "248000000" }
    ],
    "likely_cause": null,
    "series": [],
    "deltas": [],
    "href": "/requests?range=24h&project=proj_…",
    "muted": false,
    "notifications_sent": 1,
    "acknowledged_at": null,
    "created_at": "2026-09-22T12:12:03.000Z",
    "updated_at": "2026-09-22T12:12:03.000Z"
  }
}
```

Money in `figures` and `contributors` is micro-USD as a decimal string. `sentence` states the arithmetic in words and is safe to show as-is. `likely_cause`, `series` (daily `{day, value}` readings of the metric, `null` for a day with no reading) and `deltas` (a new snapshot's before and after) are filled in on [traffic changes](https://infro.io/docs/observability/traffic-changes); on every other event they are `null` or empty.

## Read findings from the API

`GET /v1/alerts` lists findings, newest first, and `GET /v1/alerts/{id}` returns one. The key needs history access: an organization-wide key reads every finding, a project-scoped key reads only findings on its project, and a key with no project reads only findings on itself. Filters: `status` (`open`, `acknowledged`, `resolved`), `type`, `since` (ISO 8601). Page with `cursor`, using the response's `next_cursor`. Channels and rules are managed in the console.

```bash
curl "https://api.infro.io/v1/alerts?status=open&type=spend.anomaly" \
  -H "Authorization: Bearer $INFRO_API_KEY"
```

## Budgets and unpriced traffic

Anomalies and forecasts describe spend; [budgets](https://infro.io/docs/org/spend-controls) enforce it. Under a blocking budget, requests that could reach an unpriced model are refused unless you allow them; allowed unpriced traffic is not governed by the budget. Price your models under Prices so both the budget and the detector see the whole of your spend.