Alerts

An alert watches one metric — spend, error rate, or p95 latency — over a window, and notifies you when it crosses a threshold. Scope it to the whole organization, one project, or a single key, and route the notification to email, Slack, or a signed webhook.

Alerts observe; they never block a request. Hard ceilings are a separate mechanism — spend controls — and the difference matters enough that this page ends with it. Everything below is also configurable in the console under Alerts.

The alert object

{
  "id": "alert_4b0e7c2f",
  "metric": "error_rate",
  "scope": { "level": "project", "id": "proj_9c2ff1" },
  "threshold": 0.05,
  "window": "5m",
  "channels": [
    { "type": "email", "to": "oncall@example.com" },
    { "type": "webhook", "url": "https://example.com/hooks/infro-alerts" }
  ],
  "created_at": 1755950000,
  "last_triggered_at": null
}

metricstringrequired

"spend", "error_rate", or "p95_latency". What the alert measures — exact semantics in the table below.

scopeobject

Which traffic to measure. Defaults to {"level": "org"} — the whole organization.

scope.levelstringrequired

"org", "project", or "key".

scope.idstring

The project or key id to watch. Required when level is "project" or "key"; omit for "org".

thresholdnumberrequired

The line to cross: USD for spend, a ratio between 0 and 1 for error_rate, milliseconds for p95_latency.

windowstringrequired

"5m", "1h", "1d", or "billing_month". The evaluation window. billing_month is valid only for spend and resets on your billing day; anything else returns 400 invalid_request_error.

channelsarrayrequired

One to five delivery channels — see Channels below. An alert with an empty channels array is rejected.

An organization can hold up to 100 alerts; a POST beyond that returns 400 invalid_request_error. Delete stale ones rather than working around the cap — a hundred alerts nobody reads is the same as none.

Threshold semantics

spend sums usage.cost across the scope's requests — the same figure usage analytics reports for the same scope and window. error_rate counts a request as errored when its final status after provider failover and fallbacks was any 4xx or 5xx. p95_latency is total request latency, submit to final byte — the latency_ms field on request traces. To keep quiet scopes from firing on noise, error_rate and p95_latency evaluate only when the window holds at least 20 requests.

Alerts are evaluated about once a minute. One fires when its metric crosses the threshold, then stays silent until it re-arms — the metric falls back below the threshold, or, for billing_month spend alerts, the billing period resets. You get one notification per incident, not one per evaluation.

Channels

Webhook deliveries are signed exactly like job webhooks: an INFRO-Signature: t=...,v1=... header where v1 is the hex HMAC-SHA256 of {t}.{raw body} keyed by the channel's signing secret, plus INFRO-Event-Id for log correlation. The verifiers on the Webhooks page work unchanged — hash the raw bytes, compare in constant time, reject timestamps more than five minutes old. Delivery is at-least-once with retries for up to 24 hours, so deduplicate on the event id.

{
  "id": "evt_9d3c41ab27f0e812",
  "type": "alert.triggered",
  "created_at": 1755950700,
  "data": {
    "alert_id": "alert_4b0e7c2f",
    "metric": "error_rate",
    "scope": { "level": "project", "id": "proj_9c2ff1" },
    "threshold": 0.05,
    "observed": 0.11,
    "window": "5m",
    "triggered_at": 1755950700
  }
}

Create an alert

POST /v1/alerts creates one. The example watches a production project for a 5% error rate over five minutes and notifies the on-call address plus a webhook.

curl https://api.infro.io/v1/alerts \
  -H "Authorization: Bearer $INFRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "metric": "error_rate",
    "scope": {"level": "project", "id": "proj_9c2ff1"},
    "threshold": 0.05,
    "window": "5m",
    "channels": [
      {"type": "email", "to": "oncall@example.com"},
      {"type": "webhook", "url": "https://example.com/hooks/infro-alerts"}
    ]
  }'

{
  "id": "alert_4b0e7c2f",
  "metric": "error_rate",
  "scope": { "level": "project", "id": "proj_9c2ff1" },
  "threshold": 0.05,
  "window": "5m",
  "channels": [
    { "type": "email", "to": "oncall@example.com" },
    {
      "type": "webhook",
      "url": "https://example.com/hooks/infro-alerts",
      "secret": "whsec_Zk8QwR2m..."
    }
  ],
  "created_at": 1755950000,
  "last_triggered_at": null
}

The webhook channel's secret (prefixed whsec_) appears only here, in the creation response — never in GET /v1/alerts. Store it beside the endpoint that verifies it, like the API key itself. To rotate, create a replacement alert, deploy its new secret, then delete the old alert — both deliver during the overlap, and your dedupe on event id absorbs the duplicates.

List and delete alerts

GET /v1/alerts returns every alert in the organization, newest first. With the 100-alert cap there is no pagination. last_triggered_at tells you which alerts actually fire and which are dead weight.

curl https://api.infro.io/v1/alerts \
  -H "Authorization: Bearer $INFRO_API_KEY"

{
  "data": [
    {
      "id": "alert_4b0e7c2f",
      "metric": "error_rate",
      "scope": { "level": "project", "id": "proj_9c2ff1" },
      "threshold": 0.05,
      "window": "5m",
      "channels": [
        { "type": "email", "to": "oncall@example.com" },
        { "type": "webhook", "url": "https://example.com/hooks/infro-alerts" }
      ],
      "created_at": 1755950000,
      "last_triggered_at": 1755950700
    }
  ]
}

DELETE /v1/alerts/{id} returns 204 with no body and takes effect immediately — the alert is never evaluated again, though a notification already dispatched may still arrive. An unknown id returns 404.

curl -X DELETE https://api.infro.io/v1/alerts/alert_4b0e7c2f \
  -H "Authorization: Bearer $INFRO_API_KEY"

Recommended defaults

Three alerts cover most teams. Start with these, then tune thresholds against what usage analytics shows as your steady state.

• Spend at 80% of budget. spend, window billing_month, threshold at 0.8 × your monthly budget, org scope. If you set a spend limit with alert_at, the 50/80/100% notifications come with it — an explicit alert adds Slack and webhook routing and any threshold you like.
• Error-rate spike. error_rate, window 5m, threshold 0.05 on your production project. Automatic failover absorbs most upstream trouble, so a sustained spike above 5% usually means something real — a provider incident, a bad deploy, or a malformed request loop.
• p95 latency on interactive paths. p95_latency, window 1h, threshold at roughly twice the steady-state p95 for the key serving your UI. Latency drifts silently; a threshold catches the regression the dashboard would have shown you next week.

Alerts vs spend limits

Alerts and spend limits answer different questions. An alert observes: it tells you a number crossed a line and changes nothing about traffic. A spend limit enforces: with behavior set to "block", a request over the limit is refused with 402 before it reaches a model. Use both — an alert at 80% of budget gives a human time to react, and the limit at 100% caps the damage if nobody does. Per-key ceilings and 402 semantics are covered in Rate & spend limits.

When an alert fires, the per-request detail behind it is one query away: filter request traces to the same scope and time range, or pull the aggregate curve from usage analytics to see whether you caught a blip or a trend.