Skip to content

Routing and reliabilityComparison

Synthetic probes or passive checks: which catches trouble first?

A gateway can learn a provider is degrading from probes it sends or from the traffic it already carries. They fail in different ways, and the right mix depends on your request volume and how many standby targets sit idle.

By INFRO Editorial

Published

9 min read

Luminous routing paths connecting one application node to many AI model endpoints

In short

On a busy route, passive signals from live traffic catch trouble first, because they measure the exact requests users send. Synthetic probes win on idle or standby targets, where no traffic produces a signal, and for recovery checks. Most gateways need passive detection as the trigger and sparse probes as the backstop.

Key takeaways

  • Passive signals detect faster on busy routes because every user request is a sample of the exact path that matters.
  • Probes are the only signal for idle targets such as standbys, weight-0 targets and off-peak hours, and for deciding when to restore a tripped target.
  • A probe that differs from real traffic in model, prompt size, streaming or tools can stay green while users fail.
  • Trip on rate-over-window with a minimum request count and a latency or TTFT baseline, never on a single failure; keep 429s out of the failure rate.
  • Probes spend your own provider quota and tokens, so size interval times targets times keys before enabling them.

The two signals, and what each one measures

A gateway deciding whether a provider is healthy has two sources of evidence. Active probes are requests the gateway invents on a timer: a tiny completion, sent at a fixed interval to each target, whose outcome and timing feed a health state. Passive checks read the outcomes of requests that applications already sent: status codes, connection errors, total latency, and time to first token (TTFT) on streams.

They answer different questions. A probe asks "can this target answer a small request right now?" Passive signals ask "what is happening to the requests real users send?" The second question is the one that matters, but it can only be answered where traffic exists.

HTTP itself gives you little help here. The HTTP specification says the protocol cannot be defined in terms of what occurs behind the interface, only the syntax and expected behavior of messages.1 A 200 tells you the server answered; it says nothing about whether the answer was slow, truncated or from a degraded backend. That is why latency and TTFT belong in any health definition beside the status code.

Routers that rank providers already lean on passive data. OpenRouter says it tracks response times, error rates and availability across providers in real time and routes on that feedback,2 and its default routing uses each provider's measured price, throughput, latency and reliability.3 The open question for your own gateway is whether passive data is enough.

Detection delay: why passive usually wins on busy routes

Detection delay is the time between the provider starting to degrade and your gateway acting on it. For each method it has a simple shape.

  • Passive: the time to accumulate enough failing requests to clear your statistical bar. At high volume that is seconds, because every request is a sample.
  • Probe: up to one interval to land the first failing probe, plus the interval times the number of consecutive failures you require.

The gap reverses as traffic falls. At one request a minute, a passive window of 30 seconds holds zero or one sample, so a rule that needs volume can never fire. A probe every 15 seconds becomes the faster signal by a wide margin. The crossover is set by your request rate per target, not by the provider.

What probes cost

Probes are not free, even with tiny prompts. They spend your own provider quota and tokens, and you pay your provider for them directly.

Three side effects matter more than the token count:

  1. Quota. Probes count against requests-per-minute and tokens-per-minute limits. On a tight tier, aggressive probing causes the 429s you are trying to detect.
  2. Cache distortion. A fixed probe prompt is a perfect prompt-cache hit, so its TTFT is flattering and unrepresentative.
  3. Key blast radius. Probing with the production key means probe traffic shows up in that key's usage and any per-key limits. Probe per key, not per provider, or you will miss a single revoked or exhausted key.

Passive checks cost nothing extra in provider spend. Their cost is engineering: windowing, baselines and a place to store per-target counters.

False trips: how each method gets it wrong

Both methods can trip a healthy target, and both can stay green on a broken one.

FailurePassiveProbe
Healthy target trippedLow volume: 1 failure in 3 requests looks like 33%One dropped probe packet or a probe timeout shorter than real latency
Broken target stays greenNo traffic reaches it (standby, weight 0, off-peak)Probe differs from real traffic: short prompt, no tools, no streaming, small context
Misread causeCounts a 429 as an outage when it is your own quotaProbe is rate-limited by the quota it is consuming
Slow to noticeNeeds a minimum sample before actingNeeds interval times consecutive failures
Hidden biasSkewed by one noisy tenant or one huge prompt shapeAlways the same prompt, so never sees a long-context or tool-calling regression
Failure modes of passive checks and synthetic probes.

The most common passive mistake is treating throttling as failure. A 429 means your account hit a limit, not that the provider is unhealthy, and counting it will eject a perfectly good target at your own peak. INFRO's docs make the same distinction: a provider 429 does not count against the connection's health.4 The circuit breaker article covers that split in depth.

The most common probe mistake is a probe that is easier than reality. If it is a one-line non-streaming completion on a small model, it can pass while 128k-token tool-calling requests time out.

Signals worth reading passively

Use more than the error rate. A degrading provider often shows up in latency before it shows up in errors.

  • Error rate over a window, split by kind: 5xx, timeout, connection failure, rejected credential. Keep 429 as its own series.
  • TTFT against a per-target baseline. Total latency mixes generation length with provider health; TTFT isolates the provider's queueing. See measuring time to first token for how to set the baseline.
  • Stream interruptions, meaning streams that opened and then died. These never appear as an error status because the status was already sent. The streaming failure modes article covers the cases.
  • Latency percentile, not mean. OpenRouter's routing preferences express latency as percentile cutoffs (p50, p75, p90, p99),3 which is the right unit: a tail that moves while the median holds is an early warning.

OpenTelemetry's GenAI semantic conventions are the place to look for standard names for these measurements; the page in the main specification now points to a separate repository where they are maintained.5 Whatever names you use, emit the same attributes from every provider adapter so one rule can read all targets.

A minimal passive detector, with a volume gate and a TTFT baseline:

detector.py
from collections import deque
import time

class Window:
    def __init__(self, seconds=30):
        self.seconds, self.events = seconds, deque()

    def add(self, ok: bool, ttft_ms: float | None, throttled: bool):
        now = time.monotonic()
        self.events.append((now, ok, ttft_ms, throttled))
        while self.events and now - self.events[0][0] > self.seconds:
            self.events.popleft()

    def verdict(self, baseline_ttft_ms, min_requests=100,
                max_error_rate=0.2, ttft_factor=3.0):
        counted = [e for e in self.events if not e[3]]   # 429s excluded
        if len(counted) < min_requests:
            return "insufficient-data"                  # do not guess
        errors = sum(1 for e in counted if not e[1]) / len(counted)
        ttfts = sorted(e[2] for e in counted if e[2] is not None)
        p95 = ttfts[int(0.95 * (len(ttfts) - 1))] if ttfts else 0
        if errors > max_error_rate or p95 > ttft_factor * baseline_ttft_ms:
            return "degraded"
        return "healthy"

The insufficient-data branch is the important one. It is the point where probes should take over.

Where probes are the only signal

Passive detection has a structural blind spot: a target with no traffic has no health. Routes create such targets on purpose. A weighted route can hold a target at weight 0 as a standby that never takes a first attempt.6 A priority route's second target only sees traffic once the first fails. Standby keys are used only when the others fail.4 If the standby has been broken for a week, you find out at the worst moment: during the incident that made you need it.

So run probes on exactly these targets:

  • standby and weight-0 targets, including alternate hosts and spare keys;
  • targets that serve a tiny share of traffic, such as a canary at a few percent;
  • off-peak windows when a normally busy target goes quiet;
  • recovery checks on a tripped target, where you must decide when to send real traffic again without using users as the test.

For recovery, a probe or a small share of live traffic both work. Probing is safer for a target that failed hard, because a failed probe costs nothing. Sending a trickle of live traffic is better when the failure was latency, since the probe may not reproduce it.

Combine the two rather than choose one.

  1. Passive is the trigger on any target above your volume gate. It reacts in seconds and measures the real request shape.
  2. Probes are the floor. Run them at a slow interval (tens of seconds) on every target, but let them trip a target only when passive data is insufficient-data.
  3. Make probes look real. Match the production model, add streaming, and use a small set of prompt shapes, including one with a tool definition and one long enough to miss the cache. Record TTFT, not just success.
  4. Require agreement before ejecting a busy target. One signal alone raises a warning; a second, or a long enough run of the first, ejects.
  5. Restore gradually. Move from ejected to a small share of traffic, and only then to full weight.
  6. Alert on the gap. If a probe is red while passive data is green on a busy route, the probe is probably unrepresentative. Fix the probe rather than the route.

Where this meets a gateway

Some of this is already structural in a gateway that fails over on request outcomes. INFRO routes retry the same target 0 to 3 times with no backoff, move to the next target on any upstream failure before the response starts, and do not restart a stream once it has begun.7 Failure rules can decide per failure kind (rate limit, server error, timeout, connection, rejected key) whether to retry, move on or stop.7 Disabled and ejected connections are excluded from a route,7 and request traces record the route and the connection that served each request.6 That gives a passive record to tune thresholds against, and spans can be exported to your own collector. See the routing docs for how ejection is decided before you rely on a specific threshold.

Whatever the gateway does by default, the mechanics above are the checklist: gate on volume, exclude throttling, read TTFT, probe the idle targets, and drill the whole thing. A failover drill is the test that your detection delay is what you think it is.

Decision table

SituationPrimary signalAdd
High-volume production routePassive error rate and TTFTSlow probes as a floor
Standby, weight-0 or spare keyProbePassive once failover sends traffic
Low-volume or bursty routeProbePassive with a long window
Recovery of a tripped targetProbe, then a trickle of live trafficPassive on the trickle
Provider with tight rate limitsPassiveProbe only per key, at a long interval
Which signal leads, by situation.

Questions readers ask

Can I use the provider's public status page instead of probes?
Treat it as a human-readable hint, not a trigger. Status pages are updated by people after an incident is acknowledged, so they lag the traffic signal, and they rarely cover your model, region or request shape. Use them to confirm and to decide when to lift a manual override.
Should a probe use streaming?
Yes, if your production traffic streams. A non-streaming probe cannot measure time to first token, and some failures only appear once a stream is open. Keep the output to a few tokens so the cost stays negligible.
How many consecutive probe failures should trip a target?
Pick the number from the detection delay you can accept: interval multiplied by required failures. Two or three failures at a 15 to 30 second interval is a common starting point, but treat that as a place to begin tuning, not a standard.

From the team that builds it

Where this shows up in INFRO

INFRO is a control plane in front of the provider accounts you already hold. The capabilities below are the parts of it this article touches, each with the documentation that defines its behaviour.

INFRO accounts are self-serve. Provider usage is billed to you by your providers; INFRO charges a software subscription and takes 0% of model spend. See plans.

Sources

Numbered in order of first citation. Each was opened and checked against the claim on the date shown.

  1. 1.RFC 9110: HTTP Semantics (opens in a new tab) — IETF / RFC Editor. Accessed Nov 6, 2026.
  2. 2.Uptime Optimization (opens in a new tab) — OpenRouter Docs. Accessed Nov 6, 2026.
  3. 3.Provider Routing (opens in a new tab) — OpenRouter Docs. Accessed Nov 6, 2026.
  4. 4.Failover & fallbacks (opens in a new tab) — INFRO Docs. Accessed Nov 6, 2026.
  5. 5.Generative AI semantic conventions (moved) (opens in a new tab) — OpenTelemetry. Accessed Nov 6, 2026.
  6. 6.Routes (opens in a new tab) — INFRO Docs. Accessed Nov 6, 2026.
  7. 7.Smart routing (opens in a new tab) — INFRO Docs. Accessed Nov 6, 2026.

About this article

Read on 2026-11-06: INFRO docs overview, smart routing, failover and routes pages; OpenRouter's provider routing and uptime pages; RFC 9110; the OpenTelemetry GenAI conventions page, which now only points to a separate repository. OTel metric names, provider status-page behavior and INFRO's ejection thresholds were left out as unconfirmed. All numbers in examples are illustrative.

Last checked against its sources on . The next review is due by . No separate named reviewer is recorded for this article.

How articles are written, sourced and corrected is set out in the editorial standards.