# Synthetic probes or passive checks: which catches trouble first?

URL: https://infro.io/blog/synthetic-probes-or-passive-checks-which-catches-trouble-first
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-11-06
Last modified: 2026-11-06
Last reviewed: 2026-11-06 (next review due 2026-12-06)
Topic: Routing and reliability
Format: Comparison

> On a busy route, passive signals from live traffic catch trouble first, because they measure the exact requests users send. Synthetic probes win on idle or standby targets, where no traffic produces a signal, and for recovery checks. Most gateways need passive detection as the trigger and sparse probes as the backstop.

Key takeaways:
- Passive signals detect faster on busy routes because every user request is a sample of the exact path that matters.
- Probes are the only signal for idle targets such as standbys, weight-0 targets and off-peak hours, and for deciding when to restore a tripped target.
- A probe that differs from real traffic in model, prompt size, streaming or tools can stay green while users fail.
- Trip on rate-over-window with a minimum request count and a latency or TTFT baseline, never on a single failure; keep 429s out of the failure rate.
- Probes spend your own provider quota and tokens, so size interval times targets times keys before enabling them.

## The two signals, and what each one measures

A gateway deciding whether a provider is healthy has two sources of evidence. **Active probes** are requests the gateway invents on a timer: a tiny completion, sent at a fixed interval to each target, whose outcome and timing feed a health state. **Passive checks** read the outcomes of requests that applications already sent: status codes, connection errors, total latency, and time to first token (TTFT) on streams.

They answer different questions. A probe asks "can this target answer a small request right now?" Passive signals ask "what is happening to the requests real users send?" The second question is the one that matters, but it can only be answered where traffic exists.

HTTP itself gives you little help here. The HTTP specification says the protocol cannot be defined in terms of what occurs behind the interface, only the syntax and expected behavior of messages.[1] A `200` tells you the server answered; it says nothing about whether the answer was slow, truncated or from a degraded backend. That is why latency and TTFT belong in any health definition beside the status code.

Routers that rank providers already lean on passive data. OpenRouter says it tracks response times, error rates and availability across providers in real time and routes on that feedback,[2] and its default routing uses each provider's measured price, throughput, latency and reliability.[3] The open question for your own gateway is whether passive data is enough.

## Detection delay: why passive usually wins on busy routes

Detection delay is the time between the provider starting to degrade and your gateway acting on it. For each method it has a simple shape.

- **Passive:** the time to accumulate enough failing requests to clear your statistical bar. At high volume that is seconds, because every request is a sample.
- **Probe:** up to one interval to land the first failing probe, plus the interval times the number of consecutive failures you require.

> **Example: Illustrative detection delay**
> A route carries 40 requests per second. A provider starts failing about half of them. A passive rule that needs 20 failures in a 30-second window and at least 100 requests trips in roughly one second of traffic after the window has enough volume.
> 
> A probe sent every 30 seconds, requiring 3 consecutive failures, takes between 60 and 90 seconds to trip, and only if the probe fails the same way the traffic does. These numbers are made up to show the mechanism; your thresholds will differ.

The gap reverses as traffic falls. At one request a minute, a passive window of 30 seconds holds zero or one sample, so a rule that needs volume can never fire. A probe every 15 seconds becomes the faster signal by a wide margin. The crossover is set by your request rate per target, not by the provider.

## What probes cost

Probes are not free, even with tiny prompts. They spend your own provider quota and tokens, and you pay your provider for them directly.

> **Example: Illustrative probe load**
> One probe every 30 seconds is 2,880 probes a day per target. At about 20 input tokens and 1 output token each, that is roughly 60,000 tokens a day per target. Multiply by targets, models, keys and regions: 4 models on 3 keys across 2 regions is 24 targets, about 1.4 million tokens a day of pure monitoring. The money is small at most list rates; the rate-limit headroom it consumes is the larger concern on a low tier.

Three side effects matter more than the token count:

1. **Quota.** Probes count against requests-per-minute and tokens-per-minute limits. On a tight tier, aggressive probing causes the 429s you are trying to detect.
2. **Cache distortion.** A fixed probe prompt is a perfect prompt-cache hit, so its TTFT is flattering and unrepresentative.
3. **Key blast radius.** Probing with the production key means probe traffic shows up in that key's usage and any per-key limits. Probe per key, not per provider, or you will miss a single revoked or exhausted key.

Passive checks cost nothing extra in provider spend. Their cost is engineering: windowing, baselines and a place to store per-target counters.

## False trips: how each method gets it wrong

Both methods can trip a healthy target, and both can stay green on a broken one.

| Failure | Passive | Probe |
| --- | --- | --- |
| Healthy target tripped | Low volume: 1 failure in 3 requests looks like 33% | One dropped probe packet or a probe timeout shorter than real latency |
| Broken target stays green | No traffic reaches it (standby, weight 0, off-peak) | Probe differs from real traffic: short prompt, no tools, no streaming, small context |
| Misread cause | Counts a 429 as an outage when it is your own quota | Probe is rate-limited by the quota it is consuming |
| Slow to notice | Needs a minimum sample before acting | Needs interval times consecutive failures |
| Hidden bias | Skewed by one noisy tenant or one huge prompt shape | Always the same prompt, so never sees a long-context or tool-calling regression |

Table: Failure modes of passive checks and synthetic probes.

The most common passive mistake is treating throttling as failure. A 429 means your account hit a limit, not that the provider is unhealthy, and counting it will eject a perfectly good target at your own peak. INFRO's docs make the same distinction: a provider 429 does not count against the connection's health.[4] The [circuit breaker article](https://infro.io/blog/llm-circuit-breakers-throttling) covers that split in depth.

The most common probe mistake is a probe that is easier than reality. If it is a one-line non-streaming completion on a small model, it can pass while 128k-token tool-calling requests time out.

## Signals worth reading passively

Use more than the error rate. A degrading provider often shows up in latency before it shows up in errors.

- **Error rate over a window**, split by kind: 5xx, timeout, connection failure, rejected credential. Keep 429 as its own series.
- **TTFT against a per-target baseline.** Total latency mixes generation length with provider health; TTFT isolates the provider's queueing. See [measuring time to first token](https://infro.io/blog/measure-time-to-first-token) for how to set the baseline.
- **Stream interruptions**, meaning streams that opened and then died. These never appear as an error status because the status was already sent. The [streaming failure modes](https://infro.io/blog/streaming-llm-failure-modes) article covers the cases.
- **Latency percentile, not mean.** OpenRouter's routing preferences express latency as percentile cutoffs (p50, p75, p90, p99),[3] which is the right unit: a tail that moves while the median holds is an early warning.

OpenTelemetry's GenAI semantic conventions are the place to look for standard names for these measurements; the page in the main specification now points to a separate repository where they are maintained.[5] Whatever names you use, emit the same attributes from every provider adapter so one rule can read all targets.

A minimal passive detector, with a volume gate and a TTFT baseline:

```python
from collections import deque
import time

class Window:
    def __init__(self, seconds=30):
        self.seconds, self.events = seconds, deque()

    def add(self, ok: bool, ttft_ms: float | None, throttled: bool):
        now = time.monotonic()
        self.events.append((now, ok, ttft_ms, throttled))
        while self.events and now - self.events[0][0] > self.seconds:
            self.events.popleft()

    def verdict(self, baseline_ttft_ms, min_requests=100,
                max_error_rate=0.2, ttft_factor=3.0):
        counted = [e for e in self.events if not e[3]]   # 429s excluded
        if len(counted) < min_requests:
            return "insufficient-data"                  # do not guess
        errors = sum(1 for e in counted if not e[1]) / len(counted)
        ttfts = sorted(e[2] for e in counted if e[2] is not None)
        p95 = ttfts[int(0.95 * (len(ttfts) - 1))] if ttfts else 0
        if errors > max_error_rate or p95 > ttft_factor * baseline_ttft_ms:
            return "degraded"
        return "healthy"
```

The `insufficient-data` branch is the important one. It is the point where probes should take over.

## Where probes are the only signal

Passive detection has a structural blind spot: a target with no traffic has no health. Routes create such targets on purpose. A weighted route can hold a target at weight 0 as a standby that never takes a first attempt.[6] A priority route's second target only sees traffic once the first fails. Standby keys are used only when the others fail.[4] If the standby has been broken for a week, you find out at the worst moment: during the incident that made you need it.

So run probes on exactly these targets:

- standby and weight-0 targets, including alternate hosts and spare keys;
- targets that serve a tiny share of traffic, such as a canary at a few percent;
- off-peak windows when a normally busy target goes quiet;
- recovery checks on a tripped target, where you must decide when to send real traffic again without using users as the test.

For recovery, a probe or a small share of live traffic both work. Probing is safer for a target that failed hard, because a failed probe costs nothing. Sending a trickle of live traffic is better when the failure was latency, since the probe may not reproduce it.

## Recommended design

Combine the two rather than choose one.

1. **Passive is the trigger** on any target above your volume gate. It reacts in seconds and measures the real request shape.
2. **Probes are the floor.** Run them at a slow interval (tens of seconds) on every target, but let them trip a target only when passive data is `insufficient-data`.
3. **Make probes look real.** Match the production model, add streaming, and use a small set of prompt shapes, including one with a tool definition and one long enough to miss the cache. Record TTFT, not just success.
4. **Require agreement before ejecting a busy target.** One signal alone raises a warning; a second, or a long enough run of the first, ejects.
5. **Restore gradually.** Move from ejected to a small share of traffic, and only then to full weight.
6. **Alert on the gap.** If a probe is red while passive data is green on a busy route, the probe is probably unrepresentative. Fix the probe rather than the route.

> **Warning: Do not probe from one place only**
> A probe from a single host or region reports a network path problem as a provider outage. If your traffic leaves through several regions or egress paths, probe from each, or compare against passive data from that path.

## Where this meets a gateway

Some of this is already structural in a gateway that fails over on request outcomes. INFRO routes retry the same target 0 to 3 times with no backoff, move to the next target on any upstream failure before the response starts, and do not restart a stream once it has begun.[7] Failure rules can decide per failure kind (rate limit, server error, timeout, connection, rejected key) whether to retry, move on or stop.[7] Disabled and ejected connections are excluded from a route,[7] and request traces record the route and the connection that served each request.[6] That gives a passive record to tune thresholds against, and spans can be exported to your own collector. See the routing docs for how ejection is decided before you rely on a specific threshold.

Whatever the gateway does by default, the mechanics above are the checklist: gate on volume, exclude throttling, read TTFT, probe the idle targets, and drill the whole thing. A [failover drill](https://infro.io/blog/llm-failover-drills) is the test that your detection delay is what you think it is.

## Decision table

| Situation | Primary signal | Add |
| --- | --- | --- |
| High-volume production route | Passive error rate and TTFT | Slow probes as a floor |
| Standby, weight-0 or spare key | Probe | Passive once failover sends traffic |
| Low-volume or bursty route | Probe | Passive with a long window |
| Recovery of a tripped target | Probe, then a trickle of live traffic | Passive on the trickle |
| Provider with tight rate limits | Passive | Probe only per key, at a long interval |

Table: Which signal leads, by situation.

## Questions

### Can I use the provider's public status page instead of probes?

Treat it as a human-readable hint, not a trigger. Status pages are updated by people after an incident is acknowledged, so they lag the traffic signal, and they rarely cover your model, region or request shape. Use them to confirm and to decide when to lift a manual override.

### Should a probe use streaming?

Yes, if your production traffic streams. A non-streaming probe cannot measure time to first token, and some failures only appear once a stream is open. Keep the output to a few tokens so the cost stays negligible.

### How many consecutive probe failures should trip a target?

Pick the number from the detection delay you can accept: interval multiplied by required failures. Two or three failures at a 15 to 30 second interval is a common starting point, but treat that as a place to begin tuning, not a standard.

## Sources

1. RFC 9110: HTTP Semantics — IETF / RFC Editor. https://www.rfc-editor.org/rfc/rfc9110.html (accessed 2026-11-06)
2. Uptime Optimization — OpenRouter Docs. https://openrouter.ai/docs/guides/best-practices/uptime-optimization (accessed 2026-11-06)
3. Provider Routing — OpenRouter Docs. https://openrouter.ai/docs/guides/routing/provider-selection (accessed 2026-11-06)
4. Failover & fallbacks — INFRO Docs. https://infro.io/docs/features/fallbacks (accessed 2026-11-06)
5. Generative AI semantic conventions (moved) — OpenTelemetry. https://opentelemetry.io/docs/specs/semconv/gen-ai/ (accessed 2026-11-06)
6. Routes — INFRO Docs. https://infro.io/docs/routes (accessed 2026-11-06)
7. Smart routing — INFRO Docs. https://infro.io/docs/features/routing (accessed 2026-11-06)
