# Classifiers in the LLM request path: route, guard, monitor

URL: https://infro.io/blog/classifiers-in-the-request-path
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-10-13
Last modified: 2026-10-13
Last reviewed: 2026-10-13 (next review due 2026-11-12)
Topic: Architecture and migration
Format: Guide

> Put a classifier inline only when its answer changes what happens to the request — which model serves it, or whether it is served at all — and give it a hard timeout and a fallback decided in advance: fail closed for a safety check, fail open to a default for routing. Ordering context (rerank) is a request of its own, and quality scoring belongs in the background on a sample, never in the caller's path. Record every classifier call as its own row linked to the request it served, so its cost and its failures stay visible.

Key takeaways:
- Inline classifiers are for decisions that change the request: which model serves it, or whether it is served. Everything else moves out of the path.
- Every inline classifier needs a timeout and a failure behaviour chosen before launch: fail closed for safety checks, fail open to a default for routing.
- A confidence floor turns an unsure answer into a known fallback instead of a coin toss.
- Rerank is its own request; quality evals run after the response, on a sample.
- Record each classifier call as its own row linked to its parent request, so the classifier's cost and outages never hide inside the main call's numbers.

Most production LLM systems end up with small models around the large one: something that decides which model should answer, something that decides whether the request should be answered at all, something that orders retrieved passages, something that scores answers after the fact. Until recently these were embedding classifiers, fine-tuned BERT models or a second LLM asked for JSON. Since September 2026 there is a model category built for exactly this job — decision models that return probabilities for typed questions and generate no text — and several vendors document them for classification, routing, gating and grading[1][2].

This guide is engine-neutral. It covers where a classifier belongs in the request path, what it costs in time and money, and what happens when it is slow, unsure or down. The last third shows how INFRO implements four of these placements and what it records. For the engines themselves, see [Decision models compared](https://infro.io/blog/decision-models-compared-2026).

## Four places a classifier can sit

[Diagram: One request, in order. Only the first three run before the model call; the eval runs afterwards.]

1. **Guard**: a yes/no question about the prompt — is this an injection, a jailbreak, off-topic — that can refuse the request before the model sees it.
2. **Route**: a choice among classes — small or large, code or prose, which language — that decides which model or provider serves the request.
3. **Rerank**: a relevance judgment for each retrieved passage, so only the useful ones reach the prompt.
4. **Monitor**: a rubric score of the finished answer, recorded for quality tracking.

The test for whether a classifier belongs *inline* is simple: does its answer change what happens to this request? A guard and a router do; the request is refused or sent elsewhere depending on the answer. A monitor does not; nothing about this request changes when it scores 2 out of 4. That one distinction decides the latency budget, the failure behaviour and where the call is made.

## Routing by request class

The case for a prompt classifier is that most traffic does not need the most capable model. A router picks a class for each prompt and the routing layer maps classes to targets: lookups and greetings to a small model, multi-step reasoning to a large one. RouteLLM, a well-known paper on learned routers, reports cost cut by more than half on standard benchmarks in some settings without loss of response quality, using routers trained on preference data[3]. Your traffic is not a benchmark, so treat that as a reason to test, not a forecast.

Three design rules make routing classifiers safe:

- **Route to classes, not models.** The classifier answers "is this hard?"; the route decides that hard means a particular model this quarter. Swapping a model then needs no change to the classifier or its thresholds.
- **Set a confidence floor and a fallback class.** When the classifier is unsure, send the request where a mistake is cheapest. For routing that is usually the larger model: an unnecessary large-model call costs money, a wrongly downgraded answer costs the user.
- **Keep the classes few and describable.** Two to five classes with a one-line description each are easier to label, evaluate and reason about than twenty.

## Guardrail checks on meaning

Pattern rules catch formats: card numbers, keys, email addresses. They cannot read intent. A prompt injection, a jailbreak framed as fiction or an off-topic request is a matter of meaning, which is where a classifier earns its place in front of the model. The deeper treatment of input screening is in [Jev for LLM guardrails and model routing](https://infro.io/blog/jev-llm-guardrails-model-routing) and [prompt injection defense in depth](https://infro.io/blog/prompt-injection-defense-in-depth); the request-path questions are narrower:

- **Run patterns first.** Redact secrets and personal data before the text goes to any classifier, so the guard does not become a new place the data travels.
- **Two thresholds per check.** A high threshold that blocks and a lower one that flags for review. A single threshold forces you to choose between letting attacks through and refusing legitimate users.
- **Batch the checks.** Ask every check about the same state in one call. Most decision models take many questions per request, so five checks cost one round trip, not five.
- **Expect misses.** Microsoft's own documentation for its decision model says that, used as a safety filter, it might miss subtle harmful content or flag benign content[1]. A classifier is one layer, not a guarantee.

## Rerank: a request of its own

Reranking orders candidate passages by relevance to a query so only the best reach the prompt. Cohere's v2 API shaped most clients: a `query`, a list of `documents`, an optional `top_n`, and `results` with each document's original `index` and a `relevance_score` between 0 and 1[4]. A decision model can serve the same contract by asking one yes/no question per passage — does this passage help answer the query? — and returning the probability of yes as the score.

Unlike a guard or router, rerank is not something that happens to another request. Your retrieval code calls it explicitly, so it lives in your latency budget for retrieval, not in a gateway's hidden path. Judge each passage alone rather than all at once: it keeps the state small and stops one long passage from crowding the others.

## Evals on live traffic belong in the background

Scoring production answers against a rubric is how you notice that a model change, a prompt change or a provider's silent update has lowered quality. Using a model as the judge works: the MT-Bench study found strong LLM judges agreeing with human preferences over 80% of the time, about the rate at which humans agree with each other, while naming position, verbosity and self-enhancement biases to control for[5].

None of that needs to happen while the user waits. A monitor should run after the response has been returned, on a sample, with its own generous timeout, and its failures should be invisible to the caller. A missed eval is a gap in a chart; an eval that fails a request is an outage you caused yourself.

## The latency and cost budget of an extra call

Every inline classifier adds one round trip to the provider that serves it, in series with everything after it. Two inline classifiers — a guard, then a router that reads the guard's redacted prompt — add two. What matters is the tail: if your p95 budget for time to first token has a fixed allowance for pre-processing, the classifiers' combined p95 plus their timeouts has to fit inside it.

> **Example: Budgeting two inline classifiers**
> Hypothetical numbers. A chat product allows 400 ms before the model call starts. Pattern rules take a few milliseconds. If the guard's p95 is 150 ms and the router's is 120 ms, the p95 sum of 270 ms fits. But the timeouts decide the worst case: a guard timeout of 2,000 ms means a slow guard can hold a request for two seconds before its failure behaviour applies. Set each timeout from the budget, not from the classifier's typical speed, and measure both on your own traffic before shipping.

The cost side is easier to reason about. Decision models bill input tokens only, so a classifier call costs roughly its state plus its question text times the input rate. Short states and one call for many checks keep it small; the place it grows is a router reading the whole conversation instead of the last user message. Whatever the rate, put the classifier's cost on its own line. A request that "costs more since we added routing" should be explainable in one query.

## Failure modes, and deciding them in advance

[Diagram: The failure behaviour follows from what the classifier protects.]

An inline classifier fails in three ways: it errors, it times out, or it answers with low confidence. Each needs a pre-decided behaviour:

- **Safety checks fail closed by default.** If the injection check cannot run, refuse the request and say why, with an error that distinguishes "blocked" from "check unavailable". Some products will choose to fail open and log the gap instead; make that a written decision per check, not a default someone discovers.
- **Routers fail open to a default.** A router that cannot answer should send the request to a safe default class, typically the capable model, never stall it.
- **Low confidence is a fallback, not an answer.** A choice with 0.4 on one option and 0.35 on another is not a decision. A confidence floor turns it into the fallback class, a review flag or a human queue. Whether the probabilities mean what they say is something to check on labelled data: neural networks are often overconfident until recalibrated[6].
- **Never let the classifier recurse.** A router whose own decision model sits behind a routed endpoint can loop. The call a router makes should bypass routing classifiers.

[Diagram: INFRO's placements side by side, with the limits and failure behaviour its docs and gateway set.]

## Classifiers in INFRO: request shapes and records

INFRO runs all four placements on a decision model you connect: TypeSafe's Jev by default (`typesafe/jev-1.13`), or any other decision model on your own account with its provider — OpenAI, Cloudflare, Perplexity, Inception, Liquid AI, Upstage, Microsoft-Decision-1 on Azure, or OpenRouter. Your provider bills each call; INFRO charges nothing per decision. Any placement can name a route instead of a model, so a classifier fails over across keys or providers like any other request.

### Guardrail classifier checks

A guardrail policy takes a `classifiers` block: a `model`, the `input` to read (`last_user`, or `user_and_tool` for every user and tool message), `on_error`, `timeout_ms` (300 to 5,000, default 2,000) and up to 10 `checks`, each with `instructions` and a `block_at` and `flag_at` threshold on the probability of yes. Presets cover prompt injection, jailbreaks, harmful requests, self-harm (flag only by default) and off-topic requests; every question and threshold can be edited. All checks of every policy that applies go in one call, after the pattern rules have redacted the prompt and before the budget check and any provider call. Responses are not classified.

A block is `403 guardrail_blocked`, naming the check and its score. When the classifier cannot answer, `on_error: "block"` (the default) refuses with `403 guardrail_unavailable`; `"allow"` serves the request and records a finding with check `unavailable`. Guardrails, classifier checks included, are part of the Business plan and above. Full contract: [Guardrails](https://infro.io/docs/features/guardrails#classifier-checks).

### The route's prompt classifier

```json
{
  "classifier": {
    "model": "typesafe/jev-1.13",
    "instructions": "How much reasoning does answering this request need?",
    "classes": [
      { "key": "small", "description": "A lookup, a greeting or a short factual answer." },
      { "key": "large", "description": "Multi-step reasoning, hard code or careful analysis." }
    ],
    "input": "last_user",
    "timeout_ms": 1500,
    "min_confidence": 0.5,
    "fallback_class": "large"
  },
  "rules": [
    {
      "name": "Hard prompts",
      "conditions": [{ "field": "class", "op": "eq", "value": "large" }],
      "targets": [{ "connection_id": "conn_…", "model": "anthropic/claude-opus-5-5" }]
    }
  ]
}
```

A route takes 2 to 20 classes. The classifier runs only when an enabled rule reads `class` or `class_confidence`, reads the prompt after guardrail redaction, and never receives images. A timeout, an error or a confidence under `min_confidence` gives `fallback_class`; with none, the request has no class and the route's default targets serve it, so routing never waits on a broken classifier. The response names the result in `X-INFRO-Route-Class` (for example `large; confidence=0.91; source=model`). A rule that tests a class the classifier cannot pick is refused when the route is saved. See [Routes](https://infro.io/docs/routes#prompt-classifier).

### Rerank

[Diagram: POST /v1/rerank, as INFRO's rerank reference documents it.]

`POST /v1/rerank` follows Cohere's shape: `model`, a `query` of up to 8 KB, 1 to 200 `documents` of up to 64 KB each, `top_n`, `return_documents`, an optional `instructions` that replaces the relevance question for your domain, and `concurrency` (1 to 16, default 8) for how many passages are judged at once on your connection. Each passage is one decision call with the query; the response returns `results` with `index` and `relevance_score`, plus `usage.cost` and the connection that served it. The request is one row in the request log with the passage count and the top score; the query and passages are not stored. Reference: [Rerank](https://infro.io/docs/api/rerank).

### Eval monitors

An eval monitor is a decision policy with a `monitor` block: `enabled`, a `sample_rate` from 0.001 to 1, and optional `project_ids` and `models` to cover. After a chat completion has been returned, a sampled share is scored in the background: the policy's question is asked of `{prompt, response}`, the latest user message and the model's answer. Each score is its own call, recorded under the request it judged, and the policy's console page shows the mean score per served model. Answers served from INFRO's response cache are not scored again, an eval that fails is skipped, and an organization can run five monitors at once. See [Decisions API: eval monitors](https://infro.io/docs/api/decisions).

### What is recorded

[Diagram: A chat request and the calls INFRO made for it, as rows in the request log.]

Each call INFRO makes for another request — a guardrail check, a route's class, an eval — is its own row in the request log, with its own estimated provider cost, linked to the request it was made for. On `GET /v1/requests` such a row carries `purpose` (`guardrail_classifier`, `route_classifier` or `eval`) and `parent_request`, plus a `decision` summary: the class picked, the checks' probabilities, or the eval's outcome and band. The prompt, the state and the response are never recorded by these features; findings keep the check, the scope that defined it and the probability, never the text. That is the property the rest of this guide argues for: a classifier's cost and its failures stay visible on their own lines, attributed to the same project, key and workflow as the request they served, instead of disappearing into it.

## Sources

1. Deploy and use Microsoft-Decision-1 in Microsoft Foundry — Microsoft. https://learn.microsoft.com/en-us/azure/foundry/foundry-models/how-to/use-foundry-models-microsoft-decision (accessed 2026-10-13)
2. Decisions API guide — OpenAI. https://developers.openai.com/api/docs/guides/decisions (accessed 2026-10-13)
3. RouteLLM: Learning to Route LLMs with Preference Data — Ong et al., arXiv. https://arxiv.org/abs/2406.18665 (accessed 2026-10-13)
4. Rerank API (v2) — Cohere. https://docs.cohere.com/reference/rerank (accessed 2026-10-13)
5. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., NeurIPS 2023. https://arxiv.org/abs/2306.05685 (accessed 2026-10-13)
6. On Calibration of Modern Neural Networks — Guo, Pleiss, Sun and Weinberger (ICML 2017). https://arxiv.org/abs/1706.04599 (accessed 2026-10-13)