Classifiers in the LLM request path: route, guard, monitor

Most production LLM systems end up with small models around the large one: something that decides which model should answer, something that decides whether the request should be answered at all, something that orders retrieved passages, something that scores answers after the fact. Until recently these were embedding classifiers, fine-tuned BERT models or a second LLM asked for JSON. Since September 2026 there is a model category built for exactly this job — decision models that return probabilities for typed questions and generate no text — and several vendors document them for classification, routing, gating and grading¹².

This guide is engine-neutral. It covers where a classifier belongs in the request path, what it costs in time and money, and what happens when it is slow, unsure or down. The last third shows how INFRO implements four of these placements and what it records. For the engines themselves, see Decision models compared.

Four places a classifier can sit

1. Guard: a yes/no question about the prompt — is this an injection, a jailbreak, off-topic — that can refuse the request before the model sees it.
2. Route: a choice among classes — small or large, code or prose, which language — that decides which model or provider serves the request.
3. Rerank: a relevance judgment for each retrieved passage, so only the useful ones reach the prompt.
4. Monitor: a rubric score of the finished answer, recorded for quality tracking.

The test for whether a classifier belongs inline is simple: does its answer change what happens to this request? A guard and a router do; the request is refused or sent elsewhere depending on the answer. A monitor does not; nothing about this request changes when it scores 2 out of 4. That one distinction decides the latency budget, the failure behaviour and where the call is made.

Routing by request class

The case for a prompt classifier is that most traffic does not need the most capable model. A router picks a class for each prompt and the routing layer maps classes to targets: lookups and greetings to a small model, multi-step reasoning to a large one. RouteLLM, a well-known paper on learned routers, reports cost cut by more than half on standard benchmarks in some settings without loss of response quality, using routers trained on preference data³. Your traffic is not a benchmark, so treat that as a reason to test, not a forecast.

Three design rules make routing classifiers safe:

• Route to classes, not models. The classifier answers "is this hard?"; the route decides that hard means a particular model this quarter. Swapping a model then needs no change to the classifier or its thresholds.
• Set a confidence floor and a fallback class. When the classifier is unsure, send the request where a mistake is cheapest. For routing that is usually the larger model: an unnecessary large-model call costs money, a wrongly downgraded answer costs the user.
• Keep the classes few and describable. Two to five classes with a one-line description each are easier to label, evaluate and reason about than twenty.

Guardrail checks on meaning

Pattern rules catch formats: card numbers, keys, email addresses. They cannot read intent. A prompt injection, a jailbreak framed as fiction or an off-topic request is a matter of meaning, which is where a classifier earns its place in front of the model. The deeper treatment of input screening is in Jev for LLM guardrails and model routing and prompt injection defense in depth; the request-path questions are narrower:

• Run patterns first. Redact secrets and personal data before the text goes to any classifier, so the guard does not become a new place the data travels.
• Two thresholds per check. A high threshold that blocks and a lower one that flags for review. A single threshold forces you to choose between letting attacks through and refusing legitimate users.
• Batch the checks. Ask every check about the same state in one call. Most decision models take many questions per request, so five checks cost one round trip, not five.
• Expect misses. Microsoft's own documentation for its decision model says that, used as a safety filter, it might miss subtle harmful content or flag benign content¹. A classifier is one layer, not a guarantee.

Rerank: a request of its own

Reranking orders candidate passages by relevance to a query so only the best reach the prompt. Cohere's v2 API shaped most clients: a query, a list of documents, an optional top_n, and results with each document's original index and a relevance_score between 0 and 1⁴. A decision model can serve the same contract by asking one yes/no question per passage — does this passage help answer the query? — and returning the probability of yes as the score.

Unlike a guard or router, rerank is not something that happens to another request. Your retrieval code calls it explicitly, so it lives in your latency budget for retrieval, not in a gateway's hidden path. Judge each passage alone rather than all at once: it keeps the state small and stops one long passage from crowding the others.

Evals on live traffic belong in the background

Scoring production answers against a rubric is how you notice that a model change, a prompt change or a provider's silent update has lowered quality. Using a model as the judge works: the MT-Bench study found strong LLM judges agreeing with human preferences over 80% of the time, about the rate at which humans agree with each other, while naming position, verbosity and self-enhancement biases to control for⁵.

None of that needs to happen while the user waits. A monitor should run after the response has been returned, on a sample, with its own generous timeout, and its failures should be invisible to the caller. A missed eval is a gap in a chart; an eval that fails a request is an outage you caused yourself.

The latency and cost budget of an extra call

Every inline classifier adds one round trip to the provider that serves it, in series with everything after it. Two inline classifiers — a guard, then a router that reads the guard's redacted prompt — add two. What matters is the tail: if your p95 budget for time to first token has a fixed allowance for pre-processing, the classifiers' combined p95 plus their timeouts has to fit inside it.

The cost side is easier to reason about. Decision models bill input tokens only, so a classifier call costs roughly its state plus its question text times the input rate. Short states and one call for many checks keep it small; the place it grows is a router reading the whole conversation instead of the last user message. Whatever the rate, put the classifier's cost on its own line. A request that "costs more since we added routing" should be explainable in one query.

Failure modes, and deciding them in advance

An inline classifier fails in three ways: it errors, it times out, or it answers with low confidence. Each needs a pre-decided behaviour:

• Safety checks fail closed by default. If the injection check cannot run, refuse the request and say why, with an error that distinguishes "blocked" from "check unavailable". Some products will choose to fail open and log the gap instead; make that a written decision per check, not a default someone discovers.
• Routers fail open to a default. A router that cannot answer should send the request to a safe default class, typically the capable model, never stall it.
• Low confidence is a fallback, not an answer. A choice with 0.4 on one option and 0.35 on another is not a decision. A confidence floor turns it into the fallback class, a review flag or a human queue. Whether the probabilities mean what they say is something to check on labelled data: neural networks are often overconfident until recalibrated⁶.
• Never let the classifier recurse. A router whose own decision model sits behind a routed endpoint can loop. The call a router makes should bypass routing classifiers.

Classifiers in INFRO: request shapes and records

INFRO runs all four placements on a decision model you connect: TypeSafe's Jev by default (typesafe/jev-1.13), or any other decision model on your own account with its provider — OpenAI, Cloudflare, Perplexity, Inception, Liquid AI, Upstage, Microsoft-Decision-1 on Azure, or OpenRouter. Your provider bills each call; INFRO charges nothing per decision. Any placement can name a route instead of a model, so a classifier fails over across keys or providers like any other request.

Guardrail classifier checks

A guardrail policy takes a classifiers block: a model, the input to read (last_user, or user_and_tool for every user and tool message), on_error, timeout_ms (300 to 5,000, default 2,000) and up to 10 checks, each with instructions and a block_at and flag_at threshold on the probability of yes. Presets cover prompt injection, jailbreaks, harmful requests, self-harm (flag only by default) and off-topic requests; every question and threshold can be edited. All checks of every policy that applies go in one call, after the pattern rules have redacted the prompt and before the budget check and any provider call. Responses are not classified.

A block is 403 guardrail_blocked, naming the check and its score. When the classifier cannot answer, on_error: "block" (the default) refuses with 403 guardrail_unavailable; "allow" serves the request and records a finding with check unavailable. Guardrails, classifier checks included, are part of the Business plan and above. Full contract: Guardrails.

The route's prompt classifier

{
  "classifier": {
    "model": "typesafe/jev-1.13",
    "instructions": "How much reasoning does answering this request need?",
    "classes": [
      { "key": "small", "description": "A lookup, a greeting or a short factual answer." },
      { "key": "large", "description": "Multi-step reasoning, hard code or careful analysis." }
    ],
    "input": "last_user",
    "timeout_ms": 1500,
    "min_confidence": 0.5,
    "fallback_class": "large"
  },
  "rules": [
    {
      "name": "Hard prompts",
      "conditions": [{ "field": "class", "op": "eq", "value": "large" }],
      "targets": [{ "connection_id": "conn_…", "model": "anthropic/claude-opus-5-5" }]
    }
  ]
}

A route takes 2 to 20 classes. The classifier runs only when an enabled rule reads class or class_confidence, reads the prompt after guardrail redaction, and never receives images. A timeout, an error or a confidence under min_confidence gives fallback_class; with none, the request has no class and the route's default targets serve it, so routing never waits on a broken classifier. The response names the result in X-INFRO-Route-Class (for example large; confidence=0.91; source=model). A rule that tests a class the classifier cannot pick is refused when the route is saved. See Routes.

Rerank

POST /v1/rerank follows Cohere's shape: model, a query of up to 8 KB, 1 to 200 documents of up to 64 KB each, top_n, return_documents, an optional instructions that replaces the relevance question for your domain, and concurrency (1 to 16, default 8) for how many passages are judged at once on your connection. Each passage is one decision call with the query; the response returns results with index and relevance_score, plus usage.cost and the connection that served it. The request is one row in the request log with the passage count and the top score; the query and passages are not stored. Reference: Rerank.

Eval monitors

An eval monitor is a decision policy with a monitor block: enabled, a sample_rate from 0.001 to 1, and optional project_ids and models to cover. After a chat completion has been returned, a sampled share is scored in the background: the policy's question is asked of {prompt, response}, the latest user message and the model's answer. Each score is its own call, recorded under the request it judged, and the policy's console page shows the mean score per served model. Answers served from INFRO's response cache are not scored again, an eval that fails is skipped, and an organization can run five monitors at once. See Decisions API: eval monitors.

What is recorded

Each call INFRO makes for another request — a guardrail check, a route's class, an eval — is its own row in the request log, with its own estimated provider cost, linked to the request it was made for. On GET /v1/requests such a row carries purpose (guardrail_classifier, route_classifier or eval) and parent_request, plus a decision summary: the class picked, the checks' probabilities, or the eval's outcome and band. The prompt, the state and the response are never recorded by these features; findings keep the check, the scope that defined it and the probability, never the text. That is the property the rest of this guide argues for: a classifier's cost and its failures stay visible on their own lines, attributed to the same project, key and workflow as the request they served, instead of disappearing into it.