# LLM guardrails at the gateway: checks before and after

URL: https://infro.io/blog/llm-guardrails-at-the-gateway
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-10-13
Last modified: 2026-10-13
Last reviewed: 2026-10-13 (next review due 2026-11-12)
Topic: Security and governance
Format: Guide

> LLM guardrails are checks that run before a model call (on the prompt) and after it (on the response), each with an action: block, redact or record. Deterministic checks for secrets and known PII formats, and allow or block lists for models, belong at a gateway every application shares; meaning-level checks such as prompt-injection signals need a classifier and tuned thresholds; authorization of what the model may do stays in the application. Decide in advance whether a check that cannot run fails open or closed, and never store the text a check matched.

Key takeaways:
- A guardrail is a check plus an action — block, redact or record — at a known point in the request. Write down which, and where, before you write the check.
- Pattern checks for secrets and well-formed PII are cheap, deterministic and belong where every application's traffic passes. They miss anything the pattern does not describe.
- Prompt injection and off-topic use are matters of meaning: they need a classifier with thresholds you tune on your own traffic, and OWASP says there may be no fool-proof prevention.
- Output checks run after the provider has generated, and billed, the response. On a stream, text already sent cannot be recalled.
- Decide per check whether it fails open or closed, and alert on every request that passed unchecked.
- Log that a check fired, which detector and how often; never the matched text.

"Guardrails" covers a lot of different things: a regular expression that catches card numbers, a provider's harm classifier, a list of approved models, a schema validator on tool arguments, a human approval step. They have one thing in common. Each is a check that runs at a known point in a request and has a decision attached to it. This guide is about the checks that sit on the path of the model call itself, on the prompt before it leaves and on the response before it returns: what they can catch, where each one belongs, what it costs, and what should happen when it cannot run.

## What a guardrail is, and what it is not

A useful working definition has three parts: **what is inspected** (which fields, which endpoints, which stage), **what is detected** (a pattern, a list, a classifier score) and **what happens** (refuse, rewrite, or record and continue). If you cannot write those three down for a guardrail, you cannot test it, and you cannot explain an incident involving it.

What a guardrail is not is a safety guarantee. OWASP's entry on prompt injection, the first risk in its 2025 Top 10 for LLM Applications[1], says it is unclear whether fool-proof prevention exists, and treats input and output filtering as one mitigation among several, beside privilege control, human approval for high-risk actions, segregating untrusted content and adversarial testing[2]. A filter reduces what gets through. The controls that limit damage when something does get through are authorization and blast radius, and those are covered in [defense in depth for prompt injection](https://infro.io/blog/prompt-injection-defense-in-depth). This article covers the filter layer and keeps that boundary in view.

## Input checks

Input checks run on the prompt before the provider sees it. Four families cover most needs.

**Secrets.** API keys, tokens and private keys pasted into prompts are common and dangerous: the prompt goes to a third party, may be logged there, and may be logged by you. Secret formats are a good fit for deterministic detection because most vendors use recognizable prefixes (`sk-`, `AKIA`, `ghp_`, `xoxb-`) or structures (JWTs, PEM blocks). The gap is secrets with no prefix, such as an AWS secret access key or a database password, which no pattern can recognize reliably.

**Personal data.** OWASP's sensitive information disclosure entry recommends input validation that detects and filters sensitive data before it reaches the model, alongside sanitization, tokenization or redaction[3]. Pattern detection works for data with a format and a checksum: card numbers pass a Luhn check, IBANs a mod-97 check, emails and phone numbers have recognizable shapes. It does not work for names, street addresses, dates of birth or anything written out in words. NIST's generative AI profile adds that what counts as sensitive data can be highly contextual[4]. A pattern detector finds formats. It does not know what is sensitive to you.

**Prompt-injection and jailbreak signals.** These are about meaning, not format. Text that tells the assistant to ignore its instructions can be phrased in endless ways, encoded or split across a retrieved document. Detecting it takes a classifier: a provider's (Azure's Prompt Shields classify user prompt attacks and indirect attacks embedded in documents[5]), an open model you host, or a decision model answering a yes/no question. All of them return a score or a label, and you choose the threshold.

**Topic and policy.** "Is this request within what this application is for?" is also a classifier question. It is the most application-specific of the four, so the instructions to the classifier have to describe your scope, and the threshold has to be tuned on your own traffic.

## Output checks

Output checks run on what the model returned. They catch secrets or PII the model reproduced from its context or training, harmful content a provider's own filter allowed, and text that matches a term you never want to show. Improper output handling, fifth in OWASP's list[1], is the reason structured output also needs validation in code before anything acts on it. That is an application concern rather than a filter, covered in the [defense-in-depth guide](https://infro.io/blog/prompt-injection-defense-in-depth).

Two properties of output checks catch people out. First, the provider has already generated the response and will bill for it; blocking it afterwards changes what the user sees, not what you pay. Second, streaming. A check that has to see a whole match before deciding must hold back the tail of the stream, which delays the first visible token slightly. A block mid-stream can stop what follows, but it cannot recall what was already sent. Azure documents the same shape for its own filter: a filtered streaming completion ends with `finish_reason: "content_filter"` after the segments already returned[5].

## Model and provider policy

The least glamorous guardrail is often the most effective: deciding which models a workload may call at all. An allowlist stops a developer pointing production traffic at an unevaluated model, keeps regulated data on models whose terms you have reviewed, and closes off expensive models where a budget would only notice afterwards. Block entries matter as much as allow entries. "Everything from this vendor except one model" and "every model except this vendor" are common policies, and a list that can only allow forces you to keep updating it as catalogs grow.

Model policy has to hold on every path a request can take: a direct model name, a named route, and any fallback the gateway tries after a failure. A policy that a route or a fallback can step around is not a policy. The broader governance process (who approves, how exceptions work) is in [model access governance](https://infro.io/blog/model-access-governance).

## Where each check belongs

[Diagram: A placement rule of thumb: put a check where the information it needs, and the consistency it needs, both exist.]

**The application** knows the user, the task and which tools are in play. Authorization of tool calls, validation of output against a schema and human approval steps belong there, because nothing outside the application can know whether this user may do this thing.

**A gateway** sees every application's traffic and knows which key, project and environment sent each request. Checks that should be the same everywhere belong there: secret and PII-format detection, literal deny terms, model allow and block lists, spend limits. Putting them in one place means a new service inherits them on day one, and a policy change takes effect everywhere at once rather than in whichever services were redeployed.

**The provider** runs its own safety systems. OpenAI's moderation endpoint returns `flagged`, per-category booleans and per-category scores from 0 to 1 for text and images, and OpenAI recommends treating the scores as signals for your own policy, noting that the model will change and thresholds may need recalibration[6]. Azure runs every prompt and completion through classifiers for hate, sexual, violence and self-harm at configurable severity levels, refuses a filtered prompt with HTTP 400 and code `content_filter`, and marks a filtered completion with `finish_reason: "content_filter"`[5]. These are valuable, but they are per provider. When you route across several providers, or fail over from one to another, the provider layer gives you a different policy on each path. Keep it as a second layer, not the only one.

## Latency, cost and false positives

Each check has a price.

- **Pattern checks** run in memory, with no network call, so their added latency is small next to a model call. Their visible cost is mostly the streaming hold-back on output.
- **Classifier checks** add a network call before the model call, so their latency adds directly to time to first token. They need a timeout, and the timeout needs a decision about what happens when it fires. They also cost money if the classifier is a paid model.
- **Provider filters** are part of the provider's own latency and are not separately visible to you.

False positives are the cost people underestimate. A card detector with no checksum flags order numbers; a phone detector with loose rules flags invoice IDs; an injection classifier at a low threshold flags security researchers and anyone pasting a support ticket that quotes an attack. The remedies are the same for each: prefer detectors with structural tests (issuer prefix plus Luhn, country length plus mod-97), start new rules in record-only mode and read what they would have blocked, and tune classifier thresholds on labelled samples of your own traffic rather than on defaults. NIST's profile warns against extrapolating system performance from narrow, anecdotal assessments (MS-2.5-001), which applies directly to a threshold picked from a handful of test prompts[4].

> **Example: Reading a rule in record-only mode**
> Illustrative. A new `card` rule runs in flag mode for a week on a support assistant. It records 140 matches. A sample review finds real card numbers that customers pasted, and no order numbers, because the detector requires a valid issuer prefix and checksum. The team switches the rule to redact. A `deny_terms` rule for an internal codename records 2,300 matches in the same week, almost all from the system prompt itself, and is rescoped before it ever blocks anything.

## Fail open or fail closed

Every check eventually fails to run: the input is larger than the scanner accepts, the classifier times out, its provider is down. What happens next has to be decided in advance, per check.

[Diagram: Neither answer is free: failing closed turns a checker outage into a feature outage, failing open lets unchecked traffic through.]

Provider filters make this choice for you, and it is worth knowing which way. Azure documents that when its content filtering system is down or cannot finish in time, the request still completes without filtering, and the only signal is an error object inside `content_filter_results`[5]. That is fail-open, and an application that never reads that field never knows. If you fail open, mark the request as unchecked and alert on the rate of unchecked requests. If you fail closed, size the checker's capacity and timeout like any other dependency on the critical path. NIST's profile asks that the system architecture can monitor outputs and handle and recover from errors when security anomalies are detected (MS-2.6-005), which is the same requirement in framework language[4].

## Logging policy for guardrails

A guardrail that redacts a card number and then writes the card number to a log has moved the problem, not solved it. The rule is simple: record **that** a check fired, **which** detector or classifier, at **which** stage, **what** action and **how many** matches — never the matched text. Keep policy changes in an append-only audit log, with the shape of each rule but not secret term lists, so you can answer "who turned this off, and when" during a review. The wider question of whether prompt content is captured at all is covered in the [prompt logging policy guide](https://infro.io/blog/llm-prompt-logging-policy).

## Guardrails in INFRO: what is checked, and what a block returns

INFRO is a BYOK control plane: requests run on the customer's own provider accounts, and the gateway applies policy on the way. Its guardrails are the gateway layer described above, and the docs are explicit that detection is pattern-based and can miss things.

**What is inspected.** `POST /v1/chat/completions`, streaming and non-streaming: text in `messages[].content` for every role, tool-call arguments, and the response's text and tool-call arguments. The `prompt` of image and video requests is checked on input. Tool definitions, image and audio parts, and `/v1/audio/*` are not inspected. An organization setting, **unguarded endpoints**, can refuse requests to endpoints guardrails cannot inspect while any input rule is active.

**Detectors.** `email`, `phone`, `card` (issuer prefix plus Luhn), `iban` (country length plus mod-97), `ssn` (the dashed US form), `ip`, `secret` (prefixed keys from OpenAI, Anthropic, AWS access key ids, GitHub, Slack, Stripe live keys, Google, plus JWTs and PEM private keys) and `deny_terms` (up to 200 literal terms per rule, never regular expressions). Text is normalized with Unicode NFKC and stripped of zero-width characters before matching. Detectors run inside the gateway and call no outside service.

**Rules.** A policy holds up to 24 rules, each a detector, an action (`block`, `redact` with placeholders such as `[EMAIL]`, or `flag`) and a stage (`input`, `output` or `both`). Policies can be set on the organization, a project and a gateway key; every applicable scope is evaluated and the strictest action wins, so a project or key can tighten what the organization set but never loosen it.

[Diagram: The order on a guarded chat request in INFRO, from the guardrails and model allowlist documentation.]

**Errors and records.** A block returns `403` with `error.type: "permission_denied"` and `code: "guardrail_blocked"`; the message names the detector and where it matched (for example `messages[1]`), never the text. When a check cannot finish (over the 2 MB scan limit, or an engine failure), each policy's `on_engine_error` decides: `block`, the default, returns `403 guardrail_unavailable`; `allow` serves the request and records it as flagged. Neither code is retryable. When a rule acts, the response carries an `X-INFRO-Guardrails` header and a `guardrails` field with the action and a count per detector, and request records from `GET /v1/requests` carry the same counts.

[Diagram: What each action does to the request, the response and the record.]

**Classifier checks.** For meaning-level checks, a policy can add up to 10 yes/no questions about the prompt, answered in one call by a decision model (Jev by default, or another decision model) on the customer's own connection, which that provider bills. Presets cover prompt injection, jailbreak, harmful requests, self-harm risk (flag only by default) and off-topic use. Each check has `block_at` and `flag_at` thresholds on the probability of yes, the call has a `timeout_ms` from 300 to 5,000 milliseconds (default 2,000), and `classifiers.on_error` chooses fail-closed (`block`, the default) or fail-open (`allow`, recorded as an `unavailable` finding). Classifier checks run after the pattern rules have redacted the prompt and before the budget check, read prompts only, never responses, and the classifier call is its own row in the request log. The [Jev guardrails guide](https://infro.io/blog/jev-llm-guardrails-model-routing) covers how to write and tune those questions.

**Model allowlists** apply at organization, project and key scope, and a model must satisfy every non-empty list. Entries are catalog ids or vendor wildcards; a `!` prefix blocks: `!openai/gpt-4o` one model, `!openai/*` a vendor. A block entry always wins in its list, and a list of block entries alone allows everything else, including models added later. Route targets and fallbacks stay subject to the lists. A denied model returns `403` with `code: "model_not_allowed"`, naming the list that refused it.

**What is stored.** Matched text is never stored, in request records, the audit log, alerts, webhooks or OpenTelemetry. The audit log records `guardrail.updated`, `guardrail.deleted` and `guardrail.settings_updated` with each rule's shape and the number of terms, not the terms; allowlist changes are recorded as policy or key updates. The console's **Try it** runs sample text through the policies that would apply without storing it.

**Plans.** Model allowlists and the audit log are on every plan. Guardrails, including classifier checks, are included on the Business plan ($599 a month) and Enterprise. The plan gate applies to turning a policy on; turning one off or deleting it works on every plan, and a policy already set keeps being enforced after a move to a smaller plan. The [guardrails reference](https://infro.io/docs/features/guardrails) lists every detector's exact match rules.

## A checklist before you turn a rule on

- [ ] For each check, write down what is inspected, what is detected and what happens.
- [ ] Run new rules in record-only mode first and read what they would have done.
- [ ] Tune classifier thresholds on labelled samples of your own traffic.
- [ ] Decide fail-open or fail-closed per check, and alert on unchecked requests.
- [ ] Put shared rules (secrets, PII formats, model lists) at the gateway; keep authorization in the application.
- [ ] Confirm no log, trace or alert stores the text a check matched.
- [ ] Keep policy changes in an audit log someone reviews.

## Sources

1. OWASP Top 10 for LLM Applications 2025 — OWASP GenAI Security Project. https://genai.owasp.org/llm-top-10/ (accessed 2026-10-13)
2. LLM01:2025 Prompt Injection — OWASP GenAI Security Project. https://genai.owasp.org/llmrisk/llm01-prompt-injection/ (accessed 2026-10-13)
3. LLM02:2025 Sensitive Information Disclosure — OWASP GenAI Security Project. https://genai.owasp.org/llmrisk/llm022025-sensitive-information-disclosure/ (accessed 2026-10-13)
4. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1) — NIST. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf (accessed 2026-10-13)
5. Content filtering for Microsoft Foundry Models — Microsoft Learn. https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/content-filter (accessed 2026-10-13)
6. Moderation — OpenAI. https://developers.openai.com/api/docs/guides/moderation (accessed 2026-10-13)