# Structured outputs across providers: JSON schema reliability

URL: https://infro.io/blog/structured-output-reliability
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-09-29
Last modified: 2026-09-29
Last reviewed: 2026-09-29 (next review due 2026-10-29)
Topic: Evaluation and models
Format: Guide

> Schema-constrained decoding removes most syntax errors on providers that support it, but not truncation, refusals, unsupported keywords, silently dropped parameters or wrong values. Write schemas to the subset every provider in your chain enforces, validate every response client-side, classify each failure, and measure parse and schema pass rates per model and provider before you fail over between them.

Key takeaways:
- Constrained decoding makes output match a schema; it does not stop truncation, refusals or wrong values, so every response still needs client-side validation.
- Each provider enforces a different JSON Schema subset. Anthropic, for example, documents no support for minimum, maximum or pattern, so put those rules in your validator.
- Check finish_reason before parsing. Truncated JSON is fixed with a higher token limit, not a retry.
- A fallback to a provider that ignores or drops the schema turns a reliable path into prose. Qualify every model in the chain for the mode you use.
- Classify failures into transport, dropped parameter, truncation, refusal, syntax, schema and semantic, and track each rate per model and provider.

The most expensive structured-output bug is the one that returns `200 OK`. The request succeeded, the provider billed it, and the payload is truncated mid-object, or is prose because the serving provider ignored your schema, or is perfectly shaped JSON with a date in the wrong year. None of those raise an HTTP error. All of them reach the code that trusted the response.

Schema-constrained decoding solved the syntax problem for the providers that offer it. It did not solve reliability, and the gap widens as soon as more than one provider can serve a request.

## What each provider actually enforces

Three providers, three request shapes, three schema subsets.

[Diagram: Structured-output modes as each vendor documents them, 2026-09-29, and what happens to the request through INFRO. "Not stated" means the page did not say.]

**OpenAI.** Structured Outputs constrains the response to a supplied JSON Schema, and OpenAI distinguishes it from JSON mode: both produce valid JSON, but only Structured Outputs ensures schema adherence[1]. It supports much, not all, of JSON Schema; the first request with a new schema has extra latency while the schema is processed; a safety refusal arrives in a separate `refusal` field instead of your object; and a response cut off by the token limit is incomplete rather than invalid, which you must handle[1].

**Anthropic.** Claude constrains output with compiled grammars, through `output_config.format` for JSON responses and `strict: true` for tool inputs; the older `output_format` parameter is deprecated[2]. The documented subset excludes recursive schemas, numeric constraints (`minimum`, `maximum`, `multipleOf`), string constraints (`minLength`, `maxLength`, `pattern`) and external `$ref`; `additionalProperties` must be `false`, and array `minItems` accepts only 0 or 1. Compiled grammars are cached for 24 hours from last use, and changing only `name` or `description` does not invalidate the cache. A response that stops for `max_tokens` can be incomplete JSON[2].

**Google Gemini.** Gemini takes a JSON Schema with the `application/json` MIME type, supports a subset that does include `minimum`, `maximum`, `minItems`, `maxItems` and `prefixItems`, warns that very large or deeply nested schemas may be rejected, and tells you to validate values in your application[3]. Its OpenAI-compatible endpoint also documents structured output[4].

The consequence: a schema that uses `pattern` or `maximum` is enforced on one provider and not on another. Write schemas to the intersection, and enforce everything else in your own validator.

## A taxonomy of failures

Treat "the JSON was bad" as seven different failures, because each has a different fix.

[Diagram: Seven failure classes, from the network to the meaning of a field. Classify every failed response into exactly one.]

| Class | Signal | Fix |
| --- | --- | --- |
| Transport | Exception, 429, 5xx, timeout | Retry policy and failover; not a schema problem |
| Parameter dropped or refused | 400 with the provider's message; or prose where JSON was expected | Change the schema or the provider; do not retry |
| Truncation | `finish_reason` `length` | Raise `max_tokens` or shrink the object |
| Refusal | A refusal field or refusal text | Handle as a product outcome |
| Syntax | `json.loads` fails | One repair attempt; investigate if frequent |
| Schema | Parses; validator rejects | One repair attempt; tighten descriptions |
| Semantic | Validates; a business rule fails | Rule-specific handling; review the prompt |

Table: The failure classes and what to do about each.

Two classes cause most production confusion. **Truncation** looks like a syntax error but is fixed by a token budget, never by retrying the same request. **Dropped parameters** look like a model misbehaving but are a routing problem: the request reached a provider that did not apply your schema.

## Designing schemas that survive a provider change

- Make every property `required`, and express optional fields as a nullable type such as `["string", "null"]`. OpenAI's examples and Anthropic's rules both expect this shape[1][2].
- Set `additionalProperties: false` on every object; Anthropic requires it[2].
- Keep numeric ranges, string lengths and patterns out of the schema you send, or accept that they are enforced only on some providers. Put them in your validator either way.
- Avoid recursive schemas and complex values inside `enum`[2].
- Use `description` on fields. It steers content without adding prompt text, and on Anthropic changing it does not force a grammar recompile[2].
- Keep the object small enough to finish within `max_tokens` with a margin. Long arrays are the usual cause of truncation.

## A schema test harness

Measure before you trust. The harness below runs a set of inputs against each model or route, classifies every response into the taxonomy, and prints rates you can compare.

```python
import json
import os
from collections import Counter

import openai
from openai import OpenAI
from pydantic import BaseModel, ValidationError

client = OpenAI(base_url="https://api.infro.io/v1", api_key=os.environ["INFRO_API_KEY"])


class Invoice(BaseModel):
    invoice_number: str
    vendor: str
    due_date: str | None
    total: float


SCHEMA = {
    "type": "json_schema",
    "json_schema": {
        "name": "invoice_extraction",
        "strict": True,
        "schema": {
            "type": "object",
            "properties": {
                "invoice_number": {"type": "string"},
                "vendor": {"type": "string"},
                "due_date": {"type": ["string", "null"], "description": "ISO 8601 date"},
                "total": {"type": "number"},
            },
            "required": ["invoice_number", "vendor", "due_date", "total"],
            "additionalProperties": False,
        },
    },
}


def business_rules(inv: Invoice, expected: dict) -> bool:
    return inv.invoice_number == expected["invoice_number"] and abs(inv.total - expected["total"]) < 0.01


def classify(model: str, text_in: str, expected: dict) -> str:
    try:
        res = client.chat.completions.create(
            model=model,
            messages=[
                {"role": "system", "content": "Extract the invoice fields. Reply in JSON."},
                {"role": "user", "content": text_in},
            ],
            response_format=SCHEMA,
            max_tokens=400,
        )
    except openai.BadRequestError:
        return "parameter_refused"
    except openai.APIError:
        return "transport"

    choice = res.choices[0]
    if choice.finish_reason == "length":
        return "truncation"
    if getattr(choice.message, "refusal", None):
        return "refusal"
    text = choice.message.content or ""
    try:
        data = json.loads(text)
    except json.JSONDecodeError:
        return "syntax" if text.lstrip().startswith(("{", "[")) else "parameter_dropped"
    try:
        invoice = Invoice.model_validate(data)
    except ValidationError:
        return "schema"
    return "ok" if business_rules(invoice, expected) else "semantic"


CASES = [
    ("Invoice #4821 from Acme Corp, due 2026-03-15, total $1,240.50.",
     {"invoice_number": "4821", "total": 1240.50}),
    ("Acme Corp bill 4822. Amount payable: 99 dollars. No due date given.",
     {"invoice_number": "4822", "total": 99.0}),
]

for model in ["extraction-primary", "extraction-standby"]:  # routes, or model ids
    counts = Counter(classify(model, text, exp) for text, exp in CASES * 25)
    total = sum(counts.values())
    print(model, {k: f"{v / total:.1%}" for k, v in counts.most_common()})
```

Run it with a few hundred real inputs, including the awkward ones: missing fields, two amounts, text in another language. The number that matters is not whether a model can produce the object once, but the rate of each failure class over many inputs, per model and per provider.

## One repair attempt, then stop

For syntax and schema failures, one repair turn costs little and often suffices: send the model its own output and the validation error, and ask for a corrected object. Cap it at one attempt, count it separately, and never repair a truncation or a refusal.

```typescript
import OpenAI from "openai";
import { z } from "zod";

const Invoice = z.object({
  invoice_number: z.string(),
  vendor: z.string(),
  due_date: z.string().nullable(),
  total: z.number().nonnegative(), // enforced here, whatever the provider supports
});

export async function extract(client: OpenAI, model: string, input: string) {
  const messages: OpenAI.Chat.ChatCompletionMessageParam[] = [
    { role: "system", content: "Extract the invoice fields. Reply in JSON." },
    { role: "user", content: input },
  ];
  for (let attempt = 0; attempt < 2; attempt++) {
    const res = await client.chat.completions.create({
      model,
      messages,
      response_format: { type: "json_object" },
      max_tokens: 400,
    });
    const choice = res.choices[0];
    if (choice.finish_reason === "length") throw new Error("truncated: raise max_tokens");
    const text = choice.message.content ?? "";
    let problem: string;
    try {
      const parsed = Invoice.safeParse(JSON.parse(text));
      if (parsed.success) return { invoice: parsed.data, repaired: attempt > 0 };
      problem = parsed.error.message;
    } catch (err) {
      problem = `Invalid JSON: ${(err as Error).message}`;
    }
    messages.push({ role: "assistant", content: text });
    messages.push({ role: "user", content: `That output failed validation: ${problem}. Return the corrected JSON only.` });
  }
  throw new Error("validation failed after one repair");
}
```

[Diagram: The order of checks for every structured response. Each stage maps to a failure class, so the counts fall out of the pipeline.]

## Fallbacks can quietly remove the schema

Failover protects availability, and it can break structured output in a way no error reports. If the primary enforces your schema and the fallback does not, a provider outage turns validated JSON into best-effort JSON or prose, at exactly the moment you are least likely to be watching.

Parameters are part of this too. Anthropic now returns a 400 when `temperature`, `top_p` or `top_k` is set to a non-default value on Claude Opus 4.7 and later models[5], so an extraction request carrying the common `temperature: 0` fails outright when it falls back to one of those models. Test each fallback with the exact request your application sends.

## Structured outputs through INFRO

INFRO accepts `response_format` on [`POST /v1/chat/completions`](https://infro.io/docs/api/chat-completions) in both standard modes, `json_object` and `json_schema`. On OpenAI-protocol connections — OpenAI, Azure OpenAI, the Gemini API's OpenAI-compatible endpoint, Groq, Together, Fireworks and others — it is forwarded unchanged; where a provider uses OpenAI's Responses protocol it is translated to that protocol's equivalent. INFRO never rewrites or repairs what comes back: invalid JSON arrives in a normal 200, and a provider that rejects the schema produces a 400 carrying the provider's own message. See [structured outputs](https://infro.io/docs/features/structured-outputs).

> **Warning: Anthropic connections drop response_format**
> When a request is served by an Anthropic connection, INFRO drops `response_format` today, so the reply is ordinary text. For JSON from Claude models through INFRO, describe the shape in the prompt or use [tool calling](https://infro.io/docs/features/tool-calling) with a single tool whose parameters are your schema, and validate the result. INFRO translates OpenAI-style function tools into Anthropic's format, but does not carry an OpenAI-style `strict` flag across.

That makes chain design explicit. If a [route](https://infro.io/docs/routes) or a `fallbacks` list mixes an OpenAI-protocol primary with an Anthropic standby, the standby will not apply your schema. Either keep structured-output routes on providers that enforce the mode you use, or make the client validator and the one-shot repair above part of every call so a standby's answer is still checked. The response's `model`, `provider` and `connection` fields name what actually served, so your harness can attribute each failure class to the provider that produced it.

INFRO's model catalog has a `json` capability flag for strict schema support, but the documentation notes it is not recorded per model yet — an empty `capabilities` array means not recorded, not unsupported — so the vendor's documentation and your own test run are the sources of truth. Request records keep `finish_reason` per request, and [request traces](https://infro.io/docs/observability/traces) let you filter to truncated responses when a schema change starts producing them.

## Checklist

- [ ] Every property `required`, optional fields nullable, `additionalProperties: false` everywhere.
- [ ] Ranges, lengths and patterns enforced in your validator, not only in the schema.
- [ ] `finish_reason` checked before parsing; truncation handled by the token budget.
- [ ] Refusals handled as a product outcome, not retried.
- [ ] Every response parsed and validated client-side, including on fallbacks.
- [ ] At most one repair attempt, counted separately.
- [ ] Each model and provider in the chain tested with the exact request, including sampling parameters.
- [ ] Failure rates tracked per class, per model, per provider.

## Questions

### Does JSON mode guarantee my schema?

No. JSON mode produces syntactically valid JSON, but nothing enforces which keys appear. Only schema-constrained modes such as OpenAI's strict Structured Outputs enforce the shape, and only on models whose vendor supports them.

## Sources

1. Structured Outputs — OpenAI. https://developers.openai.com/api/docs/guides/structured-outputs (accessed 2026-09-29)
2. Structured outputs — Anthropic. https://platform.claude.com/docs/en/build-with-claude/structured-outputs (accessed 2026-09-29)
3. Structured output — Google. https://ai.google.dev/gemini-api/docs/structured-output (accessed 2026-09-29)
4. OpenAI compatibility — Google. https://ai.google.dev/gemini-api/docs/openai (accessed 2026-09-29)
5. Model deprecations — API parameter deprecations — Anthropic. https://platform.claude.com/docs/en/about-claude/model-deprecations (accessed 2026-09-29)