Structured outputs across providers: JSON schema reliability

The most expensive structured-output bug is the one that returns 200 OK. The request succeeded, the provider billed it, and the payload is truncated mid-object, or is prose because the serving provider ignored your schema, or is perfectly shaped JSON with a date in the wrong year. None of those raise an HTTP error. All of them reach the code that trusted the response.

Schema-constrained decoding solved the syntax problem for the providers that offer it. It did not solve reliability, and the gap widens as soon as more than one provider can serve a request.

What each provider actually enforces

Three providers, three request shapes, three schema subsets.

OpenAI. Structured Outputs constrains the response to a supplied JSON Schema, and OpenAI distinguishes it from JSON mode: both produce valid JSON, but only Structured Outputs ensures schema adherence¹. It supports much, not all, of JSON Schema; the first request with a new schema has extra latency while the schema is processed; a safety refusal arrives in a separate refusal field instead of your object; and a response cut off by the token limit is incomplete rather than invalid, which you must handle¹.

Anthropic. Claude constrains output with compiled grammars, through output_config.format for JSON responses and strict: true for tool inputs; the older output_format parameter is deprecated². The documented subset excludes recursive schemas, numeric constraints (minimum, maximum, multipleOf), string constraints (minLength, maxLength, pattern) and external $ref; additionalProperties must be false, and array minItems accepts only 0 or 1. Compiled grammars are cached for 24 hours from last use, and changing only name or description does not invalidate the cache. A response that stops for max_tokens can be incomplete JSON².

Google Gemini. Gemini takes a JSON Schema with the application/json MIME type, supports a subset that does include minimum, maximum, minItems, maxItems and prefixItems, warns that very large or deeply nested schemas may be rejected, and tells you to validate values in your application³. Its OpenAI-compatible endpoint also documents structured output⁴.

The consequence: a schema that uses pattern or maximum is enforced on one provider and not on another. Write schemas to the intersection, and enforce everything else in your own validator.

A taxonomy of failures

Treat "the JSON was bad" as seven different failures, because each has a different fix.

Two classes cause most production confusion. Truncation looks like a syntax error but is fixed by a token budget, never by retrying the same request. Dropped parameters look like a model misbehaving but are a routing problem: the request reached a provider that did not apply your schema.

Designing schemas that survive a provider change

• Make every property required, and express optional fields as a nullable type such as ["string", "null"]. OpenAI's examples and Anthropic's rules both expect this shape¹².
• Set additionalProperties: false on every object; Anthropic requires it².
• Keep numeric ranges, string lengths and patterns out of the schema you send, or accept that they are enforced only on some providers. Put them in your validator either way.
• Avoid recursive schemas and complex values inside enum².
• Use description on fields. It steers content without adding prompt text, and on Anthropic changing it does not force a grammar recompile².
• Keep the object small enough to finish within max_tokens with a margin. Long arrays are the usual cause of truncation.

A schema test harness

Measure before you trust. The harness below runs a set of inputs against each model or route, classifies every response into the taxonomy, and prints rates you can compare.

import json
import os
from collections import Counter

import openai
from openai import OpenAI
from pydantic import BaseModel, ValidationError

client = OpenAI(base_url="https://api.infro.io/v1", api_key=os.environ["INFRO_API_KEY"])


class Invoice(BaseModel):
    invoice_number: str
    vendor: str
    due_date: str | None
    total: float


SCHEMA = {
    "type": "json_schema",
    "json_schema": {
        "name": "invoice_extraction",
        "strict": True,
        "schema": {
            "type": "object",
            "properties": {
                "invoice_number": {"type": "string"},
                "vendor": {"type": "string"},
                "due_date": {"type": ["string", "null"], "description": "ISO 8601 date"},
                "total": {"type": "number"},
            },
            "required": ["invoice_number", "vendor", "due_date", "total"],
            "additionalProperties": False,
        },
    },
}


def business_rules(inv: Invoice, expected: dict) -> bool:
    return inv.invoice_number == expected["invoice_number"] and abs(inv.total - expected["total"]) < 0.01


def classify(model: str, text_in: str, expected: dict) -> str:
    try:
        res = client.chat.completions.create(
            model=model,
            messages=[
                {"role": "system", "content": "Extract the invoice fields. Reply in JSON."},
                {"role": "user", "content": text_in},
            ],
            response_format=SCHEMA,
            max_tokens=400,
        )
    except openai.BadRequestError:
        return "parameter_refused"
    except openai.APIError:
        return "transport"

    choice = res.choices[0]
    if choice.finish_reason == "length":
        return "truncation"
    if getattr(choice.message, "refusal", None):
        return "refusal"
    text = choice.message.content or ""
    try:
        data = json.loads(text)
    except json.JSONDecodeError:
        return "syntax" if text.lstrip().startswith(("{", "[")) else "parameter_dropped"
    try:
        invoice = Invoice.model_validate(data)
    except ValidationError:
        return "schema"
    return "ok" if business_rules(invoice, expected) else "semantic"


CASES = [
    ("Invoice #4821 from Acme Corp, due 2026-03-15, total $1,240.50.",
     {"invoice_number": "4821", "total": 1240.50}),
    ("Acme Corp bill 4822. Amount payable: 99 dollars. No due date given.",
     {"invoice_number": "4822", "total": 99.0}),
]

for model in ["extraction-primary", "extraction-standby"]:  # routes, or model ids
    counts = Counter(classify(model, text, exp) for text, exp in CASES * 25)
    total = sum(counts.values())
    print(model, {k: f"{v / total:.1%}" for k, v in counts.most_common()})

Run it with a few hundred real inputs, including the awkward ones: missing fields, two amounts, text in another language. The number that matters is not whether a model can produce the object once, but the rate of each failure class over many inputs, per model and per provider.

One repair attempt, then stop

For syntax and schema failures, one repair turn costs little and often suffices: send the model its own output and the validation error, and ask for a corrected object. Cap it at one attempt, count it separately, and never repair a truncation or a refusal.

import OpenAI from "openai";
import { z } from "zod";

const Invoice = z.object({
  invoice_number: z.string(),
  vendor: z.string(),
  due_date: z.string().nullable(),
  total: z.number().nonnegative(), // enforced here, whatever the provider supports
});

export async function extract(client: OpenAI, model: string, input: string) {
  const messages: OpenAI.Chat.ChatCompletionMessageParam[] = [
    { role: "system", content: "Extract the invoice fields. Reply in JSON." },
    { role: "user", content: input },
  ];
  for (let attempt = 0; attempt < 2; attempt++) {
    const res = await client.chat.completions.create({
      model,
      messages,
      response_format: { type: "json_object" },
      max_tokens: 400,
    });
    const choice = res.choices[0];
    if (choice.finish_reason === "length") throw new Error("truncated: raise max_tokens");
    const text = choice.message.content ?? "";
    let problem: string;
    try {
      const parsed = Invoice.safeParse(JSON.parse(text));
      if (parsed.success) return { invoice: parsed.data, repaired: attempt > 0 };
      problem = parsed.error.message;
    } catch (err) {
      problem = `Invalid JSON: ${(err as Error).message}`;
    }
    messages.push({ role: "assistant", content: text });
    messages.push({ role: "user", content: `That output failed validation: ${problem}. Return the corrected JSON only.` });
  }
  throw new Error("validation failed after one repair");
}

Fallbacks can quietly remove the schema

Failover protects availability, and it can break structured output in a way no error reports. If the primary enforces your schema and the fallback does not, a provider outage turns validated JSON into best-effort JSON or prose, at exactly the moment you are least likely to be watching.

Parameters are part of this too. Anthropic now returns a 400 when temperature, top_p or top_k is set to a non-default value on Claude Opus 4.7 and later models⁵, so an extraction request carrying the common temperature: 0 fails outright when it falls back to one of those models. Test each fallback with the exact request your application sends.

Structured outputs through INFRO

INFRO accepts response_format on POST /v1/chat/completions in both standard modes, json_object and json_schema. On OpenAI-protocol connections — OpenAI, Azure OpenAI, the Gemini API's OpenAI-compatible endpoint, Groq, Together, Fireworks and others — it is forwarded unchanged; where a provider uses OpenAI's Responses protocol it is translated to that protocol's equivalent. INFRO never rewrites or repairs what comes back: invalid JSON arrives in a normal 200, and a provider that rejects the schema produces a 400 carrying the provider's own message. See structured outputs.

That makes chain design explicit. If a route or a fallbacks list mixes an OpenAI-protocol primary with an Anthropic standby, the standby will not apply your schema. Either keep structured-output routes on providers that enforce the mode you use, or make the client validator and the one-shot repair above part of every call so a standby's answer is still checked. The response's model, provider and connection fields name what actually served, so your harness can attribute each failure class to the provider that produced it.

INFRO's model catalog has a json capability flag for strict schema support, but the documentation notes it is not recorded per model yet — an empty capabilities array means not recorded, not unsupported — so the vendor's documentation and your own test run are the sources of truth. Request records keep finish_reason per request, and request traces let you filter to truncated responses when a schema change starts producing them.

Checklist

• Every property required, optional fields nullable, additionalProperties: false everywhere.
• Ranges, lengths and patterns enforced in your validator, not only in the schema.
• finish_reason checked before parsing; truncation handled by the token budget.
• Refusals handled as a product outcome, not retried.
• Every response parsed and validated client-side, including on fallbacks.
• At most one repair attempt, counted separately.
• Each model and provider in the chain tested with the exact request, including sampling parameters.
• Failure rates tracked per class, per model, per provider.