What are structured outputs?

Structure changes the interface

Free-form text is useful when the output is for a person. It becomes fragile when code must recover fields by searching for headings, removing Markdown fences, or guessing whether “yes” means true.

A structured-output mode supplies a schema through the model API. The provider constrains generation so a successful response follows the supported shape. The application can then parse a typed object instead of extracting data from prose.

flowchart LR
  I[Task + input] --> M[Model generation]
  S[Supported schema] --> M
  M --> R{Response state}
  R -->|complete| J[Schema-valid data]
  R -->|refusal| F[Explicit refusal]
  R -->|truncated or failed| E[Incomplete result]
  J --> V[Domain validation]

The response envelope still matters. A refusal, token limit, content filter, interrupted stream, or provider error is not an empty instance of the requested object.

A decoder can narrow the next continuation

One way to constrain generation is to track a formal language as the output grows. At each position, reject continuations that cannot preserve a permitted prefix, then select among the remaining choices. This changes the available output space; it does not establish that a permitted value is true.

PICARD demonstrates incremental checks for SQL generation. Its parsing and schema guards go beyond simple punctuation, but it is a research example of constrained decoding, not a claim about how every provider implements JSON output.

The next experiment uses a much smaller finite grammar: one object with a decision value of approve, deny, or unknown. The vocabulary contains whole teaching fragments rather than real subword tokens. Raw weights are authored; masking and renormalization are calculated. Append fragments and inspect which choices remain possible.

With constraints on, an invalid continuation has zero probability and cannot alter the prefix. Turn them off and choose a closing square bracket at the beginning: the toy records an invalid continuation and stops. Resetting or changing the constraint starts a fresh run.

Completing an approve object exposes a different boundary. The independent fixture has no approval evidence, so well-formed text still authorizes no action. An unknown enum value can preserve that missing information instead of forcing a fabricated answer. The end marker stops decoding and is not part of the JSON text.

Valid JSON can still describe an impossible invoice

Take the fictional object {"subtotal":100,"tax":20,"total":110}. Every value has the right numeric type; the object can satisfy a schema requiring these three fields. It violates the domain rule subtotal + tax = total.

Compare the output channels below before deciding what the caller should do with a parsed object.

For extraction, reject or quarantine the inconsistent invoice. For a proposed payment, also apply current authorization and exact approval before an effect. A complete provider response and a valid shape are useful gates; they cannot replace arithmetic, source evidence, or permission.

Structured response or function call?

Use a structured response when the model's answer is data for the caller: extracted invoice fields, a classification with reasons, a lesson outline, or a set of UI properties.

Use function calling when the model should choose or prepare a capability owned by the application: search_documents, create_issue, or run_test. The generated arguments describe a proposed call. The harness validates and decides whether to execute it, then returns an observation to the model if the loop continues.

The wire formats may look similar, but the ownership differs:

• structured output: parse and use a generated result;
• function call: evaluate a proposed operation before any effect;
• ordinary text: present language to a person or another language-processing step.

Do not invent a fake function solely to obtain JSON when the API provides a response-schema feature. Do not treat a response object as an action simply because it contains a field named command.

JSON Schema is a vocabulary, not one universal implementation

JSON Schema defines a broad language for describing JSON documents. Model providers normally support a documented subset and may impose additional rules on required fields, optional values, recursion, property counts, nesting, or initial schema compilation.

Treat the provider, model, API version, schema, and SDK as one compatibility unit. Validate schemas at startup or deployment instead of discovering unsupported keywords during user traffic. Keep generated clients and server validators aligned with the same canonical schema.

Strict schemas should still allow legitimate uncertainty. If a field may be unknown, model that state explicitly with null, a tagged union, or a status field. Forcing the model to invent a string because every field is required produces valid fiction.

Parseable is not correct

Schema constraints can establish properties such as:

• the output is an object;
• a required field exists;
• an enum contains one of the declared strings;
• a value has a numeric JSON type;
• no undeclared properties appear.

They do not establish that an email belongs to the current user, a date exists, two totals reconcile, a cited document supports a claim, or a refund is authorized. Those are domain and policy checks.

Perform deterministic validation after parsing. Resolve identifiers against authoritative data. Check cross-field invariants, freshness, access, quantity limits, and evidence. Preserve the raw response envelope and validation errors in the trace without logging secrets.

Failure handling belongs to the contract

Define what the caller does when generation refuses, stops early, returns a transport error, passes the schema but fails domain validation, or produces an unsupported case. Retrying the identical request is not automatically useful, and an automatic “repair” model call can change meaning while hiding the original defect.

For extraction, an explicit unknown result may be safer than a guessed value. For batch work, quarantine invalid records with their versioned input and response metadata. For an agent action, reject invalid arguments without executing and return a bounded error observation so the model can choose another path.

The useful guarantee is narrow and powerful: structured generation moves syntax from prompt folklore into an API contract. Keep truth, authorization, and product semantics in ordinary software.