Builders beware: AI Structured Outputs are not all the same
Look at how your LLM and Agent Framework implement the Grammar pattern
To build, test, and evaluate agentic workflows while minimizing the non-determinism of LLMs, a key design pattern that you will need to use is Grammar.
From Hannes’ and my upcoming book on Generative AI Design Patterns (should be in stores in October):
In many cases, you’d like the text generated by the LLM to follow a specific format — which could be as simple as a comma-separated list or as complex as a syntactically valid structured query language (SQL) statement. This is often because you are going to hand the generated text to a downstream application, which expects to operate on the LLM response without having to do all sorts of parsing and validation.
Some naive approaches would be to state the format you want in the prompt (“provide the output in JSON”) or to provide a few examples of the format you want (“structure the output like this”) and hope that the LLM always generates text that follows the syntax of the examples …
A better approach is to represent the rules you want in a generalizable way, which is called a grammar. Then, the model framework will apply your grammar specification to constrain the set of tokens it generates, so that the generated text will conform exactly to the grammar.
The pattern description goes into a lot of detail about how and why this pattern works. The long-and-short of it is that it involves taking the possible output continuations and zeroing the logits corresponding to any option that doesn’t meet the grammar rules. This is a form of constrained decoding or logits masking.
These are technical terms, and to make it user-friendly, LLM providers may call it JSON-mode or Structured Outputs. Unfortunately, not all frontier models and agentic frameworks use constrained decoding to support their JSON/XML/structured outputs — you have peek under the hood to make sure you are really are getting the logits corresponding to non-conformant continuations zeroed out. Unfortunately, in the case of proprietary models, you can not look at the code, so you have to divine what the approach they are taking is, based on how they describe it in the documentation.
Builders, be careful out there.
What various frontier models and frameworks do
We didn’t include it in the book because we want the book to be ever-green. But a blog post can go out of date. So, here’s my analysis of what the various model providers do. Corrections and explanations welcome!
Anthropic’s Structured Outputs functionality seems to rely solely on instruction following (“Precisely define your desired output format using JSON, XML, or custom templates so that Claude understands every output formatting element you require.”). It’s a bit loosey-goosey and I would be a bit apprehensive about building strongly typed agentic frameworks that rely on the model adhering to the given schema. This is especially the case if you are using the faster/smaller models that are not as good at instruction-following as the frontier model.
On the other hand, Gemini seems to employ constrained decoding. It will faithfully follow any schema definition (“allowing precise extraction and standardization of information for further processing”). The only problem is if your schema is very complex and the none of the likely continuations match the schema — in such cases, the output generation will fail. However, this is a known drawback of the Grammar pattern, and the tradeoff you are making in return for strongly typed agentic workflows.
Open AI gives you a choice — it seems to rely on constrained decoding (“Reliable type-safety: No need to validate or retry incorrectly formatted responses”), but seems to fall back on instruction-following (You could include language in your prompt to specify that) if constrained decoding doesn’t work. This seems more flexible, but I think it makes it harder to build. At least by avoiding such language in your prompt, you can get strict constrained decoding behavior.
In terms of frameworks, Pydantic AI will use the underlying model’s structured outputs functionality (“use the model’s native structured outputs functionality”). Which means you can’t trust Anthropic, can trust Gemini, and can choose strict conformance on OpenAI.
LangChain tries to provide a consistent interface — it uses the model’s underlying functionality if available, or uses a second LLM call to reformat the response into the desired format if the model doesn’t guarantee conformance. Unfortunately, the table of which models support structured outputs doesn’t distinguish between constrained decoding and instruction following. So, in practice, you are at the mercy of how the models implement this functionality. You will almost never hit the reformatting section of the codebase when using one of the frontier models.
Do you know how other models or frameworks implement Structured Outputs? Do they do constrained decoding? Or do they rely on instruction following? Can you trust the model to conform to the schema? Comment below.
