Structured Outputs: JSON Mode, Function Calling, and Tool Use

LLMs return text; applications need data. Every production LLM feature — extraction, classification, tool invocation, form filling — depends on turning free text into validated structures. This guide covers the three mechanisms (JSON mode, function calling, constrained decoding), when to use each, and the validation layer that makes them safe.

The Three Mechanisms

Mechanism	What It Guarantees	Best For
JSON mode	Valid JSON syntax	Simple structured responses
Function calling	Arguments match a schema	Tool invocation, agents
Constrained decoding	Output matches a grammar	Maximum reliability, self-hosted

None of them guarantee semantic correctness. A model can return valid JSON with wrong values. Validation is always your job.

JSON Mode

Most API providers offer a JSON mode that forces syntactically valid JSON output:

from openai import OpenAI
import json

client = OpenAI()

def extract_entities(text: str) -> dict:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": (
                "Extract entities as JSON with keys: name, organization, role, email."
            )},
            {"role": "user", "content": text},
        ],
    )
    return json.loads(response.choices[0].message.content)

JSON mode guarantees parseable JSON but not the shape you asked for. The model can omit keys, add keys, or use wrong types. Always validate against a schema.

Function Calling

Function calling is the strongest mechanism on hosted APIs: you declare a schema, and the model returns arguments that conform to it. It is the backbone of agents and tool use.

tools = [{
    "type": "function",
    "function": {
        "name": "create_issue",
        "description": "Create a GitHub issue",
        "parameters": {
            "type": "object",
            "properties": {
                "title": {"type": "string"},
                "body": {"type": "string"},
                "labels": {"type": "array", "items": {"type": "string"}},
                "priority": {"type": "string", "enum": ["low", "medium", "high"]},
            },
            "required": ["title", "priority"],
        },
    },
}]

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "File a high priority bug about login failing"}],
    tools=tools,
    tool_choice="auto",
)

tool_call = response.choices[0].message.tool_calls[0]
arguments = json.loads(tool_call.function.arguments)
# {'title': 'Login failing', 'priority': 'high', 'labels': ['bug']}

Key details:

• tool_choice="auto" lets the model decide; "required" forces a call; a specific function name forces that function.
• Validate arguments before executing: the schema constrains the model, but never trust it. Re-validate with your own schema.
• Return tool results to the model: after executing, append the result message so the model can continue.

Constrained Decoding (Self-Hosted)

When you control inference, constrained decoding forces the output to match a grammar token by token. The model literally cannot produce invalid output. Libraries like outlines and guidance implement this:

from outlines import models, generate

model = models.transformers("meta-llama/Llama-3.1-8B-Instruct")

class RefundDecision(BaseModel):
    approved: bool
    amount: float
    reason: str

generator = generate.json(model, RefundDecision)

result = generator("Decide the refund for order 4821: $120, policy allows 30 days.")
# RefundDecision(approved=True, amount=120.0, reason='Within 30-day policy')

Constrained decoding gives you schema conformance by construction, at the cost of running your own inference. It is the right choice for high-volume extraction where invalid output is expensive.

The Validation Layer

Whatever mechanism you use, validate before use. Pydantic is the standard:

from pydantic import BaseModel, Field, ValidationError

class RefundDecision(BaseModel):
    approved: bool
    amount: float = Field(ge=0, le=10000)
    reason: str = Field(min_length=1, max_length=500)

def parse_decision(raw: str) -> RefundDecision | None:
    try:
        return RefundDecision.model_validate_json(raw)
    except ValidationError as e:
        print(f"Validation failed: {e}")
        return None

Validation catches what JSON mode and function calling miss: wrong types, out-of-range values, missing required fields.

Retry with Feedback

When validation fails, feed the error back to the model and retry. This converts most failures into successes:

def generate_validated(prompt: str, schema, max_retries: int = 2):
    for attempt in range(max_retries):
        raw = llm_generate(prompt)
        try:
            return schema.model_validate_json(raw)
        except ValidationError as e:
            prompt = (
                f"{prompt}\n\nYour previous response failed validation: {e}\n"
                f"Return valid JSON matching this schema: {schema.model_json_schema()}"
            )
    raise ValueError("Failed to produce valid output after retries")

Streaming Structured Outputs

For long outputs, stream and accumulate, then validate at the end:

def stream_json(prompt: str) -> dict:
    stream = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"},
        stream=True,
    )
    buffer = ""
    for chunk in stream:
        if chunk.choices[0].delta.content:
            buffer += chunk.choices[0].delta.content
    return json.loads(buffer)

Common Failure Modes

• Truncated JSON: output hits the token limit mid-object. Fix: raise max_tokens, or retry with the partial output as context.
• Wrong schema: the model ignores your requested keys. Fix: function calling instead of JSON mode, or stricter system prompt.
• Hallucinated values: valid JSON, wrong data. Fix: grounding, evals, and human review for high-stakes fields.
• Injection via schema: if the schema itself contains untrusted strings, the model may echo them. Treat schema descriptions as data.

Implementation Checklist

• Use JSON mode for simple responses, function calling for tools
• Use constrained decoding for high-volume self-hosted extraction
• Validate every output against a schema before use
• Retry with validation errors fed back to the model
• Raise max_tokens to avoid truncated JSON
• Never execute tool arguments without re-validation
• Add evals for schema conformance and value correctness

---

MatterAI builds frontier AI infrastructure for engineering teams — from inference-optimized models to autonomous coding agents and agentic code reviews.

Explore what we're building:

• Orbital IDE — Autonomous AI coding agent with background agents and deep codebase memory
• AI Code Reviews — Agentic pre-commit reviews across GitHub, GitLab, and Bitbucket
• Axon Models — Frontier-grade reasoning models at 70% lower inference cost

Get started free - https://app.matterai.so

• Read the docs
• View pricing

---

Follow us on X · LinkedIn · GitHub