Erratic performance when using nested schemas

[aus70]

Initial Checks

• I confirm that I'm using the latest version of Pydantic AI
• I confirm that I searched for my issue in https://github.com/pydantic/pydantic-ai/issues before opening this issue

Description

Function call fails on some LLM when using nested classes as output_type

It fails with:

• Qwen/Qwen2.5-72B-Instruct-Turbo on together.ai
• Qwen/Qwen3-4B-fast on nebius.com
• Qwen/Qwen3-235B-A22B-fp8-tput on together.ai

It succeeds with:

• meta-llama/Llama-3.3-70B-Instruct-Turbo on together.ai
• Qwen3-4B-Q6_K.gguf local on llama.cpp

To replicate the bug, see the example code:
When using the classes Count, Size, Name as output_type, all models succeed, when using NameAndCount some models fails. The error messages are the following:

for togethe.ai models pydantic_ai.exceptions.ModelHTTPError: status_code: 400, model_name: Qwen/Qwen2.5-72B-Instruct-Turbo, body: {'message': 'invalid tools grammar: Aborted(). Build with -sASSERTIONS for more info.', 'type': 'invalid_request_error', 'param': 'tools', 'code': None}

for the nebius.com model: pydantic_ai.exceptions.ModelHTTPError: status_code: 400, model_name: Qwen/Qwen3-4B-fast, body: {'detail': "Invalid request. Please check the parameters and try again. Details: 1 validation error for list[function-wrap[__log_extra_fields__()]]\n Invalid JSON: EOF while parsing a value at line 1 column 0 [type=json_invalid, input_value='', input_type=str]

Possibly related to #1561 and #1414

Example Code

import os

from pydantic import BaseModel, Field
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.openai import OpenAIProvider
from pydantic_ai.settings import ModelSettings


class Count(BaseModel):
    count: int = Field(
        description="count",
    )


class Size(BaseModel):
    size: float = Field(
        description="size",
    )


class Name(BaseModel):
    name: str = Field(
        description="name",
    )


class NameAndCount(BaseModel):
    name: str = Field(
        description="name",
    )
    count: Count

# fails:
# pydantic_ai.exceptions.ModelHTTPError: status_code: 400, model_name: Qwen/Qwen3-235B-A22B-fp8-tput, body: {'message': 'invalid tools grammar: Aborted(). Build with -sASSERTIONS for more info.', 'type': 'invalid_request_error', 'param': 'tools', 'code': None}
provider = OpenAIModel(
    model_name="Qwen/Qwen3-235B-A22B-fp8-tput",
    provider=OpenAIProvider(
        base_url="https://api.together.xyz/v1",
        api_key=os.getenv("TOGETHER_API_KEY"),
    ))

# succeeds
# provider = OpenAIModel(
#     model_name="meta-llama/Llama-3.3-70B-Instruct-Turbo",
#     provider=OpenAIProvider(
#         base_url="https://api.together.xyz/v1",
#         api_key=os.getenv("TOGETHER_API_KEY"),
#     ))

# fails:
# pydantic_ai.exceptions.ModelHTTPError: status_code: 400, model_name: Qwen/Qwen2.5-72B-Instruct-Turbo, body: {'message': 'invalid tools grammar: Aborted(). Build with -sASSERTIONS for more info.', 'type': 'invalid_request_error', 'param': 'tools', 'code': None}
# provider = OpenAIModel(
#     model_name="Qwen/Qwen2.5-72B-Instruct-Turbo",
#     provider=OpenAIProvider(
#         base_url="https://api.together.xyz/v1",
#         api_key=os.getenv("TOGETHER_API_KEY"),
#     ))

# fails:
# pydantic_ai.exceptions.ModelHTTPError: status_code: 400, model_name: Qwen/Qwen3-4B-fast, body: {'detail': "Invalid request. Please check the parameters and try again. Details: 1 validation error for list[function-wrap[__log_extra_fields__()]]\n  Invalid JSON: EOF while parsing a value at line 1 column 0 [type=json_invalid, input_value='', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid"}
# provider = OpenAIModel(
#         model_name="Qwen/Qwen3-4B-fast",
#         provider=OpenAIProvider(
#             base_url="https://api.studio.nebius.com/v1",
#             api_key=os.getenv("NEBIUS_API_KEY"),
#         ),
#     )

# succeeds (Qwen3-4B-Q6_K.gguf on llama.cpp)
# provider = OpenAIModel(
#     model_name="Qwen3-4B-Q6_K.gguf",
#     provider=OpenAIProvider(
#         base_url="http://localhost:8080/v1",
#         api_key="llamacpp",
#     ),
# )

model_settings = ModelSettings(
    max_tokens=2048,
    temperature=0.0,
    timeout=60,
)

agent = Agent(
    provider,
    output_type=Size, # using Name, Count, Size works fine, but using NameAndCount fails
    model_settings=model_settings,
    retries=2,
    instrument=True,
    system_prompt="""
    #Instructions:
    - Extract the count, the name and the size from the user input.""",
)


async def main():
    import logfire
    from loguru import logger

    logfire.configure(
        environment="test",
        service_name="bug.py",
        console=logfire.ConsoleOptions(verbose=True),
        send_to_logfire=True,
    )
    logger.configure(handlers=[logfire.loguru_handler()])
    logfire.instrument_httpx(capture_all=True)

    user_input = """
      <UserInput>
      The count is 9, the name is 'test', the size is 55
      </UserInput>
"""
    result = await agent.run(user_input)
    print(result.output)


if __name__ == "__main__":
    import asyncio

    asyncio.run(main())

Python, Pydantic AI & LLM client version

Python 3.12.8 (main, Jan 14 2025, 23:36:58) [Clang 19.1.6 ] on darwin
Pydantic-ai 0.1.10
Qwen/Qwen3-235B-A22B-fp8-tput
meta-llama/Llama-3.3-70B-Instruct-Turbo
Qwen/Qwen2.5-72B-Instruct-Turbo
Qwen/Qwen3-4B-fast
Qwen3-4B-Q6_K.gguf

[tranhoangnguyen03]

To me this looks like LLM-specific limitations or the due to the way the api provider sets up their prompt formatting. This is not stemming from pydantic-ai.

[DouweM]

@aus70 Do you know if the issue is in how PydanticAI formats the JSON schema for the NameAndCount model, or if these models fundamentally don't support less-than-trivial JSON schemas? In the latter case, there's nothing we can do, but if some tweaks to the JSON schema would make the models accept it, we could do that, similar to how we do for OpenAI:

pydantic-ai/pydantic_ai_slim/pydantic_ai/models/openai.py

Line 1071 in dd22595

schema_transformer = _OpenAIJsonSchema(t.parameters_json_schema, strict=t.strict)

[aus70]

@DouweM The issue is correlated to the non-trivial JSON schema, but it's not caused directly by it, more precisely by the presence in the schema of a nested schema ($ref), because by hard-coding in a test call a JSON schema with references (see below an example related to a different set of classes) all models work fine. Obviously, I haven't been able to pinpoint the field that causes the problem.

functions = [
        {
            "name": "final_result",
            "description": "The final response which ends this conversation",
            "parameters": {
                "properties": {
                    "items": {
                        "description": "Items in the user input that are available for purchase and have been ordered by the user",
                        "items": {"$ref": "#/$defs/OrderItem"},
                        "type": "array",
                    }
                },
                "required": ["items"],
                "type": "object",
                "$defs": {
                    "OrderItem": {
                        "properties": {
                            "name": {
                                "description": "The name of the item",
                                "type": "string",
                            },
                            "sale_unit": {
                                "description": "The unit of measurement for the item (e.g., 'kg', 'piece', 'liter')",
                                "type": "string",
                            },
                            "quantity": {
                                "description": "The quantity of item ordered",
                                "type": "number",
                            },
                            "available": {
                                "description": "Is the item available in the catalog",
                                "type": "boolean",
                            },
                        },
                        "required": ["name", "sale_unit", "quantity", "available"],
                        "type": "object",
                    }
                },
            },
        }
    ]

[DouweM]

@aus70 I ran your failing script locally, and the tools definition that's tripping up Qwen looks like this:

[
  {
    "type": "function",
    "function": {
      "name": "final_result",
      "description": "The final response which ends this conversation",
      "parameters": {
        "properties": {
          "name": {
            "description": "name",
            "type": "string"
          },
          "count": {
            "$ref": "#/$defs/Count"
          }
        },
        "required": [
          "name",
          "count"
        ],
        "type": "object",
        "$defs": {
          "Count": {
            "properties": {
              "count": {
                "description": "count",
                "type": "integer"
              }
            },
            "required": [
              "count"
            ],
            "type": "object"
          }
        }
      }
    }
  }
]

It would likely be fine if Count was inlined instead of using $ref/$deps. WalkJsonSchema has a mode for that that is used by GeminiModel, but not OpenAIModel.

Can you subclass OpenAIModel like this, and give the new QwenModel a try?

from pydantic_ai.models import ModelRequestParameters
from pydantic_ai.models._json_schema import JsonSchema
from pydantic_ai.models.openai import _OpenAIJsonSchema, OpenAIModel
from pydantic_ai.settings import ModelSettings
from pydantic_ai.tools import ToolDefinition

class QwenModel(OpenAIModel):
    def customize_request_parameters(self, model_request_parameters: ModelRequestParameters) -> ModelRequestParameters:
        return _customize_request_parameters(model_request_parameters)

def _customize_request_parameters(model_request_parameters: ModelRequestParameters) -> ModelRequestParameters:
    """Customize the request parameters for OpenAI models."""

    def _customize_tool_def(t: ToolDefinition):
        schema_transformer = _QwenJsonSchema(t.parameters_json_schema, strict=t.strict)
        parameters_json_schema = schema_transformer.walk()
        if t.strict is None:
            t = replace(t, strict=schema_transformer.is_strict_compatible)
        return replace(t, parameters_json_schema=parameters_json_schema)

    return ModelRequestParameters(
        function_tools=[_customize_tool_def(tool) for tool in model_request_parameters.function_tools],
        allow_text_output=model_request_parameters.allow_text_output,
        output_tools=[_customize_tool_def(tool) for tool in model_request_parameters.output_tools],
    )

class _QwenJsonSchema(_OpenAIJsonSchema):
    def __init__(self, schema: JsonSchema, strict: bool | None):
        super().__init__(schema, strict=strict)
        self.prefer_inlined_defs = True

This is all copied from models/openai.py, with the notable exception of self.prefer_inlined_defs = True.

[aus70]

QwenModel solves all problems with Qwen models and - unsurprisingly - works also with meta-llama/Llama-3.3-70B-Instruct-Turbo. PR anybody?