Skip to content

Erratic performance when using nested schemas #1659

Closed as duplicate of#1782
Closed as duplicate of#1782
@aus70

Description

@aus70

Initial Checks

Description

Function call fails on some LLM when using nested classes as output_type

It fails with:

  • Qwen/Qwen2.5-72B-Instruct-Turbo on together.ai
  • Qwen/Qwen3-4B-fast on nebius.com
  • Qwen/Qwen3-235B-A22B-fp8-tput on together.ai

It succeeds with:

  • meta-llama/Llama-3.3-70B-Instruct-Turbo on together.ai
  • Qwen3-4B-Q6_K.gguf local on llama.cpp

To replicate the bug, see the example code:
When using the classes Count, Size, Name as output_type, all models succeed, when using NameAndCount some models fails. The error messages are the following:

for togethe.ai models pydantic_ai.exceptions.ModelHTTPError: status_code: 400, model_name: Qwen/Qwen2.5-72B-Instruct-Turbo, body: {'message': 'invalid tools grammar: Aborted(). Build with -sASSERTIONS for more info.', 'type': 'invalid_request_error', 'param': 'tools', 'code': None}

for the nebius.com model: pydantic_ai.exceptions.ModelHTTPError: status_code: 400, model_name: Qwen/Qwen3-4B-fast, body: {'detail': "Invalid request. Please check the parameters and try again. Details: 1 validation error for list[function-wrap[__log_extra_fields__()]]\n Invalid JSON: EOF while parsing a value at line 1 column 0 [type=json_invalid, input_value='', input_type=str]

Possibly related to #1561 and #1414

Example Code

import os

from pydantic import BaseModel, Field
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.openai import OpenAIProvider
from pydantic_ai.settings import ModelSettings


class Count(BaseModel):
    count: int = Field(
        description="count",
    )


class Size(BaseModel):
    size: float = Field(
        description="size",
    )


class Name(BaseModel):
    name: str = Field(
        description="name",
    )


class NameAndCount(BaseModel):
    name: str = Field(
        description="name",
    )
    count: Count

# fails:
# pydantic_ai.exceptions.ModelHTTPError: status_code: 400, model_name: Qwen/Qwen3-235B-A22B-fp8-tput, body: {'message': 'invalid tools grammar: Aborted(). Build with -sASSERTIONS for more info.', 'type': 'invalid_request_error', 'param': 'tools', 'code': None}
provider = OpenAIModel(
    model_name="Qwen/Qwen3-235B-A22B-fp8-tput",
    provider=OpenAIProvider(
        base_url="https://api.together.xyz/v1",
        api_key=os.getenv("TOGETHER_API_KEY"),
    ))

# succeeds
# provider = OpenAIModel(
#     model_name="meta-llama/Llama-3.3-70B-Instruct-Turbo",
#     provider=OpenAIProvider(
#         base_url="https://api.together.xyz/v1",
#         api_key=os.getenv("TOGETHER_API_KEY"),
#     ))

# fails:
# pydantic_ai.exceptions.ModelHTTPError: status_code: 400, model_name: Qwen/Qwen2.5-72B-Instruct-Turbo, body: {'message': 'invalid tools grammar: Aborted(). Build with -sASSERTIONS for more info.', 'type': 'invalid_request_error', 'param': 'tools', 'code': None}
# provider = OpenAIModel(
#     model_name="Qwen/Qwen2.5-72B-Instruct-Turbo",
#     provider=OpenAIProvider(
#         base_url="https://api.together.xyz/v1",
#         api_key=os.getenv("TOGETHER_API_KEY"),
#     ))

# fails:
# pydantic_ai.exceptions.ModelHTTPError: status_code: 400, model_name: Qwen/Qwen3-4B-fast, body: {'detail': "Invalid request. Please check the parameters and try again. Details: 1 validation error for list[function-wrap[__log_extra_fields__()]]\n  Invalid JSON: EOF while parsing a value at line 1 column 0 [type=json_invalid, input_value='', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid"}
# provider = OpenAIModel(
#         model_name="Qwen/Qwen3-4B-fast",
#         provider=OpenAIProvider(
#             base_url="https://api.studio.nebius.com/v1",
#             api_key=os.getenv("NEBIUS_API_KEY"),
#         ),
#     )

# succeeds (Qwen3-4B-Q6_K.gguf on llama.cpp)
# provider = OpenAIModel(
#     model_name="Qwen3-4B-Q6_K.gguf",
#     provider=OpenAIProvider(
#         base_url="http://localhost:8080/v1",
#         api_key="llamacpp",
#     ),
# )

model_settings = ModelSettings(
    max_tokens=2048,
    temperature=0.0,
    timeout=60,
)

agent = Agent(
    provider,
    output_type=Size, # using Name, Count, Size works fine, but using NameAndCount fails
    model_settings=model_settings,
    retries=2,
    instrument=True,
    system_prompt="""
    #Instructions:
    - Extract the count, the name and the size from the user input.""",
)


async def main():
    import logfire
    from loguru import logger

    logfire.configure(
        environment="test",
        service_name="bug.py",
        console=logfire.ConsoleOptions(verbose=True),
        send_to_logfire=True,
    )
    logger.configure(handlers=[logfire.loguru_handler()])
    logfire.instrument_httpx(capture_all=True)

    user_input = """
      <UserInput>
      The count is 9, the name is 'test', the size is 55
      </UserInput>
"""
    result = await agent.run(user_input)
    print(result.output)


if __name__ == "__main__":
    import asyncio

    asyncio.run(main())

Python, Pydantic AI & LLM client version

Python 3.12.8 (main, Jan 14 2025, 23:36:58) [Clang 19.1.6 ] on darwin
Pydantic-ai 0.1.10
Qwen/Qwen3-235B-A22B-fp8-tput
meta-llama/Llama-3.3-70B-Instruct-Turbo
Qwen/Qwen2.5-72B-Instruct-Turbo
Qwen/Qwen3-4B-fast
Qwen3-4B-Q6_K.gguf

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions