Making File Downloading Model Profile Compatible

[ChenghaoMou]

Description

When using OpenRouter with a Gemini model, it will try to download a file with a gs:// url even though Gemini supposedly can handle it directly.

import asyncio
import os

from dotenv import load_dotenv
from pydantic_ai import Agent
from pydantic_ai import AudioUrl
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.openrouter import OpenRouterProvider


async def main() -> None:
    load_dotenv()

    provider = OpenRouterProvider(
        api_key=os.environ["OPENROUTER_API_KEY"],
    )
    agent = Agent(
        model=OpenAIModel(
            "google/gemini-2.0-flash-001",
            provider=provider,
            profile=provider.model_profile,
        ),
        system_prompt="You are a helpful assistant that can transcribe audio to text.",
    )
    result = await agent.run(
        [
            "Here is the audio file to transcribe:",
            AudioUrl(
                url="gs://...",
                force_download=False,
            ),
        ],
    )
    print(result.output)


if __name__ == "__main__":
    asyncio.run(main())

The error stack

  File "/Users/chenghao/Developer/ai-reports/.venv/lib/python3.12/site-packages/pydantic_ai/models/openai.py", line 229, in request
    response = await self._completions_create(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chenghao/Developer/ai-reports/.venv/lib/python3.12/site-packages/pydantic_ai/models/openai.py", line 293, in _completions_create
    openai_messages = await self._map_messages(messages)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chenghao/Developer/ai-reports/.venv/lib/python3.12/site-packages/pydantic_ai/models/openai.py", line 416, in _map_messages
    async for item in self._map_user_message(message):
  File "/Users/chenghao/Developer/ai-reports/.venv/lib/python3.12/site-packages/pydantic_ai/models/openai.py", line 489, in _map_user_message
    yield await self._map_user_prompt(part)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chenghao/Developer/ai-reports/.venv/lib/python3.12/site-packages/pydantic_ai/models/openai.py", line 545, in _map_user_prompt
    downloaded_item = await download_item(item, data_format='base64', type_format='extension')
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chenghao/Developer/ai-reports/.venv/lib/python3.12/site-packages/pydantic_ai/models/__init__.py", line 689, in download_item
    raise UserError('Downloading from protocol "gs://" is not supported.')
pydantic_ai.exceptions.UserError: Downloading from protocol "gs://" is not supported.

I am aware gs url support is available with Gemini/Google model classes from https://ai.pydantic.dev/input/#user-side-download-vs-direct-file-url, but I hope this downloading behavior could be more consistent with model profiles, in addition to model classes.

References

• Add ModelProfile to let model-specific behaviors be configured independent of the model class #1835

[ChenghaoMou]

I can't find any evidence that Gemini's OpenAI-compatible API supports gs urls, so this might not be possible after all.

[ChenghaoMou]

There is something alarmingly wrong with OpenRouter's proxy:

For the same 10-minute wav file, and the following openrouter code:

import base64
import json
import os

import requests
from dotenv import load_dotenv

load_dotenv()

with open("audio.wav", "rb") as f:
    audio_bytes = f.read()
encoded_string = base64.b64encode(audio_bytes).decode("utf-8")
print(len(encoded_string))

response = requests.post(
    url="https://openrouter.ai/api/v1/chat/completions",
    headers={
        "Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}",
        "Content-Type": "application/json",
    },
    data=json.dumps(
        {
            "model": "google/gemini-2.5-flash",
            "messages": [
                {
                    "role": "system",
                    "content": "You are a helpful assistant that can transcribe audio to text.",
                },
                {
                    "role": "user",
                    "content": [
                        {
                            "type": "text",
                            "text": "Here is the audio file to transcribe:",
                        },
                        {
                            "type": "input_audio",
                            "input_audio": {
                                "data": encoded_string,
                                "format": "wav",
                            },
                        },
                    ],
                },
            ],
        }
    ),
)

print(response.json())

It almost always generate something like

{'id': 'gen-1751881264-to3mRDcMBFLeWgxq5qLN', 'provider': 'Google', 'model': 'google/gemini-2.5-flash', 'object': 'chat.completion', 'created': 1751881264, 'choices': [{'logprobs': None, 'finish_reason': 'stop', 'native_finish_reason': 'STOP', 'index': 0, 'message': {'role': 'assistant', 'content': "It's amazing how things have changed so drastically. In the past it was a very sort of conservative culture and it wasn't really accepted to do anything that was not the norm. And people that did do things that were not the norm were really looked down upon. So now it's completely different. It's almost reversed.", 'refusal': None, 'reasoning': None}}], 'usage': {'prompt_tokens': 22, 'completion_tokens': 69, 'total_tokens': 91}}

Notice how few tokens it consumed and generated (22 input tokens for 10-min audio?).

Compared to using GeminiModel directly with Pydantic AI:

async def main() -> None:
    load_dotenv()

    model = GeminiModel(
        "gemini-2.5-flash",
        provider="google-vertex",
    )
    agent = Agent(
        model=model,
        system_prompt="You are a helpful assistant that can transcribe audio to text.",
        instrument=True,
    )
    if os.path.exists("audio.wav"):
        with open("audio.wav", "rb") as f:
            audio_bytes = f.read()

    print(len(audio_bytes))
    with open("audio.wav", "wb") as f:
        f.write(audio_bytes)
    result = await agent.run(
        [
            "Here is the audio file to transcribe:",
            BinaryContent(
                data=audio_bytes,
                media_type="audio/wav",
            ),
        ],
    )
    print(result.output)


if __name__ == "__main__":
    asyncio.run(main())

Hello, can you hear me?
How about now? Can you still hear me?
Hey, what? What brings you here?
I'm just walking.
So, are you heading somewhere?
Okay. I'm going to grab something from the mall.
Do you need anything?
Okay.
You're free to go.
But, I don't think you are leaving, aren't you?
Yeah, that's weird. I thought you were leaving.
Okay, what do you want to talk about?
Okay, fair enough.
Alright, goodbye.
Understood. See you around.
But you're still here.
Sorry, I'm not sure I understand, because you said you were leaving.
Why are you talking about time?
No, I asked, why are you talking about time?
What's your obsession with seconds?
No, I understand why it's important, but I don't know why you are repeating the seconds every single turn.
That's how normal people talk nowadays?
But I think the whole purpose of this conversation is a real-life simulation. If normal people don't talk about time in their conversations, why aren't you?
So you're saying normal conversations, people will stating times as a fact, every single time.
Even if it's relevant, you think putting time in every sentence you speak is normal?
Yeah, it's very difficult for me to understand why you're putting it in every speech.
In every turn.
Yeah, normal people would be aware of the time is passing, but the fact you are announcing that time is rather strange.
But what kind of clarity does that bring to the conversation?

[ChenghaoMou]

It seems that OpenRouter does not support audio input for Gemini models, inline or not: https://discord.com/channels/1091220969173028894/1363452151132717146