Support direct file URL for Gemini models on Vertex AI and GLA

[vricciardulli]

Description

NOTE: This feature request only applies to Gemini models, both on Vertex AI and, in minor measure, on GLA.

As stated in section Document Input of the documentation, documents are provided to Gemini models only as binary data. Users can accomplish this by using:

• BinaryContent: in which case users specify the binary data themselves.
• DocumentUrl: in which case the document content is downloaded behind the scenes and then injected in the request body sent to Gemini.

Gemini also supports direct file URL as user prompt, via field fileData. The structure for this field is already present in PydanticAI but currently unused.

The benefits of supporting this type of user prompt are:

• No download happens on the client side
• Google Cloud Storage URIs are supported (Vertex AI only)
• Up to 1 public YouTube video can be directly analyzed by Gemini per request
• General public HTTP URLs are also supported (not clear which URLs are supported by the GLA)

A downside that comes to mind is that this is a very specific case (only Gemini models and only the Vertex AI provider), which doesn't fit too well with the current implementation of PydanticAI, where the type of user prompt is general among providers and models. But this is just my opinion.

Thanks for taking the time to check this 🙏 I've also opened a PR to propose a straightforward implementation for this.

[vricciardulli]

Edited title and description to reflect the fact that Gemini on the GLA does support the fileData field (link to docs), but in minor measure: I was only able to get it to work for YouTube URLs.

[kraft87]

Using the gs uri with vertex-ai is also required to get timestamps for the audio.

[dhimmel]

I believe I just hit this issue when trying to run an agent that includes the following against the google-vertex provider

ImageUrl(url="gs://bucket/path.png")

which gave the error:

httpx.UnsupportedProtocol: Request URL has an unsupported protocol 'gs://

I have no opinion whether ImageUrl should support GCS URIs or some other class, but would like some way to get the GCS URI passed through all the way to Vertex AI without download occurring in the pydantic-ai application as the data contents can both be sensitive and large.

Thanks @vricciardulli for your ongoing work in #1136.

I also found this prior issue that asked for something similar:

from vertexai.generative_models import Part

video_file = Part.from_uri(
    uri="gs://cloud-samples-data/generative-ai/video/pixel8.mp4",
    mime_type="video/mp4",
)