Support more audio format in Gemini models

[ChenghaoMou]

Description

Gemini supports the following audio format MIME types:

WAV - audio/wav
MP3 - audio/mp3
AIFF - audio/aiff
AAC - audio/aac
OGG Vorbis - audio/ogg
FLAC - audio/flac

But currently AudioUrl only supports wav and mp3. This feature would allow direct file upload/url without any unnecessary file conversion.

References

• https://ai.google.dev/gemini-api/docs/audio#supported-formats
• https://ai.pydantic.dev/input/#user-side-download-vs-direct-file-url

[ChenghaoMou]

For people looking for a workaround, you can use the following wrapper class instead:

class GeminiAudioUrl(AudioUrl):
    # Gemini supports the following audio format MIME types:

    # WAV - audio/wav
    # MP3 - audio/mp3
    # AIFF - audio/aiff
    # AAC - audio/aac
    # OGG Vorbis - audio/ogg
    # FLAC - audio/flac

    @property
    def media_type(self) -> str:  # type: ignore[override]
        """Return the media type of the audio file, based on the url."""
        if self.url.endswith(".mp3"):
            return "audio/mpeg"
        elif self.url.endswith(".wav"):
            return "audio/wav"
        elif self.url.endswith(".ogg"):
            return "audio/ogg"
        elif self.url.endswith(".flac"):
            return "audio/flac"
        elif self.url.endswith(".aiff"):
            return "audio/aiff"
        elif self.url.endswith(".aac"):
            return "audio/aac"
        else:
            raise ValueError(f"Unknown audio file extension: {self.url}")

[DouweM]

@ChenghaoMou Thanks for the report. I think it would make sense to extend AudioUrl with the additional formats, and leave it to the model (API or our model class) to complain if it was passed a file it doesn't understand. There will always be differences in what each model actually supports, and I think it makes sense for Pydantic AI implement to set of all formats supported across all models, instead of a lowest common denominator.

Would you be up for filing a PR to add these new formats?

[ChenghaoMou]

@DouweM I agree. Here is my PR #2151. I tried to find all the types I can from all those model/vendors/providers that support audio url in chat endpoints. So far, I only found Gemini with the above-mentioned link. Others might have either inline audio data, urls with a transcription endpoint or realtime endpoint which I don't think are relevant.

[DouweM]

#2151 has been merged!