Add optional src or metadata field to BinaryContent

[lshamis]

Description

There isn't a good way to tag BinaryContent with metadata. For example, I might want to do

result = agent.run_sync(
    [
        'What company is this logo from?',
        BinaryContent(
            data=image_response.content,
            media_type='image/png',
            src='https://iili.io/3Hs4FMg.png',
        ),
    ]
)

I'm trying to figure out a good way to specify as a tool argument which binary object the tool should act upon. If there is a src field, than the tool can scan for the associated data.
Maybe there is a more direct way to do this??

References

No response

[DouweM]

@lshamis So you're providing multiple BinaryContents, and then you want the LLM to call a tool with a reference to one of those BinaryContents as an argument? Would it be an option to including labels/ids for each file in your user messages, e.g. preceding each BinaryContent with "This is file '<ID>':" and then explaining to the model through the tool arguments that the file_id argument should take the ID of a file that was previously shared?

The problem with your suggested solution is that there's no way to pass that src field to the model in a way that's directly tied to the file, as they all have their own API format for passing inline binary data that doesn't include an arbitrary-text field like that. So the only way to give a name to each file is through the user messages, and while we could automatically insert a user message with the src/label, I think it's more clear if you do it explicitly, especially since you're also responsible for looking up the appropriate file again in the tool.

[lshamis]

@DouweM The issue is on the tool side, not the LLM side. I am already adding "This is file '<ID>':". Once the model specifies that it wants to act on file_id, how does the tool find the associated BinaryContent from the messages in an unambiguous way?

[DouweM]

@lshamis Is it an option to store the BinaryContents in a dict with those IDs as keys, and look have the tool look them up from there, instead of having to read through the messages to find it?

[lshamis]

Building a new object store feels like a heavy weight solution for what should just be tagged metadata.

[DouweM]

@lshamis I don't think of adding a single dict as particularly heavy-weight, and it seems more clean to me to look up a key in the dict than to have to iterate over the message history to find the ModelRequests, iterate over their parts to find the UserPromptParts, iterate over their content to find the BinaryContents, and then filter on the src field. But I imagine your real-life code is a bit more complex than the example in the description so perhaps building that dict is not as straight forward as it seems to me like it would be.

We recently added a way to set metadata on the ToolCallResult (#2060), and I can see that also being valuable in other places. But before we commit to adding it everywhere it could make sense, I want to see if there are other acceptable workarounds that get you unblocked now instead of at the next Pydantic AI release.

If tracking BinaryContent in a dict is not an option, have you considered:

• Using ImageUrl instead of BinaryContent, so you can identify by URL? At least in your example there seems to be a URL.
• Creating your own BinaryContent subclass with an extra src field?

I'm not fundamentally opposed to adding a BinaryContent.identifier field, just trying to see if it's the best/only option.