Tool support

[simonw]

For LLM 0.26: https://llm.datasette.io/en/stable/changelog.html#v0-26

Relevant docs (annoyingly they don't include curl examples): https://docs.mistral.ai/capabilities/function_calling/

[simonw]

Trying to reverse engineer the HTTP API from their Python library:

% git clone https://github.com/mistralai/client-python
Cloning into 'client-python'...
remote: Enumerating objects: 4446, done.
remote: Counting objects: 100% (1646/1646), done.
remote: Compressing objects: 100% (617/617), done.
remote: Total 4446 (delta 1207), reused 1194 (delta 1016), pack-reused 2800 (from 1)
Receiving objects: 100% (4446/4446), 1.25 MiB | 5.73 MiB/s, done.
Resolving deltas: 100% (3018/3018), done.
% cd client-python 
client-python % files-to-prompt . | ttok
551781
client-python % cd src/mistralai 
mistralai % files-to-prompt . | ttok
212410

That's enough tokens that I'll try Gemini 2.5 Flash on it:

llm -f /tmp/mistral.txt \
  -m gemini-2.5-flash-preview-05-20 \
  -s 'Generate comprehensive HTTP API documentation showing how function 
calling works, include example curl commands for each step'

Wow that worked astonishingly well: https://gist.github.com/simonw/03f2049cd9af6dc072e1ee33461f3437

[simonw]

Following the guide on https://llm.datasette.io/en/stable/plugins/advanced-model-plugins.html#supporting-tools

• Add supports_tools = True to your model class.
• If prompt.tools is populated, turn that list of llm.Tool objects into the correct format for your model.
• Look out for requests to call tools in the responses from your model. Call response.add_tool_call(llm.ToolCall(...)) for each of those. This should work for streaming and non-streaming and async and non-async cases.
• If your prompt has a prompt.tool_results list, pass the information from those llm.ToolResult objects to your model.
• Include prompt.tools and prompt.tool_results and tool calls from response.tool_calls_or_raise() in the conversation history constructed by your plugin.
• Make sure your code is OK with prompts that do not have prompt.prompt set to a value, since they may be carrying exclusively the results of a tool call.

[simonw]

From the Gemini 2.5 Flash reverse engineered docs:

    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_current_weather",
          "description": "Get the current weather in a given location",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
                "type": "string",
                "description": "The city and state, e.g. San Francisco, CA"
              },
              "unit": {
                "type": "string",
                "enum": ["celsius", "fahrenheit"],
                "description": "The unit of temperature to use. Infer this from the user''s query."
              }
            },
            "required": ["location"]
          }
        }
      }
    ],
    "tool_choice": "auto"

[simonw]

More from the Gemini docs:

    "messages": [
      {
        "role": "user",
        "content": "What is the weather in Paris, France today?"
      },
      {
        "role": "assistant",
        "content": null,
        "tool_calls": [
          {
            "id": "call_abc123",
            "type": "function",
            "function": {
              "name": "get_current_weather",
              "arguments": "{\"location\": \"Paris, France\", \"unit\": \"celsius\"}"
            }
          }
        ]
      },
      {
        "role": "tool",
        "tool_call_id": "call_abc123",
        "content": "{\"weather\": \"15C\", \"conditions\": \"sunny\"}" # This is the actual JSON result
      }
    ],

[simonw]

Getting this right is frustrating due to the lack of Mistral docs - so I pasted my code, the llm_gemini code and those generated API docs from Gemini Flash into Claude Opus 4 and got it to write the thing (after pasting in a few error messages): https://claude.ai/share/7c609a61-4b32-45ca-bdca-31bf4ef25d2d

And now this works (with extra debug output):

(llm-mistral) (base) simon@Mac llm-mistral % LLM_RAISE_ERRORS=1 llm --td -m mistral-small -T llm_version 'run the llm_version tool'
{
    'model': 'mistral-small-latest',
    'messages': [{'role': 'user', 'content': 'run the llm_version tool'}],
    'temperature': 0.7,
    'top_p': 1,
    'tools': [
        {
            'type': 'function',
            'function': {
                'name': 'llm_version',
                'description': 'Return the installed version of llm',
                'parameters': {'properties': {}, 'type': 'object'}
            }
        }
    ],
    'tool_choice': 'auto'
}

Tool call: llm_version({})
  0.26

{
    'model': 'mistral-small-latest',
    'messages': [
        {'role': 'user', 'content': 'run the llm_version tool'},
        {
            'role': 'assistant',
            'content': None,
            'tool_calls': [{'id': 'yzF7c7nW6', 'type': 'function', 'function': {'name': 'llm_version', 'arguments': '{}'}}]
        },
        {'role': 'tool', 'tool_call_id': 'yzF7c7nW6', 'content': '"0.26"'}
    ],
    'temperature': 0.7,
    'top_p': 1,
    'tools': [
        {
            'type': 'function',
            'function': {
                'name': 'llm_version',
                'description': 'Return the installed version of llm',
                'parameters': {'properties': {}, 'type': 'object'}
            }
        }
    ],
    'tool_choice': 'auto'
}
The installed version of the language model is 0.26.

[simonw]

Doesn't do conversations yet though:

llm --td -m mistral-small -T llm_version 'run the llm_version tool' -T llm_time

{
    'model': 'mistral-small-latest',
    'messages': [
        {'role': 'user', 'content': 'run the llm_version tool'},
        {
            'role': 'assistant',
            'content': None,
            'tool_calls': [{'id': 'stOdfuL8A', 'type': 'function', 'function': {'name': 'llm_version', 'arguments': '{}'}}]
        },
        {'role': 'user', 'content': ''},
        {'role': 'assistant', 'content': 'The installed version of the language model is 0.26.'},
        {'role': 'user', 'content': 'and the time'}
    ],
    'temperature': 0.7,
    'top_p': 1,
    'tools': [
        {
            'type': 'function',
            'function': {
                'name': 'llm_version',
                'description': 'Return the installed version of llm',
                'parameters': {'properties': {}, 'type': 'object'}
            }
        },
        {
            'type': 'function',
            'function': {
                'name': 'llm_time',
                'description': 'Returns the current time, as local time and UTC',
                'parameters': {'properties': {}, 'type': 'object'}
            }
        }
    ],
    'tool_choice': 'auto'
}
Error: 400: invalid_request_error - Not the same number of function calls and responses

[simonw]

Pasted that problem into Claude and it fixed it too: https://claude.ai/share/7c609a61-4b32-45ca-bdca-31bf4ef25d2d

[simonw]

Now this works:

(llm-mistral) (base) simon@Mac llm-mistral % LLM_RAISE_ERRORS=1 llm --td -m mistral-small -T llm_version 'run the llm_version tool' -T llm_time
{
    'model': 'mistral-small-latest',
    'messages': [{'role': 'user', 'content': 'run the llm_version tool'}],
    'temperature': 0.7,
    'top_p': 1,
    'tools': [
        {
            'type': 'function',
            'function': {
                'name': 'llm_version',
                'description': 'Return the installed version of llm',
                'parameters': {'properties': {}, 'type': 'object'}
            }
        },
        {
            'type': 'function',
            'function': {
                'name': 'llm_time',
                'description': 'Returns the current time, as local time and UTC',
                'parameters': {'properties': {}, 'type': 'object'}
            }
        }
    ],
    'tool_choice': 'auto'
}

Tool call: llm_version({})
  0.26

{
    'model': 'mistral-small-latest',
    'messages': [
        {'role': 'user', 'content': 'run the llm_version tool'},
        {
            'role': 'assistant',
            'content': None,
            'tool_calls': [{'id': 'jGUdf6ShQ', 'type': 'function', 'function': {'name': 'llm_version', 'arguments': '{}'}}]
        },
        {'role': 'tool', 'tool_call_id': 'jGUdf6ShQ', 'content': '"0.26"'}
    ],
    'temperature': 0.7,
    'top_p': 1,
    'tools': [
        {
            'type': 'function',
            'function': {
                'name': 'llm_version',
                'description': 'Return the installed version of llm',
                'parameters': {'properties': {}, 'type': 'object'}
            }
        },
        {
            'type': 'function',
            'function': {
                'name': 'llm_time',
                'description': 'Returns the current time, as local time and UTC',
                'parameters': {'properties': {}, 'type': 'object'}
            }
        }
    ],
    'tool_choice': 'auto'
}
The installed version of the LLM is 0.26.
(llm-mistral) (base) simon@Mac llm-mistral % llm -c 'and the time'                                                                             
{
    'model': 'mistral-small-latest',
    'messages': [
        {'role': 'user', 'content': 'run the llm_version tool'},
        {
            'role': 'assistant',
            'content': None,
            'tool_calls': [{'id': 'jGUdf6ShQ', 'type': 'function', 'function': {'name': 'llm_version', 'arguments': '{}'}}]
        },
        {'role': 'tool', 'tool_call_id': 'jGUdf6ShQ', 'content': '"0.26"'},
        {'role': 'assistant', 'content': 'The installed version of the LLM is 0.26.'},
        {'role': 'user', 'content': 'and the time'}
    ],
    'temperature': 0.7,
    'top_p': 1,
    'tools': [
        {
            'type': 'function',
            'function': {
                'name': 'llm_version',
                'description': 'Return the installed version of llm',
                'parameters': {'properties': {}, 'type': 'object'}
            }
        },
        {
            'type': 'function',
            'function': {
                'name': 'llm_time',
                'description': 'Returns the current time, as local time and UTC',
                'parameters': {'properties': {}, 'type': 'object'}
            }
        }
    ],
    'tool_choice': 'auto'
}
{
    'model': 'mistral-small-latest',
    'messages': [
        {'role': 'user', 'content': 'run the llm_version tool'},
        {
            'role': 'assistant',
            'content': None,
            'tool_calls': [{'id': 'jGUdf6ShQ', 'type': 'function', 'function': {'name': 'llm_version', 'arguments': '{}'}}]
        },
        {'role': 'tool', 'tool_call_id': 'jGUdf6ShQ', 'content': '"0.26"'},
        {'role': 'assistant', 'content': 'The installed version of the LLM is 0.26.'},
        {'role': 'user', 'content': 'and the time'},
        {
            'role': 'assistant',
            'content': None,
            'tool_calls': [{'id': 'rIfK8n1lT', 'type': 'function', 'function': {'name': 'llm_time', 'arguments': '{}'}}]
        },
        {
            'role': 'tool',
            'tool_call_id': 'rIfK8n1lT',
            'content': '"{\\"utc_time\\": \\"2025-05-28 18:24:14 UTC\\", \\"utc_time_iso\\": \\"2025-05-28T18:24:14.350670+00:00\\", \\"local_timezone\\": \\"PDT\\", \\"local_time\\": \\"2025-05-28 11:24:14\\", \\"timezone_offset\\": \\"UTC-7:00\\", \\"is_dst\\": true}"'
        }
    ],
    'temperature': 0.7,
    'top_p': 1,
    'tools': [
        {
            'type': 'function',
            'function': {
                'name': 'llm_version',
                'description': 'Return the installed version of llm',
                'parameters': {'properties': {}, 'type': 'object'}
            }
        },
        {
            'type': 'function',
            'function': {
                'name': 'llm_time',
                'description': 'Returns the current time, as local time and UTC',
                'parameters': {'properties': {}, 'type': 'object'}
            }
        }
    ],
    'tool_choice': 'auto'
}
The current time is:

- **UTC Time:** 2025-05-28 18:24:14 UTC
- **Local Time (PDT):** 2025-05-28 11:24:14
- **Timezone Offset:** UTC-7:00
- **Daylight Saving Time:** Yes

[simonw]

I'll finish this in a PR.