Eval bug: bizarre Jinja bug when trying to fix Qwen3 tool calling

[pwilkin]

Name and Version

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3080, compute capability 8.6, VMM: yes
version: 5361 (cf0a43b)
built with cc (Ubuntu 14.2.0-4ubuntu2) 14.2.0 for x86_64-linux-gnu

Operating systems

Linux

GGML backends

CUDA

Hardware

i7 9700K + GTX 3080 10GB VRAM

Models

Qwen3 14B (Q4_K_S)

Problem description & steps to reproduce

I've ran into some problems with the Qwen3 chat templates that prevent the proper usage of tools (more on that here: https://www.reddit.com/r/LocalLLaMA/comments/1klltt4/the_qwen3_chat_template_is_still_bugged/

I tweaked the template to make it work (or so I thought). However, I am still encountering a complely bizarre bug in my test case when using Qwen3 14B.

qwen3-jinja.txt

Even with the template being so careful about potential "content" blocks, I am still getting an error with the following conversation:

srv  log_server_r: response: {"error":{"code":500,"message":"[json.exception.out_of_range.403] key 'content' not found","type":"server_error"}}
request: {
  "model": "Qwen3-14B",
  "messages": [
    {
      "role": "system",
      "content": "The current date is Tue May 13 2025. You are an LLM agent that will be tasked with performing a variety of coding, analysis, classification and other tasks. \n       You will be given a set of instructions to follow. Respond to all queries as precisely as possible and follow all instructions given. \n       Pay attention to your entire context and don't forget it. Plan your responses before you write them. If the user specifies an output format, \n       follow the format exactly and do not deviate from it under any circumstances. You will use the tools provided to you to perform your tasks. /no_think"
    },
    {
      "role": "user",
      "content": "I need you to answer the following question: Previous name of current pope.\n      Among your available tools are a Wikipedia search tool and a Wikipedia page summary retrieval tool. Use these tools to find the answer. Do not stop\n      using the tools until you have enough data to provide an answer. Do not repeat searches, ask for more summaries instead. When you know the answer,\n      respond with it wrapped in the <answer></answer> tag, eg. <answer>Jane Doe</answer>."
    },
    {
      "role": "assistant",
      "tool_calls": [
        {
          "id": "8D1ylZfmAcMb4rPRuSRHEtdBApcHLqpK",
          "type": "function",
          "function": {
            "name": "searchWiki",
            "arguments": "{\"query\":\"Pope Francis\"}"
          }
        }
      ]
    },
    {
      "role": "tool",
      "tool_call_id": "8D1ylZfmAcMb4rPRuSRHEtdBApcHLqpK",
      "content": "[{\"title\":\"Pope Francis\",\"url\":\"https://en.wikipedia.org/?curid=1687680&action=raw\"},{\"title\":\"Death and funeral of Pope Francis\",\"url\":\"https://en.wikipedia.org/?curid=79767056&action=raw\"},{\"title\":\"Reactions to the death of Pope Francis\",\"url\":\"https://en.wikipedia.org/?curid=79768085&action=raw\"},{\"title\":\"Pope Francis and LGBTQ topics\",\"url\":\"https://en.wikipedia.org/?curid=58083832&action=raw\"},{\"title\":\"Theology of Pope Francis\",\"url\":\"https://en.wikipedia.org/?curid=38958547&action=raw\"}]"
    },
    {
      "role": "assistant",
      "content": "\n\nHere are the top 5 Wikipedia results for \"Pope Francis\":\n\n1. **[Pope Francis](https://en.wikipedia.org/?curid=1687680&action=raw)**  \n   *Main page about Pope Francis*\n\n2. **[Death and funeral of Pope Francis](https://en.wikipedia.org/?curid=79767056&action=raw)**  \n   *Hypothetical scenario or recent event details*\n\n3. **[Reactions to the death of Pope Francis](https://en.wikipedia.org/?curid=79768085&action=raw)**  \n   *Public and media responses to his death*\n\n4. **[Pope Francis and LGBTQ topics](https://en.wikipedia.org/?curid=58083832&action=raw)**  \n   *His stance and involvement with LGBTQ issues*\n\n5. **[Theology of Pope Francis](https://en.wikipedia.org/?curid=38958547&action=raw)**  \n   *Overview of his theological perspectives*\n\nWould you like a summary of the main **Pope Francis** page or any specific topic from the list?"
    },
    {
      "role": "user",
      "content": "Please use the tool data you just received to perform further tool calls or to provide the answer in <answer></answer> tags."
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "searchWiki",
        "description": "Searches Wikipedia for a query and returns an array of 5 top results as {title, url} objects.",
        "parameters": {
          "type": "object",
          "properties": {
            "query": {
              "type": "string",
              "description": "The query to search for."
            }
          },
          "required": [
            "query"
          ]
        }
      }
    },
    {
      "type": "function",
      "function": {
        "name": "getWikiSummary",
        "description": "Gets the summary of a Wikipedia page given a page ID and a topic.",
        "parameters": {
          "type": "object",
          "properties": {
            "pageid": {
              "type": "string",
              "description": "The page ID of the Wikipedia page to get."
            },
            "topic": {
              "type": "string",
              "description": "The topic to focus on in the summary."
            }
          },
          "required": [
            "pageid",
            "topic"
          ]
        }
      }
    }
  ],
  "stream": false,
  "top_p": 0.95,
  "frequency_penalty": 0,
  "presence_penalty": 0
}

Any ideas what could be going on here? All Jinja2 online parsers that I've tested parse this one OK (i.e. https://j2live.ttl255.com/ or https://nebula.packetcoders.io/j2-render/). I can't seem to figure out what's going on here. Is there any easy way to expose just the Jinja templater as a commandline tool so I could debug it?

First Bad Commit

33eff402

Relevant log output

srv  log_server_r: response: {"error":{"code":500,"message":"[json.exception.out_of_range.403] key 'content' not found","type":"server_error"}}

[andysalerno]

Ah, I had this same problem. I made a PR here: #13521

Summary is, in some recent commit(?) llama.cpp server stopped accepting requests that had messages where the "content" property is missing entirely. But it should accept such messages, because there is no "content" value required during a tool request message. A recent PR partially fixed the issue to allow null/empty values for "content", but did not allow scenarios where "content" is missing entirely.

[pwilkin]

Thanks @andysalerno , can confirm this is caused by regression introduced in 33eff40

[cgruver]

@pwilkin This might be related too - https://www.reddit.com/r/LocalLLaMA/comments/1klltt4/the_qwen3_chat_template_is_still_bugged/

[pwilkin]

@pwilkin This might be related too - https://www.reddit.com/r/LocalLLaMA/comments/1klltt4/the_qwen3_chat_template_is_still_bugged/

Given that I'm the author of that thread, yeah, it probably is 😆

This bug report is what happened after I fixed all the issues in the template described there.