Eval bug: Qwen3-30B-A3B-Q4_K_M: Slows down when using the \no_think mode.

[ZUIcat]

Name and Version

build: 5335 (d891942) with MSVC 19.43.34810.0 for x64

Operating systems

Windows

GGML backends

Vulkan

Hardware

AMD 8840u (780m)

Models

Qwen3-30B-A3B-Q4_K_M.gguf
https://huggingface.co/bartowski/Qwen_Qwen3-30B-A3B-GGUF/blob/main/Qwen_Qwen3-30B-A3B-Q4_K_M.gguf

Problem description & steps to reproduce

Here are the command lines I used:

".\llama.cpp_release\llama-server.exe" -m ".\_model\Qwen3-30B-A3B-Q4_K_M__bartowski.gguf" -a "Qwen3-30B-A3B-Q4_K_M__bartowski" --host 0.0.0.0 --port 8090 --slots --props --metrics -np 1 -c 20480 -ngl 999 -ctk f16 -ctv f16 --no-mmap --keep 0 --cache-reuse 256 --jinja --chat-template-file ".\_model\Qwen3-30B-A3B-Q4_K_M__bartowski.jinja"

I'm not sure if this qualifies as a bug, but I noticed something unusual.

When using the Qwen3-30B-A3B-Q4_K_M.gguf model, if /no_think is enabled in the first query, the second response becomes significantly slower. llama-server web ui show that the second response's Prompt Tokens nearly equal the sum of the first response + second query.

First response:

Prompt
- Tokens: 1
- Time: 80.396 ms
- Speed: 12.4 t/s
Generation
- Tokens: 1116
- Time: 48067.824 ms
- Speed: 23.2 t/s

Second response:

Prompt
- Tokens: 1126
- Time: 37210.091 ms
- Speed: 30.3 t/s
Generation
- Tokens: 178
- Time: 8257.332 ms
- Speed: 21.6 t/s

However, if /no_think is not used in the first query, the second response remains fast, with Prompt Tokens only reflecting the second query.

First response:

Prompt
- Tokens: 1
- Time: 43.725 ms
- Speed: 22.9 t/s
Generation
- Tokens: 1420
- Time: 62888.84 ms
- Speed: 22.6 t/s

Second response:

Prompt
- Tokens: 16
- Time: 713.299 ms
- Speed: 22.4 t/s
Generation
- Tokens: 378
- Time: 17734.479 ms
- Speed: 21.3 t/s

Additional notes:

• I used this template because Qwen3's default template throws errors.
• Without --cache-reuse 256, all second responses show high Prompt Tokens and become slow, regardless of /no_think.

Could this be a caching-related issue?

First Bad Commit

No response

Relevant log output

".\llama.cpp_release\llama-server.exe" -m ".\_model\Qwen3-30B-A3B-Q4_K_M__bartowski.gguf" -a "Qwen3-30B-A3B-Q4_K_M__bartowski" --host 0.0.0.0 --port 8090 --slots --props --metrics -np 1 -c 20480 -ngl 999 -ctk f16 -ctv f16 --no-mmap --keep 0 --cache-reuse 256 --jinja --chat-template-file ".\_model\Qwen3-30B-A3B-Q4_K_M__bartowski.jinja"

load_backend: loaded RPC backend from ggml-rpc.dll
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon 780M Graphics (AMD proprietary driver) | uma: 1 | fp16: 1 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat
load_backend: loaded Vulkan backend from ggml-vulkan.dll
load_backend: loaded CPU backend from ggml-cpu-icelake.dll
build: 5335 (d8919424) with MSVC 19.43.34810.0 for x64
system info: n_threads = 8, n_threads_batch = 8, total_threads = 16

system_info: n_threads = 8 (n_threads_batch = 8) / 16 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 |

main: binding port with default address family
main: HTTP server is listening, hostname: 0.0.0.0, port: 8090, http threads: 15
main: loading model
srv    load_model: loading model '.\_model\Qwen3-30B-A3B-Q4_K_M__bartowski.gguf'
llama_model_load_from_file_impl: using device Vulkan0 (AMD Radeon 780M Graphics) - 16128 MiB free
llama_model_loader: loaded meta data with 41 key-value pairs and 579 tensors from .\_model\Qwen3-30B-A3B-Q4_K_M__bartowski.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen3moe
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Qwen3 30B A3B
llama_model_loader: - kv   3:                           general.basename str              = Qwen3
llama_model_loader: - kv   4:                         general.size_label str              = 30B-A3B
llama_model_loader: - kv   5:                            general.license str              = apache-2.0
llama_model_loader: - kv   6:                       general.license.link str              = https://huggingface.co/Qwen/Qwen3-30B...
llama_model_loader: - kv   7:                   general.base_model.count u32              = 1
llama_model_loader: - kv   8:                  general.base_model.0.name str              = Qwen3 30B A3B Base
llama_model_loader: - kv   9:          general.base_model.0.organization str              = Qwen
llama_model_loader: - kv  10:              general.base_model.0.repo_url str              = https://huggingface.co/Qwen/Qwen3-30B...
llama_model_loader: - kv  11:                               general.tags arr[str,1]       = ["text-generation"]
llama_model_loader: - kv  12:                       qwen3moe.block_count u32              = 48
llama_model_loader: - kv  13:                    qwen3moe.context_length u32              = 32768
llama_model_loader: - kv  14:                  qwen3moe.embedding_length u32              = 2048
llama_model_loader: - kv  15:               qwen3moe.feed_forward_length u32              = 6144
llama_model_loader: - kv  16:              qwen3moe.attention.head_count u32              = 32
llama_model_loader: - kv  17:           qwen3moe.attention.head_count_kv u32              = 4
llama_model_loader: - kv  18:                    qwen3moe.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  19:  qwen3moe.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  20:                 qwen3moe.expert_used_count u32              = 8
llama_model_loader: - kv  21:              qwen3moe.attention.key_length u32              = 128
llama_model_loader: - kv  22:            qwen3moe.attention.value_length u32              = 128
llama_model_loader: - kv  23:                      qwen3moe.expert_count u32              = 128
llama_model_loader: - kv  24:        qwen3moe.expert_feed_forward_length u32              = 768
llama_model_loader: - kv  25:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  26:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  27:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  28:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  29:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  30:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  31:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  32:                tokenizer.ggml.bos_token_id u32              = 151643
llama_model_loader: - kv  33:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  34:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
llama_model_loader: - kv  35:               general.quantization_version u32              = 2
llama_model_loader: - kv  36:                          general.file_type u32              = 15
llama_model_loader: - kv  37:                      quantize.imatrix.file str              = /models_out/Qwen3-30B-A3B-GGUF/Qwen_Q...
llama_model_loader: - kv  38:                   quantize.imatrix.dataset str              = /training_data/calibration_datav3.txt
llama_model_loader: - kv  39:             quantize.imatrix.entries_count i32              = 384
llama_model_loader: - kv  40:              quantize.imatrix.chunks_count i32              = 209
llama_model_loader: - type  f32:  241 tensors
llama_model_loader: - type q8_0:   48 tensors
llama_model_loader: - type q4_K:  193 tensors
llama_model_loader: - type q5_K:   48 tensors
llama_model_loader: - type q6_K:   49 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 17.35 GiB (4.88 BPW)
load: special tokens cache size = 26
load: token to piece cache size = 0.9311 MB
print_info: arch             = qwen3moe
print_info: vocab_only       = 0
print_info: n_ctx_train      = 32768
print_info: n_embd           = 2048
print_info: n_layer          = 48
print_info: n_head           = 32
print_info: n_head_kv        = 4
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: n_swa_pattern    = 1
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 8
print_info: n_embd_k_gqa     = 512
print_info: n_embd_v_gqa     = 512
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 6144
print_info: n_expert         = 128
print_info: n_expert_used    = 8
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 2
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 32768
print_info: rope_finetuned   = unknown
print_info: ssm_d_conv       = 0
print_info: ssm_d_inner      = 0
print_info: ssm_d_state      = 0
print_info: ssm_dt_rank      = 0
print_info: ssm_dt_b_c_rms   = 0
print_info: model type       = 30B.A3B
print_info: model params     = 30.53 B
print_info: general.name     = Qwen3 30B A3B
print_info: n_ff_exp         = 768
print_info: vocab type       = BPE
print_info: n_vocab          = 151936
print_info: n_merges         = 151387
print_info: BOS token        = 151643 '<|endoftext|>'
print_info: EOS token        = 151645 '<|im_end|>'
print_info: EOT token        = 151645 '<|im_end|>'
print_info: PAD token        = 151643 '<|endoftext|>'
print_info: LF token         = 198 'Ċ'
print_info: FIM PRE token    = 151659 '<|fim_prefix|>'
print_info: FIM SUF token    = 151661 '<|fim_suffix|>'
print_info: FIM MID token    = 151660 '<|fim_middle|>'
print_info: FIM PAD token    = 151662 '<|fim_pad|>'
print_info: FIM REP token    = 151663 '<|repo_name|>'
print_info: FIM SEP token    = 151664 '<|file_sep|>'
print_info: EOG token        = 151643 '<|endoftext|>'
print_info: EOG token        = 151645 '<|im_end|>'
print_info: EOG token        = 151662 '<|fim_pad|>'
print_info: EOG token        = 151663 '<|repo_name|>'
print_info: EOG token        = 151664 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: offloading 48 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 49/49 layers to GPU
load_tensors:      Vulkan0 model buffer size = 17596.42 MiB
load_tensors:          CPU model buffer size =   166.92 MiB
....................................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 20480
llama_context: n_ctx_per_seq = 20480
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 0
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (20480) < n_ctx_train (32768) -- the full capacity of the model will not be utilized
llama_context: Vulkan_Host  output buffer size =     0.58 MiB
llama_kv_cache_unified: kv_size = 20480, type_k = 'f16', type_v = 'f16', n_layer = 48, can_shift = 1, padding = 32
llama_kv_cache_unified:    Vulkan0 KV buffer size =  1920.00 MiB
llama_kv_cache_unified: KV self size  = 1920.00 MiB, K (f16):  960.00 MiB, V (f16):  960.00 MiB
llama_context:    Vulkan0 compute buffer size =  1344.00 MiB
llama_context: Vulkan_Host compute buffer size =    44.01 MiB
llama_context: graph nodes  = 3126
llama_context: graph splits = 2
common_init_from_params: setting dry_penalty_last_n to ctx_size = 20480
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv          init: initializing slots, n_slots = 1
slot         init: id  0 | task -1 | new slot n_ctx_slot = 20480
main: model loaded
main: chat template, chat_template: {%- if tools %}
    {{- '<|im_start|>system\n' }}
    {%- if messages[0].role == 'system' %}
        {{- messages[0].content + '\n\n' }}
    {%- endif %}
    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
    {%- for tool in tools %}
        {{- "\n" }}
        {{- tool | tojson }}
    {%- endfor %}
    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
{%- else %}
    {%- if messages[0].role == 'system' %}
        {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
    {%- endif %}
{%- endif %}

{%- set ns = namespace(multi_step_tool=true, last_query_index=0) %}
{%- for message in messages %}
    {%- if ns.multi_step_tool %}
        {%- if message.role == "user" %}
            {%- if not (message.content[0:14] == '<tool_response>' and message.content[-15:] == '</tool_response>') %}
                {%- set ns.multi_step_tool = false %}
                {%- set ns.last_query_index = loop.index0 %}
            {%- endif %}
        {%- endif %}
    {%- endif %}
{%- endfor %}

{%- for message in messages %}
    {%- if message.role == "user" or (message.role == "system" and loop.index0 != 0) %}
        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>\n' }}
    {%- elif message.role == "assistant" %}
        {%- set content = message.content %}
        {%- set reasoning_content = '' %}
        {%- if message.reasoning_content is defined and message.reasoning_content %}
            {%- set reasoning_content = message.reasoning_content %}
        {%- else %}
            {%- if '<think>' in message.content and '</think>' in message.content %}
                {%- set think_start = message.content.find('<think>') + 7 %}
                {%- set think_end = message.content.find('</think>') %}
                {%- set reasoning_content = message.content[think_start:think_end] %}
                {%- set content = message.content[think_end + 8:] %}
            {%- endif %}
        {%- endif %}
        {%- if loop.index0 > ns.last_query_index %}
            {%- if loop.last or (not loop.last and reasoning_content) %}
                {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
            {%- else %}
                {{- '<|im_start|>' + message.role + '\n' + content }}
            {%- endif %}
        {%- else %}
            {{- '<|im_start|>' + message.role + '\n' + content }}
        {%- endif %}
        {%- if message.tool_calls %}
            {%- for tool_call in message.tool_calls %}
                {{- '\n' }}
                {%- if tool_call.function %}
                    {%- set tool_call = tool_call.function %}
                {%- endif %}
                {{- '<tool_call>\n{"name": "' + tool_call.name + '", "arguments": ' }}
                {%- if tool_call.arguments is string %}
                    {{- tool_call.arguments }}
                {%- else %}
                    {{- tool_call.arguments | tojson }}
                {%- endif %}
                {{- '}\n</tool_call>' }}
            {%- endfor %}
        {%- endif %}
        {{- '<|im_end|>\n' }}
    {%- elif message.role == "tool" %}
        {%- set prev_role = (loop.index0 > 0) and (messages[loop.index0 - 1].role) or '' %}
        {%- if loop.index0 == 0 or prev_role != "tool" %}
            {{- '<|im_start|>user\n' }}
        {%- endif %}
        {{- '<tool_response>\n' + message.content + '\n</tool_response>' }}
        {%- set next_role = (loop.index0 + 1 < messages|length) and (messages[loop.index0 + 1].role) or '' %}
        {%- if loop.index0 == messages|length - 1 or next_role != "tool" %}
            {{- '<|im_end|>\n' }}
        {%- endif %}
    {%- endif %}
{%- endfor %}

{%- if add_generation_prompt %}
    {{- '<|im_start|>assistant\n' }}
    {%- if enable_thinking is defined and enable_thinking == false %}
        {{- '<think>\n\n</think>\n\n' }}
    {%- endif %}
{%- endif %}
, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
'
main: server is listening on http://0.0.0.0:8090 - starting the main loop
srv  update_slots: all slots are idle

[ggerganov]

The /no_think still generates opening and closing think tags for the first response, which are not sent with the second query causing the first response to be reprocessed.

You can fix this by adding --cache-reuse 256 to your llama-server command:

[ZUIcat]

The /no_think still generates opening and closing think tags for the first response, which are not sent with the second query causing the first response to be reprocessed.

You can fix this by adding --cache-reuse 256 to your llama-server command:

Yes, as mentioned above, without --cache-reuse 256, it runs slowly regardless of /no_think. But with --cache-reuse 256, it's faster without /no_think, while with /no_think remains slow.

[ggerganov]

Sorry, I missed that you already used --cache-reuse. In that case, this is most likely another instance of the different tokenization bug: #11970 (comment)

If you could do some extra analysis by printing the tokens and confirming that this is the same issue, it would be helpful.