Misc. bug:

[JarWang2008]

Name and Version

git clone https://github.com/ggerganov/llama.cpp
make GGML_CUDA=1

I llama.cpp build info:
I UNAME_S: Linux
I UNAME_P: x86_64
I UNAME_M: x86_64
I CFLAGS: -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -std=c11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes -Werror=implicit-int -Werror=implicit-function-declaration -Wdouble-promotion -pthread -march=native -mtune=native
I CXXFLAGS: -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread -Wno-array-bounds -Wno-format-truncation -march=native -mtune=native
I NVCCFLAGS: -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread -Wno-pedantic -Xcompiler "-Wno-array-bounds -Wno-format-truncation -march=native -mtune=native "
I LDFLAGS:
I CC: cc (GCC) 7.3.1 20180303 (Red Hat 7.3.1-5)
I CXX: g++ (GCC) 7.3.1 20180303 (Red Hat 7.3.1-5)

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

Documentation/Github

Command line

cd llama.app
python  convert-hf-to-gguf.py  /data/qwen2.5/saves/export  --outtype f16 --outfile /data/qwen2.5/saves/gguf/qwen2.5-1.5b-zpert.gguf

Problem description & steps to reproduce

execute the following commands, it raised an exception:
cd llama.app
python convert-hf-to-gguf.py /data/qwen2.5/saves/export --outtype f16 --outfile /data/qwen2.5/saves/gguf/qwen2.5-1.5b-zpert.gguf

exceptiion:

Loading model: export
gguf: This GGUF file is for Little Endian only
Set model parameters
Set model tokenizer
gguf: Adding 151387 merge(s).
gguf: Setting special token type eos to 151645
gguf: Setting special token type pad to 151643
gguf: Setting special token type bos to 151643
gguf: Setting add_bos_token to False
gguf: Setting chat_template to {%- if tools %}
{{- '<|im_start|>system\n' }}
{%- if messages[0]['role'] == 'system' %}
{{- messages[0]['content'] }}
{%- else %}
{{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
{%- endif %}
{{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within XML tags:\n" }}
{%- for tool in tools %}
{{- "\n" }}
{{- tool | tojson }}
{%- endfor %}
{{- "\n\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{"name": , "arguments": }\n</tool_call><|im_end|>\n" }}
{%- else %}
{%- if messages[0]['role'] == 'system' %}
{{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
{%- else %}
{{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- for message in messages %}
{%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
{{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
{%- elif message.role == "assistant" %}
{{- '<|im_start|>' + message.role }}
{%- if message.content %}
{{- '\n' + message.content }}
{%- endif %}
{%- for tool_call in message.tool_calls %}
{%- if tool_call.function is defined %}
{%- set tool_call = tool_call.function %}
{%- endif %}
{{- '\n<tool_call>\n{"name": "' }}
{{- tool_call.name }}
{{- '", "arguments": ' }}
{{- tool_call.arguments | tojson }}
{{- '}\n</tool_call>' }}
{%- endfor %}
{{- '<|im_end|>\n' }}
{%- elif message.role == "tool" %}
{%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
{{- '<|im_start|>user' }}
{%- endif %}
{{- '\n<tool_response>\n' }}
{{- message.content }}
{{- '\n</tool_response>' }}
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
{{- '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|im_start|>assistant\n' }}
{%- endif %}

Exporting model to '/data/qwen2.5/saves/gguf/qwen2.5-1.5b-zpert.gguf'
gguf: loading model part 'model.safetensors'
token_embd.weight, n_dims = 2, torch.bfloat16 --> float16
blk.0.attn_norm.weight, n_dims = 1, torch.bfloat16 --> float32
blk.0.ffn_down.weight, n_dims = 2, torch.bfloat16 --> float16
blk.0.ffn_gate.weight, n_dims = 2, torch.bfloat16 --> float16
blk.0.ffn_up.weight, n_dims = 2, torch.bfloat16 --> float16
blk.0.ffn_norm.weight, n_dims = 1, torch.bfloat16 --> float32
Can not map tensor 'model.layers.0.self_attn.k_proj.bias'

First Bad Commit

No response

Relevant log output

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.