NotImplementedError when merge lora for qwen2 vl 7B

[thesby]

Your hardware and system info
Write your system info like CUDA version/system/GPU/torch version here(在这里给出硬件信息和系统信息，如CUDA版本，系统，GPU型号和torch版本等)
system: ubuntu 22.04
python: 3.11
GPU: 4060ti 16G
torch: 2.4

Additional context

swift export \
    --ckpt_dir output/qwen2-vl-7b-instruct/v0-20241025-202858/checkpoint-200 \
    --merge_lora true

[INFO:swift] model_kwargs: {'device_map': 'cuda:0'}
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
We suggest you to set `torch_dtype=torch.float16` for better efficiency with AWQ.
`Qwen2VLRotaryEmbedding` can now be fully parameterized by passing the model config through the `config` argument. All other arguments will be removed in v4.46
Loading checkpoint shards: 100%|█████████████████████████████| 2/2 [00:20<00:00, 10.04s/it]
[INFO:swift] model.max_model_len: 32768
[INFO:swift] PeftModelForCausalLM: 1846.6985M Params (0.0000M Trainable [0.0000%]), 873.1431M Buffers.
[INFO:swift] system: You are a helpful assistant.
[INFO:swift] Merge LoRA...
Traceback (most recent call last):
  File "/home/xxx/miniconda3/envs/swift/lib/python3.10/site-packages/swift/cli/export.py", line 5, in <module>
    export_main()
  File "/home/xxx/miniconda3/envs/swift/lib/python3.10/site-packages/swift/utils/run_utils.py", line 32, in x_main
    result = llm_x(args, **kwargs)
  File "/home/xxx/miniconda3/envs/swift/lib/python3.10/site-packages/swift/llm/export.py", line 197, in llm_export
    merge_lora(args, device_map=args.merge_device_map)
  File "/home/xxx/miniconda3/envs/swift/lib/python3.10/site-packages/swift/llm/infer.py", line 121, in merge_lora
    Swift.merge_and_unload(model)
  File "/home/xxx/miniconda3/envs/swift/lib/python3.10/site-packages/swift/tuners/base.py", line 739, in merge_and_unload
    model.merge_and_unload()
  File "/home/xxx/miniconda3/envs/swift/lib/python3.10/site-packages/peft/tuners/lora/model.py", line 868, in merge_and_unload
    return self._unload_and_optionally_merge(
  File "/home/xxx/miniconda3/envs/swift/lib/python3.10/site-packages/peft/tuners/lora/model.py", line 490, in _unload_and_optionally_merge
    target.merge(safe_merge=safe_merge, adapter_names=adapter_names)
  File "/home/xxx/miniconda3/envs/swift/lib/python3.10/site-packages/peft/tuners/tuners_utils.py", line 657, in merge
    raise NotImplementedError
NotImplementedError

[thesby]

是因为AWQ模型不能 merge lora 吗？那我要怎么推理呢

[Jintao-Huang]

awq不支持merge-lora. 只能直接推理

[Jintao-Huang]

CUDA_VISIBLE_DEVICES swift infer --ckpt_dir xxx --load_dataset_config true

单样本推理可以查看文档

[thesby]

可以把 AWQ 训练的 lora 和原模型合并吗？

[Jintao-Huang]

这样会有精度损失啊

不过 你额外指定--model_id_or_path 试试, 测测性能如何

[thesby]

我直接用原始模型训练了。但是我把 finetune 后的模型，做了一下 GPTQ 8bit 量化，对比量化前的模型，效果差不少。

[Jintao-Huang]

效果差不少是 直接无法正常输出嘛，因为刚修复了一个gptq的bug

[Jintao-Huang]

gptq_quantizer.serialization_keys.append('block_name_to_quantize')

https://github.com/modelscope/ms-swift/pull/2378/files

[thesby]

效果差不少是 直接无法正常输出嘛，因为刚修复了一个gptq的bug

感谢回复。我微调的 image_caption 任务，原模型能够正常描述人物属性，但是量化后描述不太准确。能够正常描述，但是准确率下降有点多。