Target module not supported: 运行官方文档中Qwen2.5-VL Grounding推理脚本示例报错

[caseclose]

Describe the bug
运行官方文档中的Qwen2.5-VL Grounding任务示例的推理脚本出现报错.
推理脚本直接使用：

CUDA_VISIBLE_DEVICES=0 \
swift infer \
    --adapters swift/test_grounding \
    --stream true \
    --load_data_args true \
    --max_new_tokens 512 \
    --dataset_num_proc 4

报错信息：

Traceback (most recent call last):
  File "Code/ms-swift/swift/cli/infer.py", line 5, in <module>
    infer_main()
  File "Code/ms-swift/swift/llm/infer/infer.py", line 237, in infer_main
    return SwiftInfer(args).main()
  File "Code/ms-swift/swift/llm/infer/infer.py", line 32, in __init__
    model, self.template = prepare_model_template(args)
  File "Code/ms-swift/swift/llm/infer/utils.py", line 145, in prepare_model_template
    model = prepare_adapter(args, model)
  File "Code/ms-swift/swift/llm/infer/utils.py", line 136, in prepare_adapter
    model = tuner.from_pretrained(model, adapter)
  File "Code/ms-swift/swift/tuners/base.py", line 909, in from_pretrained
    peft_model = load_peft_model(model, 'default')
  File "Code/ms-swift/swift/tuners/base.py", line 895, in load_peft_model
    return PeftModel.from_pretrained(
  File "Code/ms-swift/swift/tuners/peft.py", line 345, in from_pretrained
    return module_class.from_pretrained(model, model_id, *args, **kwargs)
  File "/root/miniconda3/lib/python3.9/site-packages/peft/peft_model.py", line 429, in from_pretrained
    model = MODEL_TYPE_TO_PEFT_MODEL_MAPPING[config.task_type](model, config, adapter_name)
  File "/root/miniconda3/lib/python3.9/site-packages/peft/peft_model.py", line 1395, in __init__
    super().__init__(model, peft_config, adapter_name)
  File "/root/miniconda3/lib/python3.9/site-packages/peft/peft_model.py", line 138, in __init__
    self.base_model = cls(model, {adapter_name: peft_config}, adapter_name)
  File "Code/ms-swift/swift/tuners/peft.py", line 292, in __new_init__
    self.__init_origin__(model, config, adapter_name)
  File "/root/miniconda3/lib/python3.9/site-packages/peft/tuners/lora/model.py", line 139, in __init__
    super().__init__(model, config, adapter_name)
  File "/root/miniconda3/lib/python3.9/site-packages/peft/tuners/tuners_utils.py", line 166, in __init__
    self.inject_adapter(self.model, adapter_name)
  File "/root/miniconda3/lib/python3.9/site-packages/peft/tuners/tuners_utils.py", line 372, in inject_adapter
    self._create_and_replace(peft_config, adapter_name, target, target_name, parent, current_key=key)
  File "Code/ms-swift/swift/tuners/peft.py", line 92, in _create_and_replace_hook
    return self._create_and_replace_origin(peft_config, adapter_name, target, *args, **kwargs)
  File "/root/miniconda3/lib/python3.9/site-packages/peft/tuners/lora/model.py", line 223, in _create_and_replace
    new_module = self._create_new_module(lora_config, adapter_name, target, **kwargs)
  File "/root/miniconda3/lib/python3.9/site-packages/peft/tuners/lora/model.py", line 320, in _create_new_module
    raise ValueError(
ValueError: Target module Qwen2_5_VLModel(
  (embed_tokens): Embedding(152064, 3584)
  (layers): ModuleList(
    (0-27): 28 x Qwen2_5_VLDecoderLayer(
      (self_attn): Qwen2_5_VLSdpaAttention(
        (q_proj): Linear(in_features=3584, out_features=3584, bias=True)
        (k_proj): Linear(in_features=3584, out_features=512, bias=True)
        (v_proj): Linear(in_features=3584, out_features=512, bias=True)
        (o_proj): Linear(in_features=3584, out_features=3584, bias=False)
        (rotary_emb): Qwen2_5_VLRotaryEmbedding()
      )
      (mlp): Qwen2MLP(
        (gate_proj): Linear(in_features=3584, out_features=18944, bias=False)
        (up_proj): Linear(in_features=3584, out_features=18944, bias=False)
        (down_proj): Linear(in_features=18944, out_features=3584, bias=False)
        (act_fn): SiLU()
      )
      (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
      (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
    )
  )
  (norm): Qwen2RMSNorm((3584,), eps=1e-06)
  (rotary_emb): Qwen2_5_VLRotaryEmbedding()
) is not supported. Currently, only the following modules are supported: `torch.nn.Linear`, `torch.nn.Embedding`, `torch.nn.Conv2d`, `transformers.pytorch_utils.Conv1D`.

Your hardware and system info
pip list环境信息：

Package                           Version                   Editable project location
--------------------------------- ------------------------- ---------------------------------------------------------------------------------------------
absl-py                           1.4.0
accelerate                        1.6.0
addict                            2.4.0
aiofiles                          23.2.1
aiohttp                           3.9.3
aiohttp-cors                      0.7.0
aioprometheus                     23.12.0
aiorwlock                         1.4.0
aiosignal                         1.3.1
airportsdata                      20241001
albumentations                    1.4.0
alembic                           1.13.1
aliyun-python-sdk-core            2.13.36
aliyun-python-sdk-kms             2.16.1
altair                            5.0.1
annotated-types                   0.7.0
anthropic                         0.31.2
antlr4-python3-runtime            4.7.2
anyio                             3.7.1
apex                              0.1
appdirs                           1.4.4
apted                             1.0.3
archspec                          0.2.3
argon2-cffi                       21.3.0
argon2-cffi-bindings              21.2.0
arrow                             1.2.3
astor                             0.8.1
asttokens                         2.2.1
async-lru                         2.0.4
async-timeout                     4.0.3
attrdict                          2.0.1
attrs                             23.1.0
audioread                         3.0.1
autoawq                           0.2.8
av                                14.0.1
awscli                            1.29.26
azure-core                        1.29.2
azure-identity                    1.14.0
azure-storage-blob                12.17.0
azure-storage-file-datalake       12.12.0
Babel                             2.12.1
backcall                          0.2.0
backoff                           2.2.1
base58                            2.1.1
bcrypt                            4.0.1
beautifulsoup4                    4.12.2
better-exceptions                 0.3.3
binpacking                        1.5.2
bitsandbytes                      0.45.1
black                             24.1.0
blake3                            1.0.2
bleach                            6.0.0
blessed                           1.20.0
blinker                           1.7.0
blis                              0.7.10
boltons                           23.0.0
boto3                             1.28.26
botocore                          1.31.26
braceexpand                       0.1.7
Brotli                            1.0.9
brotlipy                          0.7.0
build                             1.2.1
cachetools                        5.3.1
catalogue                         2.0.9
certifi                           2024.8.30
cffi                              1.16.0
cfgv                              3.3.1
chardet                           5.2.0
charset-normalizer                3.3.2
circuitbreaker                    1.4.0
click                             8.1.6
clip                              1.0
cloudpickle                       3.0.0
cmake                             3.27.2
colorama                          0.4.4
colorful                          0.5.6
colossalai                        0.3.1
comm                              0.1.4
compressed-tensors                0.9.1
conda                             24.9.2
conda-content-trust               0.1.3
conda-libmamba-solver             24.7.0
conda-package-handling            2.3.0
conda_package_streaming           0.10.0
confection                        0.1.1
ConfigArgParse                    1.7
contexttimer                      0.3.3
contourpy                         1.1.0
cpm-kernels                       1.0.11
crcmod                            1.7
cryptography                      39.0.1
cupy-cuda11x                      13.0.0
cycler                            0.11.0
cymem                             2.0.7
dacite                            1.9.2
DataProperty                      1.0.1
datasets                          3.2.0
debugpy                           1.6.7.post1
decorator                         5.1.1
decord                            0.6.0
deepspeed                         0.16.3
defusedxml                        0.7.1
Deprecated                        1.2.14
depyf                             0.18.0
diffusers                         0.32.2
dill                              0.3.7
diskcache                         5.6.3
distilabel                        1.5.2
distlib                           0.3.7
distro                            1.9.0
dm-tree                           0.1.8
dnspython                         2.4.2
docker-pycreds                    0.4.0
docopt                            0.6.2
docutils                          0.16
easydict                          1.10
einops                            0.8.0
einx                              0.3.0
et-xmlfile                        1.1.0
evaluate                          0.4.2.dev0                
exceptiongroup                    1.1.2
executing                         1.2.0
fabric                            3.2.1
Farama-Notifications              0.0.4
fastapi                           0.110.0
fastjsonschema                    2.18.0
fastrlock                         0.8.2
ffmpy                             0.3.1
filelock                          3.17.0
fire                              0.6.0
flag_gems                         2.0
flake8                            7.1.1
flash-attn                        2.7.2.post1
flashinfer                        0.2.0.post1+cu124torch2.4
Flask                             3.0.2
Flask-BasicAuth                   0.2.0
Flask-Cors                        4.0.0
Flask-Login                       0.6.3
fonttools                         4.42.0
fqdn                              1.5.1
frozendict                        2.4.4
frozenlist                        1.4.0
fschat                            0.2.36
fsspec                            2023.10.0
ftfy                              6.1.1
func_timeout                      4.3.5
future                            1.0.0
fuzzywuzzy                        0.18.0
gevent                            24.2.1
geventhttpclient                  2.0.11
gguf                              0.10.0
gitdb                             4.0.10
GitPython                         3.1.32
google-api-core                   2.11.1
google-auth                       2.22.0
google-auth-oauthlib              1.0.0
google-cloud-core                 2.3.3
google-cloud-storage              2.10.0
google-crc32c                     1.5.0
google-resumable-media            2.5.0
googleapis-common-protos          1.60.0
gpustat                           1.1
gradio                            4.26.0
gradio_client                     0.15.1
greenlet                          3.0.3
grokadamw                         0.1.2
grpcio                            1.57.0
gymnasium                         0.29.1
h11                               0.14.0
h5py                              3.9.0
hf_transfer                       0.1.6
hjson                             3.1.0
httpcore                          1.0.4
httptools                         0.6.1
httpx                             0.27.0
huggingface-hub                   0.30.2
identify                          2.5.26
idna                              3.7
imageio                           2.34.0
immutabledict                     4.2.1
importlib-metadata                6.8.0
importlib-resources               6.0.1
iniconfig                         2.0.0
interegular                       0.3.3
invoke                            2.2.0
iopath                            0.1.10
ipdb                              0.13.13
ipykernel                         6.29.5
ipython                           8.14.0
ipywidgets                        7.4.2
isodate                           0.6.1
isoduration                       20.11.0
itsdangerous                      2.1.2
jax-jumpy                         1.0.0
jedi                              0.19.0
jieba                             0.42.1
Jinja2                            3.1.2
jiter                             0.5.0
jmespath                          0.10.0
joblib                            1.3.2
json5                             0.9.14
jsonlines                         4.0.0
jsonpatch                         1.32
jsonpickle                        3.0.2
jsonpointer                       2.1
jsonschema                        4.23.0
jsonschema-specifications         2023.7.1
jupyter_client                    8.3.0
jupyter_core                      5.3.1
jupyter-events                    0.7.0
jupyter-lsp                       2.2.0
jupyter_server                    2.7.0
jupyter_server_terminals          0.4.4
jupyterlab                        4.0.4
jupyterlab-pygments               0.2.2
jupyterlab_server                 2.24.0
kiwisolver                        1.4.4
langcodes                         3.3.0
lark                              1.2.2
latex2sympy2                      1.9.1
lazy_loader                       0.3
Levenshtein                       0.26.1
libmambapy                        1.5.8
librosa                           0.9.0
liger_kernel_nightly              0.5.2.dev20250203052716
lightgbm                          4.1.0
linkify-it-py                     2.0.2
lit                               16.0.6
litellm                           1.42.5
llvmlite                          0.42.0
lm-format-enforcer                0.10.9
lmdb                              1.4.1
lmdeploy                          0.7.0.post2
lmms_eval                         0.1.1                     
locust                            2.24.0
loguru                            0.7.0
lomo-optim                        0.1.1
lxml                              5.2.1
lz4                               4.3.3
Mako                              1.3.2
mamba                             1.5.8
Markdown                          3.4.4
markdown-it-py                    2.2.0
markdown2                         2.4.10
MarkupSafe                        2.1.3
matplotlib                        3.7.2
matplotlib-inline                 0.1.6
mbstrdecoder                      1.1.3
mccabe                            0.7.0
mdit-py-plugins                   0.3.3
mdurl                             0.1.2
menuinst                          2.1.1
mistral_common                    1.5.1
mistune                           3.0.1
mmengine                          0.8.4
mmengine-lite                     0.10.4
model-index                       0.1.11
modelscope                        1.24.1
monotonic                         1.6
mosaicml-streaming                0.9.0
mpmath                            1.3.0
ms_swift                          3.3.0.post1
msal                              1.23.0
msal-extensions                   1.0.0
msgpack                           1.0.8
msgspec                           0.18.6
multidict                         6.0.4
multiprocess                      0.70.15
MultiScaleDeformableAttention     1.0
munch                             2.5.0
murmurhash                        1.0.9
mypy-extensions                   1.0.0
natsort                           8.4.0
nbclient                          0.8.0
nbconvert                         7.7.3
nbformat                          5.9.2
nest-asyncio                      1.6.0
networkx                          3.2.1
nh3                               0.2.14
ninja                             1.11.1.1
nltk                              3.8
nodeenv                           1.8.0
notebook                          7.0.2
notebook_shim                     0.2.3
numba                             0.59.0
numexpr                           2.10.0
numpy                             1.26.4
nvidia-cublas-cu12                12.4.5.8
nvidia-cuda-cupti-cu12            12.4.127
nvidia-cuda-nvrtc-cu12            12.4.127
nvidia-cuda-runtime-cu12          12.4.127
nvidia-cudnn-cu12                 9.1.0.70
nvidia-cufft-cu12                 11.2.1.3
nvidia-curand-cu12                10.3.5.147
nvidia-cusolver-cu12              11.6.1.9
nvidia-cusparse-cu12              12.3.1.170
nvidia-ml-py                      12.560.30
nvidia-nccl-cu12                  2.21.5
nvidia-nvjitlink-cu12             12.4.127
nvidia-nvtx-cu12                  12.4.127
nvitop                            1.4.2
oauthlib                          3.2.2
oci                               2.110.0
omegaconf                         2.3.0
open-clip-torch                   2.20.0
openai                            1.54.3
OpenCC                            1.1.9
opencensus                        0.11.4
opencensus-context                0.1.3
opencv-python                     4.9.0.80
opencv-python-headless            4.9.0.80
opendatalab                       0.0.10
openmim                           0.3.9
openpyxl                          3.1.2
opentelemetry-api                 1.23.0
openxlab                          0.1.2
opt-einsum                        3.3.0
ordered-set                       4.1.0
orjson                            3.10.15
oss2                              2.17.0
outlines                          0.1.11
outlines_core                     0.1.26
overrides                         7.4.0
packaging                         24.1
paddle-bfloat                     0.1.7
pandas                            1.5.3
pandocfilters                     1.5.0
paramiko                          3.3.1
parso                             0.8.3
partial-json-parser               0.2.1.1.post4
pathspec                          0.12.1
pathvalidate                      3.2.0
pathy                             0.10.2
peft                              0.15.2
pexpect                           4.8.0
pickleshare                       0.7.5
pillow                            10.4.0
pip                               24.2
platformdirs                      3.10.0
pluggy                            1.5.0
pooch                             1.8.2
portalocker                       3.1.1
pre-commit                        3.3.3
preshed                           3.0.8
prettytable                       3.12.0
prometheus_client                 0.20.0
prometheus-fastapi-instrumentator 7.0.0
prompt-toolkit                    3.0.39
protobuf                          3.20.3
psutil                            5.9.8
ptyprocess                        0.7.0
pure-eval                         0.2.2
py-cpuinfo                        9.0.0
py-spy                            0.3.14
py3nvml                           0.2.7
pyairports                        2.1.1
pyan3                             1.2.0
pyarrow                           15.0.0
pyarrow-hotfix                    0.6
pyasn1                            0.5.0
pyasn1-modules                    0.3.0
pyav                              12.0.5
pybind11                          2.13.1
pycocoevalcap                     1.2
pycocotools                       2.0.6
pycodestyle                       2.12.1
pycosat                           0.6.4
pycountry                         24.6.1
pycparser                         2.21
pycryptodome                      3.18.0
pydantic                          2.9.2
pydantic_core                     2.23.4
pydeck                            0.8.1b0
pydub                             0.25.1
pyext                             0.7
pyflakes                          3.2.0
Pygments                          2.16.1
PyJWT                             2.8.0
pymongo                           4.5.0
PyNaCl                            1.5.0
pynvml                            12.0.0
pyOpenSSL                         23.0.0
pyparsing                         3.0.9
pyproject_hooks                   1.1.0
PySocks                           1.7.1
pytablewriter                     1.2.0
pytest                            8.3.2
pytest-asyncio                    0.24.0
python-dateutil                   2.8.2
python-dotenv                     1.0.1
python-json-logger                2.0.7
python-Levenshtein                0.26.1
python-multipart                  0.0.9
python-rapidjson                  1.18
python-snappy                     0.6.1
pytz                              2023.3
PyYAML                            6.0.1
pyzmq                             26.2.0
quantile-python                   1.1
qudida                            0.0.4
qwen-vl-utils                     0.0.8
rank-bm25                         0.2.2
RapidFuzz                         3.10.1
ray                               2.9.3
ray-cpp                           2.9.3
rdkit                             2024.3.2
referencing                       0.30.2
regex                             2024.5.15
requests                          2.32.3
requests-oauthlib                 1.3.1
resampy                           0.4.3
RestrictedPython                  7.0
retrying                          1.3.4
rfc3339-validator                 0.1.4
rfc3986-validator                 0.1.1
rich                              13.9.4
rouge                             1.0.1
rouge-chinese                     1.0.3
rouge-score                       0.1.2
roundrobin                        0.0.4
rpds-py                           0.9.2
rsa                               4.7.2
ruamel.yaml                       0.17.21
ruamel.yaml.clib                  0.2.6
ruff                              0.5.3
s3transfer                        0.6.1
sacrebleu                         2.4.1
sacred                            0.8.4
safetensors                       0.4.5
schedulefree                      1.3
scikit-image                      0.22.0
scikit-learn                      1.5.0
scipy                             1.11.1
seaborn                           0.13.2
semantic-version                  2.10.0
Send2Trash                        1.8.2
sentence-transformers             2.2.2
sentencepiece                     0.2.0
sentry-sdk                        1.40.6
setproctitle                      1.3.3
setuptools                        69.5.1
sgl-kernel                        0.0.2.post18
sglang                            0.4.1.post7
shapely                           2.0.3
shellingham                       1.5.4
shortuuid                         1.0.11
simplejson                        3.20.1
simsimd                           1.4.0
six                               1.16.0
smart-open                        6.3.0
smmap                             5.0.0
sniffio                           1.3.0
sortedcontainers                  2.4.0
soundfile                         0.12.1
soupsieve                         2.4.1
spacy                             3.6.1
spacy-legacy                      3.0.12
spacy-loggers                     1.0.4
SQLAlchemy                        2.0.27
sqlitedict                        2.1.0
srsly                             2.4.7
sse-starlette                     2.0.0
st-img-pastebutton                0.0.4
stack-data                        0.6.2
starlette                         0.36.3
streamlit                         1.42.0
sty                               1.0.6
svgwrite                          1.4.3
sympy                             1.13.1
tabledata                         1.3.3
tabulate                          0.9.0
tblib                             3.0.0
tcolorpy                          0.1.4
tenacity                          8.2.3
tensor-parallel                   2.0.0
tensorboard                       2.14.0
tensorboard-data-server           0.7.1
tensorboardX                      2.6.2.2
termcolor                         2.3.0
terminado                         0.17.1
thinc                             8.1.11
threadpoolctl                     3.2.0
tifffile                          2024.2.12
tiktoken                          0.7.0
timeout-decorator                 0.5.0
timm                              0.9.5
tinycss2                          1.2.1
tokenizers                        0.21.0
toml                              0.10.2
tomli                             2.0.1
tomlkit                           0.12.0
toolz                             0.12.0
torch                             2.5.1
torchao                           0.8.0+cu124
torchaudio                        2.5.1
torchtext                         0.18.0
torchvision                       0.20.1
tornado                           6.3.2
tqdm                              4.67.1
tqdm-multiprocess                 0.0.11
traitlets                         5.9.0
transformers                      4.49.0
transformers-stream-generator     0.0.5
tree-sitter                       0.21.3
tree-sitter-languages             1.10.2
triton                            3.1.0
tritonclient                      2.47.0
trl                               0.15.0.dev0
typepy                            1.3.2
typer                             0.12.3
typing_extensions                 4.12.2
tzdata                            2023.3
tzlocal                           5.2
uc-micro-py                       1.0.2
universal_pathlib                 0.2.6
uri-template                      1.3.0
urllib3                           1.26.20
uvicorn                           0.27.1
uvloop                            0.19.0
validators                        0.22.0
vector-quantize-pytorch           1.18.5
virtualenv                        20.24.2
vllm                              0.7.2
vllm-flash-attn                   2.6.2
wandb                             0.16.3
wasabi                            1.1.2
watchdog                          4.0.0
watchfiles                        0.21.0
wavedrom                          2.0.3.post3
wcwidth                           0.2.6
webcolors                         1.13
webdataset                        0.2.100
webencodings                      0.5.1
websocket-client                  1.6.1
websockets                        11.0.3
Werkzeug                          3.0.1
wheel                             0.38.4
widgetsnbextension                3.4.2
wrapt                             1.15.0
xformers                          0.0.28.post3
xgrammar                          0.1.10
XlsxWriter                        3.2.0
xmltodict                         0.13.0
xxhash                            3.3.0
yapf                              0.32.0
yarl                              1.9.2
zipp                              3.16.2
zmq                               0.0.0
zope.event                        5.0
zope.interface                    6.2
zss                               1.2.0
zstandard                         0.19.0
zstd                              1.5.5.1