[Bug]: Wrong context length for Qwen 2.5 7B-Instruct?

[tjoymeed]

vllm 0.8.4
ms-swfit 3.4.0.dev0

---

HI all,

I met with the following error:

The model card of qwen-2.5-7B-instruct has the context length of 128K. But the vllm max_model_len is only 32K?

What's wrong?

Thanks a lot!

https://huggingface.co/Qwen/Qwen2.5-7B-Instruct

[rank3]: File "/home/tom/workspace/myvenv_msswift/lib/python3.10/site-packages/vllm/engine/arg_utils.py", line 1154, in create_engine_config
[rank3]: model_config = self.create_model_config()
[rank3]: File "/home/tom/workspace/myvenv_msswift/lib/python3.10/site-packages/vllm/engine/arg_utils.py", line 1042, in create_model_config
[rank3]: return ModelConfig(
[rank3]: File "/home/tom/workspace/myvenv_msswift/lib/python3.10/site-packages/vllm/config.py", line 480, in init
[rank3]: self.max_model_len = _get_and_verify_max_len(
[rank3]: File "/home/tom/workspace/myvenv_msswift/lib/python3.10/site-packages/vllm/config.py", line 2970, in _get_and_verify_max_len
[rank3]: raise ValueError(
[rank3]: ValueError: User-specified max_model_len (72000) is greater than the derived max_model_len (max_position_embeddings=32768 or model_max_length=None in model's config.json). This may lead to incorrect model outputs or CUDA errors. To allow overriding this maximum, set the env var VLLM_ALLOW_LONG_MAX_MODEL_LEN=1

[Jintao-Huang]

Yes, the context length of this model is 32K. For more details, please check the config.json.

[tjoymeed]

But the huggingface model card said it is 128K?

https://huggingface.co/Qwen/Qwen2.5-7B-Instruct

[TikoTako]

https://huggingface.co/Qwen/Qwen2.5-7B-Instruct#processing-long-texts <- its 128k if you use yarn