-
Notifications
You must be signed in to change notification settings - Fork 2.3k
Description
Describe the bug
Specific versions of huggingface_hub>=0.35.0 and vllm<=0.10 are required to run the evaluation job in the Submit your final project! portion of the Instruction Tuning topic.
To Reproduce
Run the hf jobs evaluation script with huggingface_hub<0.35.0 or vllm>0.10
Screenshots
Existing CLI command
Error when huggingface_hub<0.35 caused by huggingface_hub expecting a script file in previous versions:
FileNotFoundError: [Errno 2] No such file or directory: 'lighteval'
Error when vllm>=0.10 caused by vllm deprecating the use of prompt_token_ids in recent versions:
TypeError: LLM.generate() got an unexpected keyword argument 'prompt_token_ids'
Additional context
The commands that work:
- Install a recent version of
huggingface_hub
pip install -U huggingface_hub>=0.35.0
- Change the CLI command to use previous version of
vllm
hf jobs uv run --flavor "a10g-large" --secrets HF_TOKEN --with "lighteval[vllm]" --with 'vllm<=0.10' lighteval vllm "model_name=<your-username>/<your-model-name>" "lighteval|gsm8k|0|0" --push-to-hub --results-org <your-username>