Skip to content

Incorrect CLI command for evaluating model with hf jobs for instruction tuning topic #247

@Josh-Weston

Description

@Josh-Weston

Describe the bug
Specific versions of huggingface_hub>=0.35.0 and vllm<=0.10 are required to run the evaluation job in the Submit your final project! portion of the Instruction Tuning topic.

To Reproduce

Run the hf jobs evaluation script with huggingface_hub<0.35.0 or vllm>0.10

Screenshots

Existing CLI command

Image

Error when huggingface_hub<0.35 caused by huggingface_hub expecting a script file in previous versions:
FileNotFoundError: [Errno 2] No such file or directory: 'lighteval'

Error when vllm>=0.10 caused by vllm deprecating the use of prompt_token_ids in recent versions:
TypeError: LLM.generate() got an unexpected keyword argument 'prompt_token_ids'

Additional context

The commands that work:

  1. Install a recent version of huggingface_hub

pip install -U huggingface_hub>=0.35.0

  1. Change the CLI command to use previous version of vllm

hf jobs uv run --flavor "a10g-large" --secrets HF_TOKEN --with "lighteval[vllm]" --with 'vllm<=0.10' lighteval vllm "model_name=<your-username>/<your-model-name>" "lighteval|gsm8k|0|0" --push-to-hub --results-org <your-username>

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions