The LLM Judge function takes too long, resulting in low GPU utilization.

[Fu-Fu-Fu-Fu]

The GPU is used normally during the model inference stage, but the LLM judge is time-consuming when calculating the score, resulting in low overall GPU utilization. Are there any methods or instructions to use parallel processing when calling the LLM judge API, or to separate the inference and scoring processes?
Thank you!

[kcz358]

The reason is probably because you are sending requests one-by-one. I would recommend you to change the call llm as judge that in process_result use threading. Also one thing to notice is that if you put the function in process_result when launching the lmms-eval on different gpu cards, by default it is parallel with num_gpu workers