Missing PRM class for HuggingFaceH4 PRMs

[danielkorat]

Hi @plaguss @lewtun

I would like to use the PRMs from the training example.
However, I can’t find a suitable PRM class for generating completions.

reward_models.py contains only PRM classes for the MathShepherd and RLHFLlow PRM types.

Am I missing something? Is this missing class available somewhere?

Thank you 🙏

[plaguss]

Hi @danielkorat, to generate PRM samples we used distilabel, an example can be seen here, it follows the same approach taken in MathShepherd's paper. The dataset used for the training example you shared is HuggingFaceH4/prm800k-trl-dedup, we just cleaned up and prepare the format to work with TRL.

[danielkorat]

Thank you @plaguss

I think you misunderstood my issue.
I would like to run test_time_compute.py using your trained model HuggingFaceH4/Qwen2.5-Math-1.5B-Instruct-PRM-0.2.
However, it seems that the MathSheperd and RLHFlow PRM classes are not suitable for this model, so it's not possible.

Thanks

[plaguss]

You are right, that's missing, let me check