feat: rlhf generation samples log to swanlab

[Zeyi-Lin]

PR type

• Bug Fix
• New Feature
• Document Updates
• More Models or Datasets Support

PR information

This PR supports using swanlab to record intermediate results when performing grpo and rm reinforcement learning training in ms-swift. The results can be seen in the following link:

:( I am unable to upload images to the Github Issue normally.

Demo:

• grpo: https://swanlab.cn/@ZeyiLin/ms-swift-rlhf/runs/d661ty9mslogsgk41fp0p/chart
• rm: https://swanlab.cn/@ZeyiLin/ms-swift-rlhf/runs/rd08p2cc0rqebk0xmpa5v/chart

[hjh0119]

Thanks! It's really needed.

could you also patch the TRL profiling function to upload profiling_metrics to SwanLab?

Related code:
trl: https://github.com/huggingface/trl/blob/v0.19.1/trl/extras/profiling.py#L31-L68
swift: https://github.com/modelscope/ms-swift/blob/main/swift/trainers/rlhf_trainer/grpo_trainer.py#L30C1

[Zeyi-Lin]

@hjh0119 Great suggestion, I have implemented the patch function. Please check if there are other areas that need improvement.

demo：https://swanlab.cn/@ZeyiLin/ms-swift-rlhf/runs/40xdzwqpxpnf50edof9zg/chart

[hjh0119]

LGTM

Nit: maybe patch profiling_context as well?

[Zeyi-Lin]

LGTM

Nit: maybe patch profiling_context as well?Nit：也许也修补 profiling_context？

Thanks. I guess you mean profiling_decorator? In the latest commit I have added patch_profiling_decorator.

:( Unit test errors do not seem to be caused by my changes; it seems to be an error caused by a Docker container.