GKD代码加载模型卡死

[fffffurina]

Describe the bug
gkd代码，模型一直加载不上，直到timeout报错

[Jintao-Huang]

有报错截图嘛，在哪里抛出来的

[fffffurina]

应该是同时加载学生和教师两个模型导致的NCCL

[Jintao-Huang]

脚本发一下

还有是从ms-swift的哪一行抛出来的，你的截图看不出什么信息

[fffffurina]

您好 这是完整的报错信息和脚本

Train: 0%| | 0/410 [00:00<?, ?it/s][INFO:swift] use_logits_to_keep: True
[rank0]:[E626 15:29:37.152046198 ProcessGroupNCCL.cpp:629] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=4, OpType=_ALLGATHER_BASE, NumelIn=16379, NumelOut=65516, Timeout(ms)=600000) ran for 600048 milliseconds before timing out.
[rank0]:[E626 15:29:37.152254235 ProcessGroupNCCL.cpp:2168] [PG ID 2 PG GUID 5(mesh_sequence) Rank 0] failure detected by watchdog at work sequence id: 4 PG status: last enqueued work: 4, last completed work: 3
[rank0]:[E626 15:29:37.152271886 ProcessGroupNCCL.cpp:667] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank2]:[E626 15:29:37.167661126 ProcessGroupNCCL.cpp:2168] [PG ID 2 PG GUID 5(mesh_sequence) Rank 2] failure detected by watchdog at work sequence id: 5 PG status: last enqueued work: 5, last completed work: 4
[rank2]:[E626 15:29:37.167673934 ProcessGroupNCCL.cpp:667] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank1]:[E626 15:29:37.167721025 ProcessGroupNCCL.cpp:2168] [PG ID 2 PG GUID 5(mesh_sequence) Rank 1] failure detected by watchdog at work sequence id: 4 PG status: last enqueued work: 4, last completed work: 3
[rank1]:[E626 15:29:37.167731398 ProcessGroupNCCL.cpp:667] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank0]:[E626 15:29:38.929465964 ProcessGroupNCCL.cpp:681] [Rank 0] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank0]:[E626 15:29:38.929478934 ProcessGroupNCCL.cpp:695] [Rank 0] To avoid data inconsistency, we are taking the entire process down.
[rank0]:[E626 15:29:38.931160412 ProcessGroupNCCL.cpp:1895] [PG ID 2 PG GUID 5(mesh_sequence) Rank 0] Process group watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=4, OpType=_ALLGATHER_BASE, NumelIn=16379, NumelOut=65516, Timeout(ms)=600000) ran for 600048 milliseconds before timing out.
Exception raised from checkTimeout at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:632 (most recent call first):
frame 0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fdfd0e741b6 in /anaconda3_new/envs/swift-gkd/lib/python3.11/site-packages/torch/lib/libc10.so)
frame 1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x2b4 (0x7fdfd21bdc74 in /anaconda3_new/envs/swift-gkd/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame 2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x890 (0x7fdfd21bf7d0 in /anaconda3_new/envs/swift-gkd/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame 3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7fdfd21c06ed in /anaconda3_new/envs/swift-gkd/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame 4: + 0x145c0 (0x7fe01b2435c0 in /anaconda3_new/envs/swift-gkd/lib/python3.11/site-packages/torch/lib/libtorch.so)
frame 5: + 0x7ea5 (0x7fe02531cea5 in /usr/lib64/libpthread.so.0)
frame 6: clone + 0x6d (0x7fe024734b0d in /usr/lib64/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
what(): [PG ID 2 PG GUID 5(mesh_sequence) Rank 0] Process group watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=4, OpType=_ALLGATHER_BASE, NumelIn=16379, NumelOut=65516, Timeout(ms)=600000) ran for 600048 milliseconds before timing out.
Exception raised from checkTimeout at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:632 (most recent call first):
frame 0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fdfd0e741b6 in /anaconda3_new/envs/swift-gkd/lib/python3.11/site-packages/torch/lib/libc10.so)
frame 1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x2b4 (0x7fdfd21bdc74 in /anaconda3_new/envs/swift-gkd/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame 2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x890 (0x7fdfd21bf7d0 in /anaconda3_new/envs/swift-gkd/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame 3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7fdfd21c06ed in /anaconda3_new/envs/swift-gkd/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame 4: + 0x145c0 (0x7fe01b2435c0 in /anaconda3_new/envs/swift-gkd/lib/python3.11/site-packages/torch/lib/libtorch.so)
frame 5: + 0x7ea5 (0x7fe02531cea5 in /usr/lib64/libpthread.so.0)
frame 6: clone + 0x6d (0x7fe024734b0d in /usr/lib64/libc.so.6)

Exception raised from ncclCommWatchdog at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1901 (most recent call first):
frame 0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fdfd0e741b6 in /anaconda3_new/envs/swift-gkd/lib/python3.11/site-packages/torch/lib/libc10.so)
frame 1: + 0xe5c6fc (0x7fdfd1e1b6fc in /anaconda3_new/envs/swift-gkd/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame 2: + 0x145c0 (0x7fe01b2435c0 in /anaconda3_new/envs/swift-gkd/lib/python3.11/site-packages/torch/lib/libtorch.so)
frame 3: + 0x7ea5 (0x7fe02531cea5 in /usr/lib64/libpthread.so.0)
frame 4: clone + 0x6d (0x7fe024734b0d in /usr/lib64/libc.so.6)

W0626 15:29:38.514000 190 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 256 closing signal SIGTERM
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/anaconda3_new/envs/swift-gkd/lib/python3.11/site-packages/torch/distributed/run.py", line 922, in
main()
File "/anaconda3_new/envs/swift-gkd/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 355, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/anaconda3_new/envs/swift-gkd/lib/python3.11/site-packages/torch/distributed/run.py", line 918, in main
run(args)
File "/anaconda3_new/envs/swift-gkd/lib/python3.11/site-packages/torch/distributed/run.py", line 909, in run
elastic_launch(
File "/anaconda3_new/envs/swift-gkd/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 138, in call
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/anaconda3_new/envs/swift-gkd/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

/ms-swift-gkd/swift/cli/rlhf.py FAILED

source ~/bashrc_x86_new
source activate swift-gkd
output_dir=
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True'
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
NPROC_PER_NODE=8
swift rlhf
--rlhf_type gkd
--model //pretrain_model/Qwen2.5-7B-Instruct
--teacher_model //pretrain_model/Qwen2.5-14B-Instruct
--train_type full
--dataset
--seq_kd false
--torch_dtype bfloat16
--num_train_epochs 1
--per_device_train_batch_size 2
--per_device_eval_batch_size 2
--learning_rate 2e-5
--gradient_accumulation_steps 1
--eval_steps 200
--save_steps 200
--save_total_limit 3
--logging_steps 5
--lazy_tokenize true
--columns '{"input": "query", "target": "response"}'
--packing true
--packing_cache $output_dir
--max_length 32768
--sequence_parallel_size 4
--max_completion_length 1512
--output_dir $output_dir
--warmup_ratio 0.05
--save_only_model true
--dataloader_num_workers 4
--dataset_num_proc 4
--deepspeed zero3
--attn_impl flash_attn
--gradient_accumulation_steps 2
--sft_alpha 0.1
--lmbda 0

[Jintao-Huang]

目前不支持 --sequence_parallel_size 4在gkd中。请参考examples/train/rlhf/gdk中的示例代码

[fffffurina]

不使用packing和sequence_parallel，正常max_length=9600也还是会有相同的问题

[fffffurina]

不使用packing和sequence_parallel，正常max_length=9600也还是会有相同的问题

[Jintao-Huang]

https://github.com/Jintao-Huang/llmscope/tree/main/examples/train/rlhf/gkd

这里的例子跑的通吗

[fffffurina]

找到问题了， if isinstance(teacher_model, str):
teacher_model = AutoModelForCausalLM.from_pretrained(teacher_model, **teacher_model_init_kwargs)
这个trl里面的，应该直接挪到gkd trainer里面来执行，不要到父类里执行，这样就不会模型过长等待了

[Jintao-Huang]

没懂你的意思

[fffffurina]

if isinstance(teacher_model, str):
teacher_model = AutoModelForCausalLM.from_pretrained(teacher_model, **teacher_model_init_kwargs) 这行代码原本是trl gkd trainer init中的，我把它挪到swift gkd trainer init中就不会模型等待了，正常运行，具体原因不清楚

[fffffurina]

可能是trl init 里面已经被prepare_deepspeed了，然后swift init又prepare了一遍

[Jintao-Huang]

trl中的 应该被del了

ms-swift/swift/trainers/rlhf_trainer/gkd_trainer.py

Line 19 in 76c56b0

del HFGKDTrainer.__init__

[Jintao-Huang]

你的trl版本是什么