Skip to content

Qwen3-8B-Base SFT 全参微调保存第一个模型后hang住 #4053

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
leileilin opened this issue Apr 30, 2025 · 3 comments
Closed

Qwen3-8B-Base SFT 全参微调保存第一个模型后hang住 #4053

leileilin opened this issue Apr 30, 2025 · 3 comments

Comments

@leileilin
Copy link

Describe the bug
What the bug is, and how to reproduce, better with screenshots(描述bug以及复现过程,最好有截图)

Qwen3-8B-Base SFT 全参微调保存第一个模型后hang住,而且查看模型保存目录有两个保存模型目录文件。

执行命令如下:
deepspeed --hostfile=/etc/mpi/hostfile
swift/cli/sft.py
--model $PRETRAIN_MODEL
--torch_dtype bfloat16
--train_type full
--use_chat_template
--dataset $data_path
--packing True
--num_train_epochs 3
--per_device_train_batch_size $per_node_bsz
--data_seed 42
--weight_decay 0.1
--learning_rate 1e-5
--attn_impl flash_attn
--deepspeed zero1
--gradient_accumulation_steps $gradient_accumulation_steps
--warmup_ratio 0.01
--dataset_num_proc 8
--system "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."
--save_total_limit 5
--save_strategy epoch
--max_length 8192
--truncation_strategy delete
--split_dataset_ratio 0
--eval_strategy no
--output_dir $output_dir
--use_liger_kernel true
--lazy_tokenize true
--use_hf \

bug截图如下:

Image

Your hardware and system info
Write your system info like CUDA version/system/GPU/torch version here(在这里给出硬件信息和系统信息,如CUDA版本,系统,GPU型号和torch版本等)

GPU型号:H800
CUDA版本:12.4
torch版本:2.6.0
系统版本:
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Additional context
Add any other context about the problem here(在这里补充其他信息)

@Jintao-Huang
Copy link
Collaborator

这里应该是在保存模型,只是存的比较慢

@Jintao-Huang
Copy link
Collaborator

可以试试 --save_only_model true

@leileilin
Copy link
Author

这里应该是在保存模型,只是存的比较慢

好的,后面好像又没有这个问题了,非常神奇,但是当时确实慢的离谱。。。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants