Hanging after tqdm starts [COLOCATE MODE]

[Armaanm11]

Describe the bug
@hjh0119 ms-swift hangs after tqdm shows up in colocate mode:

WANDB_API_KEY= \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
NPROC_PER_NODE=8  \
swift rlhf \
    --rlhf_type grpo \
    --model ./DeepSeek-R1-Distill-Qwen-14B \
    --reward_funcs cosine  \
    --cosine_max_len 12000 \
    --cosine_max_len_value_correct 0.5 \
    --cosine_min_len_value_correct 0.75 \
    --cosine_max_len_value_wrong 0 \
    --cosine_min_len_value_wrong -0.25 \
    --train_type lora \
    --use_vllm true \
    --lora_rank 64 \
    --lora_alpha 128 \
    --use_rslora true \
    --use_dora true \
    --num_infer_workers 8 \
    --target_modules all-linear \
    --torch_dtype bfloat16 \
    --dataset 'math_data.jsonl' \
    --max_completion_length 12000 \
    --num_train_epochs 1 \
    --per_device_train_batch_size 2 \
    --per_device_eval_batch_size 2 \
    --learning_rate 1e-5 \
    --gradient_accumulation_steps 4 \
    --eval_steps 50 \
    --save_steps 10 \
    --save_total_limit 20 \
    --logging_steps 1 \
    --report_to wandb \
    --max_length 16000 \
    --sleep_level 1 \
    --vllm_gpu_memory_utilization 0.6 \
    --tensor_parallel_size 8 \
    --output_dir 14b_trained \
    --warmup_ratio 0.05 \
    --num_generations 8 \
    --temperature 1 \
    --top_p 0.95 \
    --top_k 40 \
    --repetition_penalty 1.05 \
    --deepspeed zero3 \
    --log_completions true \
    --num_iterations 1 \
    --async_generate false \

Your hardware and system info
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Wed_Jan_15_19:20:09_PST_2025
Cuda compilation tools, release 12.8, V12.8.61
Build cuda_12.8.r12.8/compiler.35404655_0

Package Version Editable project location

---

absl-py 2.2.1
accelerate 1.5.2
addict 2.4.0
aiofiles 23.2.1
aiohappyeyeballs 2.6.1
aiohttp 3.11.14
aiohttp-cors 0.8.0
aiosignal 1.3.2
airportsdata 20250224
aliyun-python-sdk-core 2.16.0
aliyun-python-sdk-kms 2.16.5
altair 5.5.0
annotated-types 0.7.0
antlr4-python3-runtime 4.13.2
anyio 4.9.0
argon2-cffi 23.1.0
argon2-cffi-bindings 21.2.0
arrow 1.3.0
arxiv 2.1.3
astor 0.8.1
asttokens 3.0.0
async-lru 2.0.5
async-timeout 5.0.1
attrdict 2.0.1
attrs 25.3.0
audioread 3.0.1
babel 2.17.0
beautifulsoup4 4.13.3
binpacking 1.5.2
bitsandbytes 0.45.4
blake3 1.0.4
bleach 6.2.0
blinker 1.9.0
boto3 1.37.21
botocore 1.37.21
Brotli 1.1.0
cachetools 5.5.2
certifi 2025.1.31
cffi 1.17.1
cfgv 3.4.0
charset-normalizer 3.4.1
click 8.1.8
cloudpickle 3.1.1
colorama 0.4.6
colorful 0.5.6
comm 0.2.2
compressed-tensors 0.9.1
contourpy 1.3.1
cpm-kernels 1.0.11
crcmod 1.7
cryptography 44.0.2
cupy-cuda12x 13.4.1
cycler 0.12.1
dacite 1.9.2
datasets 3.2.0
debugpy 1.8.13
decorator 5.2.1
decord 0.6.0
deepspeed 0.16.4
defusedxml 0.7.1
depyf 0.18.0
dill 0.3.8
diskcache 5.6.3
distlib 0.3.9
distro 1.9.0
dnspython 2.7.0
docker-pycreds 0.4.0
duckduckgo_search 5.3.1b1
editdistance 0.8.1
einops 0.8.1
email_validator 2.2.0
et_xmlfile 2.0.0
evalscope 0.13.1
evaluate 0.4.3
exceptiongroup 1.2.2
executing 2.2.0
expecttest 0.3.0
fastapi 0.115.12
fastapi-cli 0.0.7
fastjsonschema 2.21.1
fastrlock 0.8.3
feedparser 6.0.11
ffmpy 0.5.0
filelock 3.18.0
fire 0.7.0
flake8 7.1.2
fonttools 4.56.0
fqdn 1.5.1
frozenlist 1.5.0
fsspec 2024.9.0
func-timeout 4.3.5
future 1.0.0
fuzzywuzzy 0.18.0
gguf 0.10.0
gitdb 4.0.12
GitPython 3.1.44
google-api-core 2.24.2
google-auth 2.38.0
googleapis-common-protos 1.69.2
gradio 5.23.1
gradio_client 1.8.0
griffe 0.49.0
groovy 0.1.2
grpcio 1.71.0
h11 0.14.0
h2 4.2.0
h5py 3.13.0
hjson 3.1.0
hpack 4.1.0
httpcore 1.0.7
httptools 0.6.4
httpx 0.28.1
huggingface-hub 0.29.3
human-eval 1.0.3
hyperframe 6.1.0
icecream 2.1.4
identify 2.6.9
idna 3.10
imageio 2.37.0
immutabledict 4.2.1
importlib_metadata 8.6.1
iniconfig 2.1.0
interegular 0.3.3
ipykernel 6.29.5
ipython 8.34.0
ipywidgets 8.1.5
isoduration 20.11.0
isort 6.0.1
jedi 0.19.2
jieba 0.42.1
Jinja2 3.1.6
jiter 0.9.0
jmespath 0.10.0
joblib 1.4.2
json5 0.10.0
jsonlines 4.0.0
jsonpointer 3.0.0
jsonschema 4.23.0
jsonschema-specifications 2024.10.1
jupyter 1.1.1
jupyter_client 8.6.3
jupyter-console 6.6.3
jupyter_core 5.7.2
jupyter-events 0.12.0
jupyter-lsp 2.2.5
jupyter_server 2.15.0
jupyter_server_terminals 0.5.3
jupyterlab 4.3.6
jupyterlab_pygments 0.3.0
jupyterlab_server 2.27.3
jupyterlab_widgets 3.0.13
kiwisolver 1.4.8
lagent 0.2.4
langdetect 1.0.9
lark 1.2.2
latex2sympy2 1.9.1
latex2sympy2_extended 1.10.1
lazy_loader 0.4
Levenshtein 0.27.1
librosa 0.11.0
liger_kernel 0.5.5
llvmlite 0.43.0
lm-format-enforcer 0.10.11
lmdeploy 0.7.2.post1
lora_ga 0.1
lxml 5.3.1
Markdown 3.7
markdown-it-py 3.0.0
MarkupSafe 3.0.2
math-verify 0.7.0
matplotlib 3.10.1
matplotlib-inline 0.1.7
mccabe 0.7.0
mdurl 0.1.2
mistral_common 1.5.4
mistune 3.1.3
mmengine 0.10.7
mmengine-lite 0.10.7
modelscope 1.24.0
mpmath 1.3.0
ms-opencompass 0.1.6
ms_swift 3.3.0.dev0 /home/user1/GRPO/ms-swift
ms-vlmeval 0.0.14
msgpack 1.1.0
msgspec 0.19.0
multidict 6.2.0
multiprocess 0.70.16
nanobind 2.5.0
narwhals 1.32.0
nbclient 0.10.2
nbconvert 7.16.6
nbformat 5.10.4
nest-asyncio 1.6.0
networkx 3.4.2
ninja 1.11.1.4
nltk 3.9.1
nodeenv 1.9.1
notebook 7.3.3
notebook_shim 0.2.4
numba 0.60.0
numpy 1.26.4
nvidia-cublas-cu12 12.4.5.8
nvidia-cuda-cupti-cu12 12.4.127
nvidia-cuda-nvrtc-cu12 12.4.127
nvidia-cuda-runtime-cu12 12.4.127
nvidia-cudnn-cu12 9.1.0.70
nvidia-cufft-cu12 11.2.1.3
nvidia-curand-cu12 10.3.5.147
nvidia-cusolver-cu12 11.6.1.9
nvidia-cusparse-cu12 12.3.1.170
nvidia-cusparselt-cu12 0.6.2
nvidia-ml-py 12.570.86
nvidia-nccl-cu12 2.21.5
nvidia-nvjitlink-cu12 12.4.127
nvidia-nvtx-cu12 12.4.127
omegaconf 2.0.6
openai 1.68.2
OpenCC 1.1.9
opencensus 0.11.4
opencensus-context 0.1.3
opencv-python 4.11.0.86
opencv-python-headless 4.11.0.86
openpyxl 3.1.5
optimum 1.24.0
orjson 3.10.16
oss2 2.19.1
outlines 0.1.11
outlines_core 0.1.26
overrides 7.7.0
packaging 24.2
pandas 2.2.3
pandocfilters 1.5.1
parso 0.8.4
partial-json-parser 0.2.1.1.post5
peft 0.15.0
pexpect 4.9.0
phx-class-registry 4.1.0
pillow 11.1.0
pip 22.0.2
platformdirs 4.3.7
plotly 6.0.1
pluggy 1.5.0
ply 3.11
pooch 1.8.2
portalocker 3.1.1
pre_commit 4.2.0
prettytable 3.16.0
prometheus_client 0.21.1
prometheus-fastapi-instrumentator 7.1.0
prompt_toolkit 3.0.50
propcache 0.3.1
proto-plus 1.26.1
protobuf 5.29.4
psutil 7.0.0
ptyprocess 0.7.0
pure_eval 0.2.3
py-cpuinfo 9.0.0
py-spy 0.4.0
pyarrow 19.0.1
pyasn1 0.6.1
pyasn1_modules 0.4.1
pyav 14.2.1
pybind11 2.13.6
pycodestyle 2.12.1
pycountry 24.6.1
pycparser 2.22
pycryptodome 3.22.0
pydantic 2.10.6
pydantic_core 2.27.2
pydeck 0.9.1
pydub 0.25.1
pyflakes 3.2.0
Pygments 2.19.1
Pympler 1.1
pynvml 12.0.0
pyparsing 3.2.3
pypinyin 0.53.0
pytest 8.3.5
python-dateutil 2.9.0.post0
python-dotenv 1.1.0
python-json-logger 3.3.0
python-Levenshtein 0.27.1
python-multipart 0.0.20
pytz 2025.2
PyYAML 6.0.2
pyzmq 26.3.0
rank-bm25 0.2.2
RapidFuzz 3.12.2
ray 2.40.0
referencing 0.36.2
regex 2024.11.6
requests 2.32.3
requests-toolbelt 1.0.0
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rich 13.9.4
rich-toolkit 0.14.0
rouge 1.0.1
rouge-chinese 1.0.3
rouge_score 0.1.2
rpds-py 0.24.0
rsa 4.9
ruff 0.11.2
s3transfer 0.11.4
sacrebleu 2.5.1
safehttpx 0.1.6
safetensors 0.5.3
scikit-image 0.25.2
scikit-learn 1.6.1
scipy 1.15.2
seaborn 0.13.2
semantic-version 2.10.0
Send2Trash 1.8.3
sentence-transformers 4.0.1
sentencepiece 0.2.0
sentry-sdk 2.24.1
setproctitle 1.3.5
setuptools 69.5.1
sgmllib3k 1.0.0
shellingham 1.5.4
shortuuid 1.0.13
simple-ddl-parser 1.7.1
simplejson 3.20.1
six 1.17.0
smart-open 7.1.0
smmap 5.0.2
sniffio 1.3.1
socksio 1.0.0
sortedcontainers 2.4.0
soundfile 0.13.1
soupsieve 2.6
soxr 0.5.0.post1
stack-data 0.6.3
starlette 0.46.1
streamlit 1.44.0
sty 1.0.6
swankit 0.1.6
swanlab 0.5.3
sympy 1.13.1
tabulate 0.9.0
tenacity 9.0.0
tensorboard 2.19.0
tensorboard-data-server 0.7.2
termcolor 2.5.0
terminado 0.18.1
threadpoolctl 3.6.0
tifffile 2025.3.13
tiktoken 0.9.0
timeout-decorator 0.5.0
timm 1.0.15
tinycss2 1.4.0
tokenizers 0.21.1
toml 0.10.2
tomli 2.2.1
tomlkit 0.13.2
torch 2.5.1
torchaudio 2.5.1
torchvision 0.20.1
tornado 6.4.2
tqdm 4.67.1
traitlets 5.14.3
transformers 4.50.1
transformers-stream-generator 0.0.5
triton 3.1.0
trl 0.16.0
typer 0.15.2
types-python-dateutil 2.9.0.20241206
typing_extensions 4.13.0
tzdata 2025.2
uri-template 1.3.0
urllib3 2.3.0
uvicorn 0.34.0
uvloop 0.21.0
validators 0.34.0
virtualenv 20.29.3
vllm 0.7.3
wandb 0.19.8
watchdog 6.0.0
watchfiles 1.0.4
wcwidth 0.2.13
webcolors 24.11.1
webencodings 0.5.1
websocket-client 1.8.0
websockets 15.0.1
Werkzeug 3.1.3
widgetsnbextension 4.0.13
word2number 1.1
wrapt 1.17.2
xformers 0.0.28.post3
xgrammar 0.1.11
XlsxWriter 3.2.2
xtuner 0.1.23
xxhash 3.5.0
yapf 0.30.0
yarl 1.18.3
zipp 3.21.0
zstandard 0.23.0

[tejeshbhalla]

same happens to me any fix ?

[hjh0119]

try

1. vllm 0.7.3
2. Add the parameters --offload_optimizer true \ --offload_model true \ --gc_collect_after_offload true

If it still doesn't work, please provide your running command/error stack.

[tejeshbhalla]

hi @hjh0119 for me the inference works my process hangs on that once i return the rewards it just doesnt go ahead
Here's the command i ran

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 NPROC_PER_NODE=7 swift rlhf --rlhf_type grpo --model /root/highspeedstorage/ft-volume/mistral-agentic-turbo --train_type lora --dataset ms-swift/all_processed_logs.jsonl --torch_dtype bfloat16 --num_train_epochs 1 --max_length 80000 --per_device_train_batch_size 1 --per_device_eval_batch_size 1 --gradient_accumulation_steps 1 --eval_steps 1000 --save_steps 100 --learning_rate 1e-6 --save_total_limit 2 --logging_steps 100 --output_dir output --warmup_ratio 0.05 --dataloader_num_workers 4 --max_completion_length 32000 --reward_funcs accuracy format repetition --num_generations 7 --system "" --use_vllm true --vllm_gpu_memory_utilization 0.95 --vllm_max_model_len 80000 --deepspeed zero3_offload --temperature 1 --top_p 1.0 --top_k 80 --log_completions true --num_infer_workers 1 --async_generate true --move_model_batches 7 --offload_optimizer true --offload_model true --gc_collect_after_offload true --lora_rank 128 --lora_alpha 256 --model_type mistral_2501 --rope_scaling dynamic

Also i don't see any errors particularly it just hangs even with nccl debugging turned on all i see is some CACHE warning thats it

It seems to be deepspeed causing it though any ideas?

[tejeshbhalla]

Nvm i made it work degrading to vllm prob sorted other pacakages out

[Armaanm11]

Same issue. No error message and it just hangs

WANDB_API_KEY= \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
NPROC_PER_NODE=8  \
swift rlhf \
    --rlhf_type grpo \
    --model ./DeepSeek-R1-Distill-Qwen-14B \
    --reward_funcs cosine  \
    --cosine_max_len 13000 \
    --cosine_max_len_value_correct 0.5 \
    --cosine_min_len_value_correct 0.9 \
    --cosine_max_len_value_wrong 0 \
    --cosine_min_len_value_wrong -0.1 \
    --train_type lora \
    --use_vllm true \
    --lora_rank 64 \
    --lora_alpha 128 \
    --num_infer_workers 8 \
    --target_modules all-linear \
    --torch_dtype bfloat16 \
    --dataset 'math_data.jsonl' \
    --max_completion_length 18000 \
    --num_train_epochs 2 \
    --per_device_train_batch_size 2 \
    --per_device_eval_batch_size 2 \
    --learning_rate 5e-5 \
    --gradient_accumulation_steps 2 \
    --eval_steps 25 \
    --save_steps 5 \
    --save_total_limit 20 \
    --logging_steps 1 \
    --report_to wandb \
    --max_length 18000 \
    --vllm_gpu_memory_utilization 0.8 \
    --tensor_parallel_size 8 \
    --output_dir 14b_trained \
    --warmup_ratio 0.1 \
    --num_generations 16 \
    --temperature 1 \
    --top_p 0.95 \
    --top_k 50 \
    --repetition_penalty 1.05 \
    --deepspeed zero3_offload \
    --sleep_level 1 \
    --log_completions true \
    --num_iterations 1 \
    --async_generate false \
    --dataloader_num_workers 1 \
    --dataset_num_proc 1 \
    --move_model_batches 16 \
    --offload_optimizer true \
    --offload_model true \
    --gc_collect_after_offload true