Yuxiang Ji^1,2, Ziyu Ma², Yong Wang^2†, Guanhua Chen³, Xiangxiang Chu², Liaoni Wu^1
1Xiamen University    ²AMAP, Alibaba Group    ³Southern University of Science and Technology
^†Project leader.    

• [Sep 25, 2025]: Codebase released. (work in progress)

• Overview
• Quick start
• Preliminary results
• Acknowledgement
• Citation

We propose Tree-GRPO, adopting a tree-search rollout strategy in place of independent chain-based rollouts for LLM agent RL. Based on ReAct step-level nodes, Tree-GRPO carries out rollout sampling over a semantically well-defined search tree. The Tree-GRPO brings two major advantages: (i) Less rollout budget, (ii) Tree-based process supervion signal.

The overview of Tree-GRPO training pipeline.

Based on Qwen2.5-3b, our Tree-GRPO achieves superior performance over the chain-based method while using only a quarter of the rollout budget.

conda create -n treegrpo python=3.12.9
conda activate treegrpo
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0  
pip install vllm==0.8.5.post1
# verl
pip install -e .
# flash attention 2
pip install flash-attn --no-build-isolation
pip install swanlab

conda create -n retriever python=3.10.13
conda activate retriever
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0  
pip install transformers datasets pyserini
## install the gpu version faiss to guarantee efficient RL rollout
pip install faiss-gpu==1.7.3
## API function
pip install uvicorn fastapi

Download the wiki dump indexing and corpus.

save_path=/the/path/to/save
python scripts/download.py --save_path $save_path
cat $save_path/part_* > $save_path/e5_Flat.index
gzip -d $save_path/wiki-18.jsonl.gz

Process the single-hop QA and multi-hop QA datasets into parquet.

bash scripts/data_process/data_process_multihop.sh
bash scripts/data_process/data_process_single.sh

The training and test set of web-agent QA will be released soon.

Launch a local retrieval server for single-hop QA and multi-hop QA,

conda activate retriever
bash retrieval_launch.sh

or launch a real search API (bing) server for web-agent QA.

# Change your own api in bing_search_launch.sh
# Note that the bing search server will cache the search results by redis, then stores them permanently as .json periodically and when the server terminates
bash bing_search_launch.sh

Run GRPO training with Qwen2.5-3b on multi-hop QA setting.

conda activate treegrpo
bash train_multihopqa_grpo.sh

Run Tree-GRPO training with Qwen2.5-3b on multi-hop QA setting.

bash train_multihopqa_tree_search.sh

The training logs are tracking in Swanlab.

The codebase is built upon Search-R1 and veRL. The implementation is inspired by TreeRL. We express our gratitude to these open-source projects.

@article{ji2025treesearch,
  title={Tree Search for LLM Agent Reinforcement Learning}, 
  author={Yuxiang Ji and Ziyu Ma and Yong Wang and Guanhua Chen and Xiangxiang Chu and Liaoni Wu},
  journal={arXiv preprint arXiv:2509.21240},
  year={2025}
}