LLM-assisted Diffusion Model for Procedural Weather Generation in Domain-Generalized Semantic Segmentation

WeatherDG is a framework that can generate realistic and diverse autonomous driving scene images and improve semantic segmentation performance under adverse conditions such as snow, rain, fog, and low-light environments.

⭐ If you like our work or find it useful, please give us a star or cite below. Thanks!

• Collaborations of Foundation Model: Propose a novel data augmentation framework based on SD and LLM for domain generalization in adverse weather conditions.
• LLM-Agents Utilize collaborations of LLM agents for prompt generation to encourage SD to generate realistic driving-screen samples under adverse weather conditons.
• Sampling strategy Propose a probabilistic sampling strategy for enriching underrepresented objects in adverse weather conditions.

• Python ≥ 3.8
• PyTorch ≥ 1.10 and torchvision that matches the PYTorch installation. Follow official instruction
• HuggingFace installations: diffusers, transformers, safetensors
• pip install --user -U nltk

1. Download the Pretrained Model:
   Download and unzip the pretrained model from Google Drive or Tencent Cloud, and change the --sd_path in scripts/gen_data_weather.sh to be the model path.

2. Run the Script: Execute the script using the following command:

   sh scripts/gen_data_weather.sh
   

3. (Alternative) Download generated datasets from this link

You can use the generated dataset for domain adaptive semantic segmentation training. For more details, please refer to MIC and DAFormer

@ARTICLE{10960638,
  author={Qian, Chenghao and Guo, Yuhu and Mo, Yuhong and Li, Wenjing},
  journal={IEEE Robotics and Automation Letters}, 
  title={WeatherDG: LLM-Assisted Procedural Weather Generation for Domain-Generalized Semantic Segmentation}, 
  year={2025},
  volume={10},
  number={6},
  pages={5919-5926},
  keywords={Meteorology;Training;Diffusion models;Autonomous vehicles;Semantic segmentation;Lighting;Data models;Adaptation models;Layout;Data augmentation;Domain generalization;LLM;semantic segmentation;weather generation},
  doi={10.1109/LRA.2025.3559821}}