支持Qwen3 MoE的Megatron LoRA训练 #4126

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open

Bond-H opened this issue May 8, 2025 · 0 comments

Bond-H commented May 8, 2025

MoE模型使用DeepSpeed训练特别慢，GPU利用率不到30%

Megatron提供的示例应该是全参数微调的，需要的资源太大了，能否提供LoRA训练示例，这边改了下2张H20都跑不起来报OOM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment