The order of the parameters is reversed. The two parameter (save_every, total_epochs)positions should be interchanged.

[RuleNHao]

examples/distributed/ddp-tutorial-series/multigpu.py

Line 102 in 35eb814

mp.spawn(main, args=(world_size, args.total_epochs, args.save_every, args.batch_size), nprocs=world_size)

[RuleNHao]

It took me hours to find the bug. This is an unforgettable experience.

[RuleNHao]

And it also need to use sampler.set_epoch() method at the beginning of each epoch.

[hudeven]

cc: @suraj813

[subramen]

@RuleNHao thanks for catching this!

set_epoch() is an important suggestion to ensure random shuffling; thank you for bringing this up, I'll update the example to include this.