When using gradient accumulation, optimizer.step() runs every single step. Is it as expected？

[tyxsspa]

Describe the bug

Should these three lines

                optimizer.step()
                lr_scheduler.step()
                optimizer.zero_grad()

be indented to fall within the scope of if accelerator.sync_gradients:? When gradient accumulation is enabled, gradient updates should only occur within the if block. Could this be a bug?
https://github.com/huggingface/diffusers/blob/main/examples/flux-control/train_control_lora_flux.py#L1299-L1301

Reproduction

None

Logs

System Info

None

Who can help?

No response

[a-r-r-o-w]

This is not a bug as accelerate only updates the parameters when accelerator.sync_gradients is True. The flag is only True when accelerate has accumulated gradients the configured number of times. Even if we put the if-conditional ourself, it would result in the same behaviour.

See https://github.com/huggingface/accelerate/blob/8039158d71418e4520113e43bfe3567ffeedd7db/src/accelerate/optimizer.py#L161

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[a-r-r-o-w]

Marking as closed because this is resolved given above explanation.