train with singel GPU

[silk-juan]

That was a wonderful work, I have seen you use 4 V100 GPUs for the experiments.
May i ask if it possible to train with single GPU, and what should i change in code.
Looking forward to your reply.
Thanks

[zjh-lgtm]

Hello, have you managed to resolve it yet?

[byminji]

Hi, I haven't checked the reproducibility with 1 GPU training, but you would need to change the $accumulation_steps to ensure that $total_batch_size = $num_gpus * $batch_size * $accumulations_steps.

For example, in a zero-shot setting, our default setting was $total_batch_size (256) = $num_gpus (4) x $batch_size (4) x $accumulations_steps (16). Suppose that you're using 1gpu while keeping the batch size per GPU. Then you need to increase the accumulation steps as $total_batch_size (256) = $num_gpus (1) * $batch_size (4) * $accumulations_steps (64).

# Original command:
torchrun --nproc_per_node=4 main.py -cn zero_shot \
data=zero_shot_k400 output=ckpt/zero_shot_k400_tc_clip trainer=tc_clip

# 1GPU case:
torchrun --nproc_per_node=1 main.py -cn zero_shot \
data=zero_shot_k400 output=ckpt/zero_shot_k400_tc_clip trainer=tc_clip accumulation_steps=64