cudnn deterministic not guaranteed when seed already set in imagenet

[yzs981130]

Context

• Pytorch version: not related
• Operating System and version: not related

Your Environment

• Is it a CPU or GPU environment?: GPU
• Which example are you using: imagenet/main.py https://github.com/pytorch/examples/blob/32f0c1e1073913e75fa75b848bccdbf011b2a569/imagenet/main.py

Expected Behavior

The whole training will be deterministic when the seed is set, according to the annotations here:

examples/imagenet/main.py

Lines 91 to 95 in 32f0c1e

	warnings.warn('You have chosen to seed training. '
	'This will turn on the CUDNN deterministic setting, '
	'which can slow down your training considerably! '
	'You may see unexpected behavior when restarting '
	'from checkpoints.')

Current Behavior

The cudnn.benchmark = true here will indeed introduce nondeterminism, according to PyTorch docs

examples/imagenet/main.py

Line 207 in 32f0c1e

cudnn.benchmark = True

I will send a pr soon.