Skip to content

cudnn deterministic not guaranteed when seed already set in imagenet #1056

Closed
@yzs981130

Description

@yzs981130

Context

  • Pytorch version: not related
  • Operating System and version: not related

Your Environment

Expected Behavior

The whole training will be deterministic when the seed is set, according to the annotations here:

warnings.warn('You have chosen to seed training. '
'This will turn on the CUDNN deterministic setting, '
'which can slow down your training considerably! '
'You may see unexpected behavior when restarting '
'from checkpoints.')

Current Behavior

The cudnn.benchmark = true here will indeed introduce nondeterminism, according to PyTorch docs

cudnn.benchmark = True

I will send a pr soon.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions