Enhancements to EarlyStopping handler

[bonassifabio]

🚀 Feature

I would propose several enhancements to the EarlyStopping handler.

1. Introduce a new parameter, min_delta_mode, mimicking the parameter threshold_mode of ReduceLROnPlateau. If min_delta_mode="abs" (default) everything works as of now, while if min_delta_mode="rel" the score comparison becomes score >= best_score (1 + min_delta).
2. Introduce a new parameter mode that work's as in ReduceLROnPlateau, allowing to handle validation metrics to be minimized.

Other ideas I'd like to discuss are:

• Natively implementing min_epochs/min_evals, preventing EarlyStopping from terminating until it has been evaluated at least X times. I know this could be addressed as in How to set early stopping not enabled during the first n epochs #3317 but this solution might be more compact?
• It would be nice to have the same threshold and threshold_mode nomenclature as ReduceLROnPlateau, but I realize this would probably break older code.
• I personally find the parameter name cumulative_delta quite counter-intuitive. When I first read about that parameter, I understood it was taking the cumulative sum of the improvements and comparing this cumulative sum to min_delta.

[vfdev-5]

Thanks for suggestions, @bonassifabio !

I personally find the parameter name cumulative_delta quite counter-intuitive. When I first read about that parameter, I understood it was taking the cumulative sum of the improvements and comparing this cumulative sum to min_delta.

We can start deprecating it in favor of another argument.

Introduce a new parameter, min_delta_mode, mimicking the parameter threshold_mode of ReduceLROnPlateau. If min_delta_mode="abs" (default) everything works as of now, while if min_delta_mode="rel" the score comparison becomes score >= best_score (1 + min_delta).

Sounds good!

Introduce a new parameter mode that work's as in ReduceLROnPlateau, allowing to handle validation metrics to be minimized.

How about doing a similar thing as in Checkpoint.get_default_score_fn ?

Natively implementing min_epochs/min_evals, preventing EarlyStopping from terminating until it has been evaluated at least X times. I know this could be addressed as in #3317 but this solution might be more compact?

Maybe, we can provide a helper function for that?

It would be nice to have the same threshold and threshold_mode nomenclature as ReduceLROnPlateau, but I realize this would probably break older code.

Do you propose to rename min_delta and min_delta_mode into threshold and threshold_mode ?

[bonassifabio]

Thank you for your quick response!

We can start deprecating it in favor of another argument.

Sounds good to me. Let me think if I can find a better argument name.

How about doing a similar thing as in Checkpoint.get_default_score_fn ?

Also this solution sounds good to me. Actually, I just now realized I'm defining my own score_function but I didn't think to embed the sign there! 🙂

Maybe, we can provide a helper function for that?

At that point would it similar to using a filter? Or do you mean something else?

Do you propose to rename min_delta and min_delta_mode into threshold and threshold_mode ?

Yes, ideally that would be my proposal. Do you think this transition makes sense?

[vfdev-5]

At that point would it similar to using a filter? Or do you mean something else?

I was thinking about providing a prebaked events filtering helper function such user does not need to code it (maybe something like Checkpoint.get_default_score_fn):

es_handler = EarlyStopping(
    patience=3, 
    score_function=score_function, 
    trainer=trainer
)

evaluator.add_event_handler(
  # Function name `get_training_epoch_filtering_fn` can be changed
  Events.COMPLETED(event_filter=es_handler.get_training_epoch_filtering_fn(after=5)), 
  es_handler,
)

Yes, ideally that would be my proposal. Do you think this transition makes sense?

I agree that cumulative_delta is not clear as name. We can align to PyTorch if this is consistently used in their API.

[bonassifabio]

Alright, then I'll come back with a PR in the near future.

I agree that cumulative_delta is not clear as name. We can align to PyTorch if this is consistently used in their API.

Might something like update_below_threshold=True work better?

[github-actions]

Hey 👋, I've just created a thread for this issue on PyTorch-Ignite Discord where you can quickly talk to the community on the topic.

^{🤖 This comment was automatically posted by Discuss on Discord}