update max_model when trial fails #6596
Conversation
| @@ -296,22 +296,8 @@ void handler(object o, EventArgs e) | |||
| tuner.Update(trialResult); | |||
| trialResultManager?.AddOrUpdateTrialResult(trialResult); | |||
| aggregateTrainingStopManager.Update(trialResult); | |||
There was a problem hiding this comment.
We need to update tuner/trialresultmanager/trainningStopManager even when a trial fails for any reasons as long as the experiment still continues
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## main #6596 +/- ##
=======================================
Coverage 68.38% 68.38%
=======================================
Files 1176 1176
Lines 248210 248210
Branches 25915 25915
=======================================
+ Hits 169728 169734 +6
+ Misses 71708 71705 -3
+ Partials 6774 6771 -3
Flags with carried forward coverage won't be shown. Click here to find out more. |
| { | ||
| logger.Trace($"trial cancelled - {JsonSerializer.Serialize(trialSettings)}, continue training"); | ||
| logger.Trace($"catch exception when training trial {trialSettings.TrialId} - {JsonSerializer.Serialize(trialSettings)}, continue training"); |
There was a problem hiding this comment.
Exception thrown during Trial {trialSettings.TrialId} with configuration {JsonSerializer.Serialize(trialSettings)}
Exception Details: ex.Message
Abandoning Trial {trialSettings.TrialId} and continue training.
There was a problem hiding this comment.
@LittleLittleCloud is there a way to localization for these strings? Do we do it anywhere else in this repo?
There was a problem hiding this comment.
@luisquintanilla thoughts on strings here?
There was a problem hiding this comment.
@JakeRadMSFT I don't think strings are localized anywhere in mlnet, aren't they? @michaelgsharp
ghost
locked as resolved and limited conversation to collaborators
We are excited to review your PR.
So we can do the best job, please check:
Fixes #nnnnin your description to cause GitHub to automatically close the issue(s) when your PR is merged.also fix #6604