Added cross entropy support to validation training, edited metric reporting

[mstfbl]

Fix #4807

In this PR, I've added cross entropy metric calculation & reporting during validation training in the Image Classification Trainer. Now, validation training can obtain cross entropy values for each epoch during validation training.

I have attached below screenshots of training and validation metrics with this repro.

Before (note that learning rate is constant, even when the given ExponentialLRDecay is used, and cross entropy during validation is reported as 0):

After:

I have also edited ToString() to report accuracy and cross entropy for the validation phase as well. Reporting learning rate decay during validation isn't necessary, as no learning rate decay occurs in validation.

I have also added tests to verify the proper decaying of the cross-entropy rate with this PR.

[codecov]

Codecov Report

Merging #5255 into master will increase coverage by 0.24%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #5255      +/-   ##
==========================================
+ Coverage   73.49%   73.74%   +0.24%     
==========================================
  Files        1014     1022       +8     
  Lines      188680   190258    +1578     
  Branches    20330    20468     +138     
==========================================
+ Hits       138677   140308    +1631     
+ Misses      44493    44429      -64     
- Partials     5510     5521      +11     

Flag	Coverage Δ
#Debug	73.74% <100.00%> (+0.24%)	⬆️
#production	69.50% <100.00%> (+0.20%)	⬆️
#test	87.64% <100.00%> (+0.21%)	⬆️

Impacted Files	Coverage Δ
.../Microsoft.ML.Vision/ImageClassificationTrainer.cs	91.13% <100.00%> (+0.04%)	⬆️
...cenariosWithDirectInstantiation/TensorflowTests.cs	91.78% <100.00%> (+0.29%)	⬆️
src/Microsoft.ML.FastTree/RegressionTree.cs	75.51% <0.00%> (-8.17%)	⬇️
src/Microsoft.ML.LightGbm/LightGbmTrainerBase.cs	78.92% <0.00%> (-6.07%)	⬇️
src/Microsoft.ML.AutoML/API/AutoCatalog.cs	69.35% <0.00%> (-4.84%)	⬇️
....ML.AutoML/PipelineSuggesters/PipelineSuggester.cs	79.83% <0.00%> (-3.37%)	⬇️
src/Microsoft.ML.TimeSeries/ExtensionsCatalog.cs	90.62% <0.00%> (-3.00%)	⬇️
...rosoft.ML.AutoML/ColumnInference/TextFileSample.cs	59.60% <0.00%> (-2.65%)	⬇️
src/Microsoft.ML.Maml/MAML.cs	23.78% <0.00%> (-2.43%)	⬇️
...osoft.ML.Functional.Tests/IntrospectiveTraining.cs	98.73% <0.00%> (-1.27%)	⬇️
... and 50 more

[wangyems]

The print of cross-entropy looks good to me!
I'm a little skeptical that whether we train on validation set after searching this and quote "it is training data used for testing, but neither as part of the low-level training nor as part of the final testing". It seems like this case may fall into the category of low-level training: the validation set is fixed during the training(I might be wrong because I merely draw this conclusion by looking at the screenshot that the accuracy of validation set is often higher than training set's accuracy in the second screenshot). If the validation set is not fixed(something like cross-validation) then just ignore this comment.

[mstfbl]

Hey Ye,
Thank you so much for your review. After checking in with @harshithapv, we've arrived at the concolusion that the validation stage does not require learning rate scheduler support, as validation is only for computing loss and analyzing how the training phase is going. As there is no actual model training during the validation phase, there is also no need for decaying learning rates during validation.
I'll be making the necessary changes soon! #Resolved

[mstfbl]

I have added learning rate to the end of the string here for cosmetic reasons. Now there is uniform alignment of columns' statistics between training and validation. :) #Resolved

[mstfbl]

@wangyems I have removed learning rate decay for validation from my PR, and now the accuracy statistics for before and after are the same. Now cross entropy is being reported for validation, and as it should be, there is no change in learning rates during the validation phase,

[wangyems]

lgtm