LightGBMMulticlass doesn't normalize its Score column

[LittleLittleCloud]

when I try AutoML v0.16.0 multi-classification, sometimes I get prediction score <0, and the summation of all scores doesn't equal to 1. which is unexpected. Maybe some trainer's output doesn't normalize.

I publish a mini-reproducible project in this repo for facilitating debug.

[justinormont]

With 100k rows, you are going to be in TrainTest mode instead of CV.

My guess for the non-normalized Score column:

Perhaps LightGBMMulticlass changed its behavior, or we modified our wrapper for it.

In TLC (internally) LightGBMMulticlass returns a normalized Score column. In current ML.NET, LightGBMMulticlass returns the raw values of each class. This is unexpected behavior to me.

Work around for Model Builder:

I'd recommend just normalizing the Score column temporarily.

Todo:

• Investigate why it changed
  • Check if LightGBMMulticlass or our wrapper for it has changed its behavior
  • Check if the above was expected
• Ensure we normalize the score column in our wrapper for LightGBMMulticlass

/cc @codemzs, @eerhardt

[harishsk]

When I run the sample code in LightGBM.cs I do see that the scores are being normalized.

Let me take a look at your sample repro and get back to you.

[harishsk]

I downloaded your project file and tried to build it and ran into the following error:

AutoMLMultiClassificationBug-master\AutoMLMultiClassBug\obj\project.assets.json' not found. Run a NuGet package restore to generate this file

Running nuget restore isn't helping.
Are you able to build your project locally from the downloaded zip file?

[harishsk]

After messing with the nuget package sources, I was able to build and run the project. But I can't reproduce the issue.

This is my output:

Running AutoML multiclass classification experiment for 5 seconds...
Total models produced: 1
Best model's trainer: AveragedPerceptronOva
Metrics of best model from validation data --
LogLoss: 0.6980664075650147
LogLossReduction: 0.006876844013740892
MacroAccuracy: 0.22116438506449212
MicroAccuracy: 0.5572916666666666
Predicted number for test data:
0.5885952
0.4082032
0.0027440954
0.00040919252
4.829548E-05
Press any key to continue...

I see that my run produces only 1 model and that the out normalizes to 1.

Do you get the same results if you download the code from github and run?

[justinormont]

@harishsk: You'll need additional runtime. As you note, only one model was tested. With longer runtime, LightGBM will have a chance to run and be selected as the winning model.

Try changing Program.cs line 53 to from 5s to 60s:
https://github.com/LittleLittleCloud/AutoMLMultiClassificationBug/blob/c034bf95568fee8f439633fb374c906ba71365f8/AutoMLMultiClassBug/Program.cs#L53

With the longer runtime, I see LightGBM chosen as the winning model; and the predicted Scores column is not normalized:

[harishsk]

When I change the run time to 60 seconds I get 5 models, with the best model as FastTreeOva and normalized output. But if I change the run time to 30 seconds, I get the output similar to yours.
Thanks for the tip. Let me take a look and report back.

[harishsk]

@LittleLittleCloud Thank you for raising this issue.
It turned out to be a bug in how OVA model parameters were saved. I have submitted a PR for this issue.

[justinormont]

@harishsk: Wow. That had to be difficult to locate the source of the issue. Thanks.

[rauhs]

Related: #3647 ?

[harishsk]

It is very likely that this PR fixes that issue as well. Let me verify tomorrow and report back.

[eerhardt]

#4051 could also be related here as well.

[justinormont]

Thanks @harishsk for the fast fix.