LightGBM trainer filters out rows with NaN features

[rauhs]

LightGBM (binary trainer) will filter out NaN (feature)-values even though we have the option of HandleMissingValue which allows LightGBM to properly deal with missing values:

https://lightgbm.readthedocs.io/en/latest/Advanced-Topics.html

machinelearning/src/Microsoft.ML.LightGbm/LightGbmTrainerBase.cs

Line 178 in 7a4372e

[TlcModule.SweepableDiscreteParam("UseMissing", new object[] { true, false })]

I think this:

machinelearning/src/Microsoft.ML.LightGbm/LightGbmTrainerBase.cs

Line 439 in 7a4372e

var loadFlags = CursOpt.AllLabels | CursOpt.Features;

should also specify the flag "AllFeatures" so they're allowed through.

[yaeldekel]

Hi @rauhs , good suggestion, thanks!

[mstfbl]

Hi @rauhs , it is true that LightGBM filters NaN values by default. We have HandleMissingValue so that one can disable this by setting HandleMissingValue=false.
In addition, there is also the zero_as_missing flag in LightGBM, which allows the missing value to be representated by the numerical value 0. I have made a PR to address your suggestion and to add support for the zero_as_missing flag in LightGBM.

[mstfbl]

Update: When the requested change from Features to AllFeatures is made here: ...

machinelearning/src/Microsoft.ML.LightGbm/LightGbmTrainerBase.cs

Line 439 in 7a4372e

var loadFlags = CursOpt.AllLabels | CursOpt.Features;

,

3 binary classification LightGbm tests fail: LightGBMClassificationTest, GossLightGBMTest, DartLightGBMTest. They use breast-cancer.txt as test and train file, as shown here:

machinelearning/test/Microsoft.ML.Predictor.Tests/TestPredictors.cs

Lines 476 to 508 in bc7d86e

	public void LightGBMClassificationTest()
	{
	var learners = new[] { TestLearners.LightGBMClassifier };
	var binaryClassificationDatasets = new List<TestDataset> { TestDatasets.breastCancerPipe };
	foreach (var learner in learners)
	{
	foreach (TestDataset dataset in binaryClassificationDatasets)
	Run_TrainTest(learner, dataset);
	}
	Done();
	}
	[LightGBMFact]
	[TestCategory("Binary"), TestCategory("LightGBM")]
	public void GossLightGBMTest()
	{
	var binaryPredictors = new[] { TestLearners.LightGBMGoss };
	var binaryClassificationDatasets = new List<TestDataset> { TestDatasets.breastCancerPipe };
	RunAllTests(binaryPredictors, binaryClassificationDatasets, extraTag: "goss");
	Done();
	}
	[LightGBMFact]
	[TestCategory("Binary")]
	[TestCategory("LightGBM")]
	public void DartLightGBMTest()
	{
	var binaryPredictors = new[] { TestLearners.LightGBMDart };
	var binaryClassificationDatasets = new List<TestDataset> { TestDatasets.breastCancerPipe };
	RunAllTests(binaryPredictors, binaryClassificationDatasets, extraTag: "dart");
	Done();
	}

The exact failures are due to calculated results not quite matching the baseline results. For example:

Message:
System.AggregateException : One or more errors occurred. (Values to compare are 240 and 238
AllowedVariance: 1E-07
delta: 2
delta2: 2
) (Output and baseline mismatch at line 11, expected ' positive || 240 | 1 | 0.9959' but got ' positive || 238 | 3 | 0.9876' : 'LightGBMBinary\LightGBM-TrainTest-breast-cancer-out.txt')

Again, these failing tests pass when AllFeatures is replaced by Features. @rauhs , would you happen to have any idea why this is happening?

[mstfbl]

Update: I investigated the IDataView data that is being passed to LightGbm right before its training. This particular IDataView contains 3 colums: Attributes "Attr", Label, and Features.

The Label and Features columns are correctly mapped, but the Attributes column is not. It seems like this Attributes column is taking the 7th column in the breast-cancer.txt file.

This might indicate a bug that exists within RoleMappedData and/or TextLoader.

[rauhs]

1. There are a bunch of "?" in that column in the original data set:

https://github.com/dotnet/machinelearning/blob/master/test/data/breast-cancer.txt#L58

2. They're also completely filtered out in:

https://github.com/dotnet/machinelearning/blob/master/test/data/breast-cancer-withheader.txt#L1

Note: The first line in that file "// Number of Instances: 699 (as of 15 July 1992)" is a lie. There are only 682 instances.

Not sure if this is all on purpose or some accident. I don't see any comments.

[mstfbl]

Hi @rauhs, thank you, I also saw the "?" missing values in breast-cancer.txt. It looks like these missing values are not being imputed with the CursOpt.AllFeatures change. The TestDatasets.breastCancerPipe dataset has missing values, whereas TestDatasets.breastCancerPipeMissing does not have missing values. Thus if the failing tests use TestDatasets.breastCancerPipeMissing, these builds will pass with these changes.

[justinormont]

@mstfbl commented 2020-02-05T17:30:49Z
Hi @rauhs, thank you, I also saw the "?" missing values in breast-cancer.txt. It looks like these missing values are not being imputed with the CursOpt.AllFeatures change. The TestDatasets.breastCancerPipe dataset has missing values, whereas TestDatasets.breastCancerPipeMissing does not have missing values. Thus if the failing tests use TestDatasets.breastCancerPipeMissing, these builds will pass with these changes.

I'd expect breastCancerPipe to change, and breastCancerPipeMissing to not change due to adding the AllFeatures flag.

Column 6 of breast-cancer.txt has the only NA values, which is read into both Mixed and More columns, which the NAHandle does run on for breastCancerPipeMissing.

So all cases of NA values are filled for breastCancerPipeMissing before LightGBM would see it (so no changes should be seen). And breastCancerPipe lets thru the NA values (so a baseline change should be seen).

Also, the next one should not change breastCancerPipeMissingFilter, as is drops the rows with with missing (NA) values.

tldr: This test output should change. Update breastCancerPipe baseline for LightGBM.