Skip to content

Stop LightGbm Warning for Default Metric Input [Issue #3965 Fix] #4007

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Aug 5, 2019

Conversation

rayankrish
Copy link

Issue #3965 reported that a warning, "LightGBM] [Warning] Unknown parameter metric=" is produced when the default metric is used. This warning came after this commit which aimed to provide a consistent user experience from an ML.NET implementation of LightGbm with standalone LightGbm. If a user were to set EvaluationMetric = EvaluateMetricType.Default, they might expect that this would set the EvaluationMetric to "" and assigned the metric based on the objective as shown in the LightGbm docs. When the correction was made, this warning began to appear when the metric parameter was set to "". which was also being produced in LightGbm alone. The only way to prevent this error would be to not assign a parameter to the metric at all.

This warning has not appeared in previous versions of ML.NET and can be prevent by assigning the correct metric based on the objective as was previously done.

To prevent this warning, the changes from this commit were reverted.

@najeeb-kazmi
Copy link
Member

If the only reason we are reverting the defaults is this warning that is not affecting training, we should weigh the two options:

  1. Keep the defaults at the expense of having this warning, which might cause confusion for the user.

  2. Revert defaults and avoid the warning at the expense of having inconsistent user experience compared to standalone LightGbm as described in LightGbm default evaluation metrics in ML.NET do not conform to standalone LightGbm #3822 and Change default EvaluationMetric for LightGbm trainers to conform to d… #3859, although this will also cause confusion for the user.

I have a preference for (1) from the point of view of a LightGbm user coming to ML.NET and finding that the defaults are different from what they are used to. That this warning is being printed when the correct LightGbm default i.e. "" is being passed to LightGbm seems to be an issue that should be fixed on the LightGbm side.

@rayankrish rayankrish merged commit 820d359 into dotnet:master Aug 5, 2019
@ghost ghost locked as resolved and limited conversation to collaborators Mar 21, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants