Skip to content

LightGbm default evaluation metrics in ML.NET do not conform to standalone LightGbm #3822

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
najeeb-kazmi opened this issue Jun 4, 2019 · 0 comments · Fixed by #3859
Closed
Assignees

Comments

@najeeb-kazmi
Copy link
Member

Found while investigating fix for #3761

In ML.NET LightGbm wrapper, the default EvaluationMetric is set to EvaluateMetricType.Error for multiclass, EvaluationMetricType.LogLoss for binary, and so on. On the other hand, in standalone LightGbm, the default evaluation metric is "", which means that LightGbm will automatically select the default metric for the given objective function.

This leads to inconsistent behavior from the user's perspective: If a user specified EvaluationMetric = EvaluateMetricType.Default, the parameter passed to LightGbm would be the empty string "" but if they do not specify EvaluationMetric at all, the parameter passed to LightGbm would be Error for multiclass, LogLoss for binary, and so on.

We need to investigate whether these metrics are indeed the defaults for the respective objective functions in LightGbm, and if they are not, then change the defaults in ML.NET to conform to standalone LightGbm. Note that this would be a breaking change.

@najeeb-kazmi najeeb-kazmi self-assigned this Jun 4, 2019
eerhardt pushed a commit to eerhardt/machinelearning that referenced this issue Jun 4, 2019
@ghost ghost locked as resolved and limited conversation to collaborators Mar 21, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
1 participant