DOC: Explain prediction when decision_function is zero

LogisticRegression and liblinear's predictions differ when the
decision function is zero. Explain why and what to do in that case.

Fixes scikit-learn#3600 (by documenting the won't fix status).

Author: MechCoder

doc/modules/linear_model.rst

@@ -701,6 +701,21 @@ so the "multinomial" setting does not learn sparse models.
 
   * :ref:`example_linear_model_plot_logistic_path.py`
 
+.. _liblinear_differences:
+
+.. topic:: Differences from liblinear:
+
+   There might be a difference in the scores obtained between
+   :class:`LogisticRegression` with ``solver=liblinear``
+   or :class:`LinearSVC` and the external liblinear library directly,
+   when ``fit_intercept=False`` and the fit ``coef_`` (or) the data to
+   be predicted are zeroes. This is because for the sample(s) with
+   ``decision_function`` zero, :class:`LogisticRegression` and :class:`LinearSVC`
+   predict the negative class, while liblinear predicts the positive class.
+   Note that a model with ``fit_intercept=False`` and having many samples with
+   ``decision_function`` zero, is likely to be a underfit, bad model and you are
+   advised to set ``fit_intercept=True`` and increase the intercept_scaling.
+
 .. note:: **Feature selection with sparse logistic regression**
 
    A logistic regression with L1 penalty yields sparse models, and can

sklearn/linear_model/logistic.py

@@ -826,7 +826,12 @@ class LogisticRegression(BaseEstimator, LinearClassifierMixin,
     to have slightly different results for the same input data. If
     that happens, try with a smaller tol parameter.
 
-    References:
+    Predict output may not match that of standalone liblinear in certain
+    cases. See :ref:`differences from liblinear <liblinear_differences>`
+    in the narrative documentation.
+
+    References
+    ----------
 
     LIBLINEAR -- A Library for Large Linear Classification
         http://www.csie.ntu.edu.tw/~cjlin/liblinear/

sklearn/linear_model/tests/test_logistic.py

@@ -514,3 +514,21 @@ def test_logistic_regression_multinomial():
     clf_path.fit(X, y)
     assert_array_almost_equal(clf_path.coef_, clf_int.coef_, decimal=3)
     assert_almost_equal(clf_path.intercept_, clf_int.intercept_, decimal=3)
+
+
+def test_liblinear_decision_function_zero():
+    """Test negative prediction when decision_function values are zero.
+
+    Liblinear predicts the positive class when decision_function values
+    are zero. This is a test to verify that we do not do the same.
+    See Issue: https://github.com/scikit-learn/scikit-learn/issues/3600
+    and the PR https://github.com/scikit-learn/scikit-learn/pull/3623
+    """
+    rng = np.random.RandomState(0)
+    X, y = make_classification(n_samples=5, n_features=5)
+    clf = LogisticRegression(fit_intercept=False)
+    clf.fit(X, y)
+
+    # Dummy data such that the decision function becomes zero.
+    X = np.zeros((5, 5))
+    assert_array_equal(clf.predict(X), np.zeros(5))

sklearn/svm/classes.py

@@ -109,6 +109,10 @@ class frequencies.
     The underlying implementation (liblinear) uses a sparse internal
     representation for the data that will incur a memory copy.
 
+    Predict output may not match that of standalone liblinear in certain
+    cases. See :ref:`differences from liblinear <liblinear_differences>`
+    in the narrative documentation.
+
     **References:**
     `LIBLINEAR: A Library for Large Linear Classification
     <http://www.csie.ntu.edu.tw/~cjlin/liblinear/>`__