Merge pull request scikit-learn#2862 from MechCoder/LogCV

[MRG] Logistic Regression CV

Author: agramfort

doc/modules/classes.rst

@@ -639,6 +639,7 @@ From text
    linear_model.LassoLarsIC
    linear_model.LinearRegression
    linear_model.LogisticRegression
+   linear_model.LogisticRegressionCV
    linear_model.MultiTaskLasso
    linear_model.MultiTaskElasticNet
    linear_model.MultiTaskLassoCV

doc/modules/linear_model.rst

@@ -649,10 +649,10 @@ rather than regression. Logistic regression is also known in the literature as
 logit regression, maximum-entropy classification (MaxEnt)
 or the log-linear classifier. In this model, the probabilities describing the possible outcomes of a single trial are modeled using a `logistic function <http://en.wikipedia.org/wiki/Logistic_function>`_.
 
-The implementation of logistic regression in scikit-learn can be accessed from 
-class :class:`LogisticRegression`. This 
+The implementation of logistic regression in scikit-learn can be accessed from
+class :class:`LogisticRegression`. This
 implementation can fit a multiclass (one-vs-rest) logistic regression with optional
-L2 or L1 regularization. 
+L2 or L1 regularization.
 
 As an optimization problem, binary class L2 penalized logistic regression minimizes
 the following cost function:
@@ -663,7 +663,14 @@ Similarly, L1 regularized logistic regression solves the following optimization
 
 .. math:: \underset{w, c}{min\,} \|w\|_1 + C \sum_{i=1}^n \log(\exp(- y_i (X_i^T w + c)) + 1) .
 
-L1 penalization yields sparse predicting weights.
+The solvers implemented in the class :class:`LogisticRegression`
+are "liblinear" (which is a wrapper around the C++ library,
+LIBLINEAR), "newton-cg" and "lbfgs".
+
+The lbfgs and newton-cg solvers only support L2 penalization and are found
+to converge faster for some high dimensional data. L1 penalization yields
+sparse predicting weights.
+
 For L1 penalization :func:`sklearn.svm.l1_min_c` allows to calculate
 the lower bound for C in order to get a non "null" (all feature weights to
 zero) model.
@@ -685,6 +692,12 @@ which is shipped with scikit-learn.
    thus be used to perform feature selection, as detailed in
    :ref:`l1_feature_selection`.
 
+:class:`LogisticRegressionCV` implements Logistic Regression with
+builtin cross-validation to find out the optimal C parameter. In
+general the "newton-cg" and "lbfgs" solvers are found to be faster
+due to warm-starting. For the multiclass case, One-vs-All is used
+and an optimal C is obtained for each class.
+
 
 Stochastic Gradient Descent - SGD
 =================================

doc/tutorial/statistical_inference/supervised_learning.rst

@@ -370,8 +370,8 @@ function or **logistic** function:
     >>> logistic = linear_model.LogisticRegression(C=1e5)
     >>> logistic.fit(iris_X_train, iris_y_train)
     LogisticRegression(C=100000.0, class_weight=None, dual=False,
-              fit_intercept=True, intercept_scaling=1, penalty='l2',
-              random_state=None, tol=0.0001)
+              fit_intercept=True, intercept_scaling=1, max_iter=100,
+              penalty='l2', random_state=None, solver='liblinear', tol=0.0001)
 
 This is known as :class:`LogisticRegression`.
 

doc/whats_new.rst

@@ -21,6 +21,9 @@ New features
    - Add the :func:`metrics.label_ranking_average_precision_score` metrics. By
      `Arnaud Joly`_.
 
+   - Added :class:`linear_model.LogisticRegressionCV`. By
+     `Manoj Kumar`_, `Fabian Pedregosa`_, `Gael Varoquaux`_
+     and `Alexandre Gramfort`_.
 
 Enhancements
 ............
@@ -29,6 +32,9 @@ Enhancements
    - Add support for sample weights in scorer objects.  Metrics with sample
      weight support will automatically benefit from it.
 
+   - Added ``newton-cg`` and `lbfgs` solver support in
+     :class:`linear_model.LogisticRegression`. By `Manoj Kumar`_.
+
 
 Documentation improvements
 ..........................

sklearn/linear_model/__init__.py

@@ -22,7 +22,8 @@
 from .stochastic_gradient import SGDClassifier, SGDRegressor
 from .ridge import (Ridge, RidgeCV, RidgeClassifier, RidgeClassifierCV,
                     ridge_regression)
-from .logistic import LogisticRegression
+from .logistic import (LogisticRegression, LogisticRegressionCV,
+                       logistic_regression_path)
 from .omp import (orthogonal_mp, orthogonal_mp_gram, OrthogonalMatchingPursuit,
                   OrthogonalMatchingPursuitCV)
 from .passive_aggressive import PassiveAggressiveClassifier
@@ -48,6 +49,7 @@
            'LinearRegression',
            'Log',
            'LogisticRegression',
+           'LogisticRegressionCV',
            'ModifiedHuber',
            'MultiTaskElasticNet',
            'MultiTaskElasticNetCV',
@@ -71,6 +73,7 @@
            'lars_path',
            'lasso_path',
            'lasso_stability_path',
+           'logistic_regression_path',
            'orthogonal_mp',
            'orthogonal_mp_gram',
            'ridge_regression',