syntax

Author: MatusChladek

README.ipynb

@@ -0,0 +1,117 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Regularization methods\n",
+    "## Shrinkage methods for regression models restriction\n",
+    "\n",
+    "There are two reasons why we are often not satisfied with the least squares\n",
+    "estimates.\n",
+    "\n",
+    "* The first is *prediction accuracy*: the least squares estimates often have\n",
+    "low bias but large variance. Prediction accuracy can sometimes be\n",
+    "improved by shrinking or setting some coefficients to zero. By doing\n",
+    "so we sacrifice a little bit of bias to reduce the variance of the predicted\n",
+    "values, and hence may improve the overall prediction accuracy.\n",
+    "\n",
+    "* The second reason is *interpretation*. With a large number of predictors,\n",
+    "we often would like to determine a smaller subset that exhibit\n",
+    "the strongest effects. In order to get the “big picture,” we are willing\n",
+    "to sacrifice some of the small details.\n",
+    "\n",
+    "Some of the most often used methods for model restriction include *Forward- and Backward-Stepwise Selection* or *Stagewise regression*. By retaining a subset of the predictors and discarding the rest, subset selection\n",
+    "produces a model that is interpretable and has possibly lower prediction\n",
+    "error than the full model. However, because it is a discrete process—\n",
+    "variables are either retained or discarded—it often exhibits high variance,\n",
+    "and so doesn’t reduce the prediction error of the full model. *Shrinkage*\n",
+    "*methods* (namely Ridge,Lasso and Elastic Net) are more continuous and don’t suffer as much from high\n",
+    "variability.\n",
+    "\n",
+    "As a continuous shrinkage method, Ridge regression achieves its better prediction performance through a bias–variance trade-off. However, ridge regression cannot produce a parsimonious model, for it always keeps all the predictors in the model.\n",
+    "\n",
+    "A promising technique called the Lasso was proposed by Tibshirani (1996). The lasso is a penalized least squares method imposing an L1-penalty on the regression coefficients. Owing to the nature of the L1-penalty, the lasso does both continuous shrinkage and automatic variable selection simultaneously. On the other hand introduction of L1 penalty causes no closed form solution in non-orthonormal case. This makes computation of Lasso estimates a quadratic programming problem.\n",
+    "\n",
+    "## Lasso estimation algorithms\n",
+    "### The LARS Algorithm\n",
+    "​\n",
+    "At the first step it identifies the variable\n",
+    "most correlated with the response. Rather than fit this variable completely,\n",
+    "LAR moves the coefficient of this variable continuously toward its leasts quares\n",
+    "value (causing its correlation with the evolving residual to decrease\n",
+    "in absolute value). As soon as another variable “catches up” in terms of\n",
+    "correlation with the residual, the process is paused. The second variable\n",
+    "then joins the active set, and their coefficients are moved together in a way\n",
+    "that keeps their correlations tied and decreasing.This process is continued until all the variables are in the model, and ends at the full least-squares\n",
+    "fit.\n",
+    "​\n",
+    "##### Naive LARS\n",
+    "​\n",
+    "1. Standardize the predictors to have mean zero and unit norm. Start\n",
+    "with the residual $r={y}-\\overline{y}; \\, \\beta_1,\\beta_2,...,\\beta_p = 0$\n",
+    "2. Find the predictor $x_j$ most correlated with $r$\n",
+    "3. Move $\\beta_j$ from 0 towards its least-squares coefficient $<x_j,r>$ until some\n",
+    "other competitor $x_k$ has as much correlation with the current residual\n",
+    "as does $x_j$\n",
+    "4. Move $\\beta_j$ and $\\beta_k$ in the direction defined by their joint least squares\n",
+    "coefficient of the current residual on $(x_j,x_k)$, until some other competitor\n",
+    "$x_l$ has as much correlation with the current residual\n",
+    "5. Continue in this way until all $p$ predictors have been entered. After\n",
+    "$min(N − 1,p)$ steps, we arrive at the full least-squares solution.\n",
+    "​\n",
+    "Tibshirani,Hastie&Friedman (2009) showed that LAR is almost identical to lasso path and differ only when coefficient crosses zero value. It appears that just simple modification of the LAR algorithm  gives the entire\n",
+    "lasso path, which is also piecewise-linear.\n",
+    "​\n",
+    "##### Lasso modification\n",
+    "​\n",
+    "* 4a. If a non-zero coefficient hits zero, drop its variable from the active set\n",
+    "of variables and recompute the current joint least squares direction.\n",
+    "​\n",
+    "All of the steps above introduce an efficient way for computing lasso having the same order of computation as Cholesky or QR decomposition which are used for least squares fitting.\n",
+    "\n",
+    "### Path-wise coordinate descent algorithm\n",
+    "\n",
+    "An alternate approach to the LARS algorithm for computing the lasso\n",
+    "solution is simple coordinate descent. This idea was proposed by Fu (1998)\n",
+    "and Daubechies et al. (2004), and later studied and generalized by Friedman\n",
+    "et al. (2007). The idea is to fix the penalty\n",
+    "parameter $\\lambda$ in the Lagrangian form and optimize successively over\n",
+    "each parameter, holding the other parameters fixed at their current values. This method is also called “one-at-atime”\n",
+    "coordinate-wise descent algorithm in the literature.\n",
+    "\n",
+    "A key point here: coordinate descent works so well because minimization can be done quickly,\n",
+    "and the relevant equations can be updated as we cycle through the variables.That makes it faster than LARS algorithm especially in large problems.\n",
+    "\n",
+    "By rearranging general Lasso Lagrangian form we can view problem as univariate lasso problem with explicit solution resulting in update\n",
+    "\n",
+    "$$\\hat\\beta(\\lambda )← S(\\sum_{i=1}^{N}{x_{ij}(y_i-\\hat y^{(j)})},\\lambda)$$\n",
+    "Here $S(\\hat\\beta, \\lambda) = sign(\\hat\\beta)(|\\hat\\beta|−\\lambda)_+$ is so called soft-thresholding operator. The first argument to $S(·)$ is the simple least-squares coefficient\n",
+    "of the partial residual on the standardized variable $x_{ij}$ . Repeated iteration\n",
+    "of updating function above—cycling through each variable in turn until convergence—yields\n",
+    "the lasso estimate $\\hat\\beta(\\lambda)$"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python [conda env:tf]",
+   "language": "python",
+   "name": "conda-env-tf-py"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}