Chapter 16 - Ensemble Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 16 - Ensemble Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 16.1 - Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* The idea of ensemble learning is to build a prediction model by combining the strengths of a collection of simpler base models. \n",
    "* Bagging, random forests, boosting, and stacking are examples of ensemble methods for learning.\n",
    "* One could characterize any dictionary method, such as regression splines, as an ensemble method, with the basis functions serving the role of weak learners.\n",
    "* Ensemble learning can be broken down into two tasks: developing a population of base learners from the training data, and then combining them to form the composite predictor.\n",
    "* This section takes it a step further -- build an ensemble model by conducting a regularized and supervised search in a high-dimensional space of weak learners."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 16.2 - Boosting and Regularization Paths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Intuition for the success of the shrinkage strategy of gradient boosting can be obtained by drawing analogies with penalized linear regression with a large basis expansion.\n",
    "* Consider the set of all trees that can be built on the data.  Now, consider a linear combination of all of these trees. Think of the trees themselves as basis functions in this linear expansion.  If you fit a regularized (e.g. lasso) least squares regression to this formula you end up with something that is really similar to the boosted version of these trees.\n",
    "* Boosting’s forward stagewise strategy with shrinkage approximately minimizes the same loss function with a lasso-style $L_1$ penalty. The model is built up slowly, searching through “model space” and adding shrunken basis functions derived from important predictors. \n",
    "\n",
    "**The “Bet on Sparsity” Principle**\n",
    "* The sometimes superior performance of boosting over procedures such as the support vector machine may be largely due to the implicit use of the $L_1$ versus $L_2$ penalty. \n",
    "* The shrinkage resulting from the $L_1$ penalty is better suited to sparse situations, where there are few basis functions with nonzero coefficients \n",
    "\n",
    "**Regularization Paths, Over-fitting and Margins**\n",
    "* Boosting is \"slow to overfit.\" Misclassification error is less sensitive to variance than is mean-squared error.\n",
    "* Gradient boosting with shrinkage fits an $L_1$ regularized monotone path in this space of its trees."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 16.3 - Learning Ensembles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* One way of post-processing boosting (or random forests) would be to fit a lasso on the collection of trees produced by the gradient boosting or random forest algorithms. \n",
    "* This can also reduce the # of trees necessary for prediction time.\n",
    "\n",
    "**Learning a Good Ensemble**\n",
    "* Not all ensembles of trees will perform well with post-processing. \n",
    "* In terms of basis functions, we want a collection that covers the space well in places where they are needed, and are sufficiently different from each other for the post-processor to be effective.\n",
    "\n",
    "**Rule Ensembles**\n",
    "* The idea is to enlarge an ensemble of trees by constructing a set of rules from each of the trees in the collection.\n",
    "* We can characterize a tree by its rules. Using the set of all rules from all trees we can create an enlarged model space and apply a lasso to these rules (the rules become the basis functions). "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}