Chapter 07 - Model Assessment and Selection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 7 - Model Assessment and Selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 7.1 - Introduction\n",
    "* The generalization performance of a learning method relates to its prediction capability on independent test data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 7.2 - Bias and Model Complexity\n",
    "* Training error is the average of the loss function evaluated over the training sample.\n",
    "* Test error, also referred to as generalization error, is the prediction error over an independent test sample.\n",
    "* Training error is not a good estimate of the test error.\n",
    "* Estimating test error is an essential task - this can be done analytically or empirically.\n",
    "\n",
    "Two goals:\n",
    "* Model selection: estimating the performance of different models in order to choose the best one.\n",
    "* Model assessment: having chosen a final model, estimating its prediction error (generalization error) on new data.\n",
    "\n",
    "The best approach for both problems is to randomly divide the dataset into three parts: \n",
    "* The training set is used to fit the models. \n",
    "* The validation set is used to estimate prediction error for model selection.\n",
    "* The test set is used for assessment of the generalization error of the final chosen model. Ideally, should be kept in a “vault,” and be brought out only at the end of the data analysis.\n",
    "\n",
    "Bias-variance tradeoff:\n",
    "* As model complexity increases, typically so does prediction variance.\n",
    "* As prediction variance increases, typically training error (bias) decreases.\n",
    "* As prediction variance decreases, typically training error (bias) increases. This is the bias-variance tradeoff.\n",
    "* Test error does not necessarily move in the same direction as train error (e.g. the more overfit you are the worse your test error can get)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_boston\n",
    "boston = load_boston()\n",
    "feature_names = boston.feature_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(boston.data, boston.target, test_size=0.33, random_state=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate bias-variance tradeoff chart for K-Neighbors regression for n_neighbors = 1 to 50. \n",
    "# Smaller n_neighbors = more model complexity.\n",
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "train_errors = []\n",
    "train_variances = []\n",
    "test_errors = []\n",
    "test_variances = []\n",
    "neighbor_list = list(range(1, 50))\n",
    "\n",
    "for n_neighbors in neighbor_list:\n",
    "    knn = KNeighborsRegressor(n_neighbors=n_neighbors, weights='uniform', algorithm='auto', leaf_size=30, p=2, \n",
    "                              metric='minkowski', metric_params=None, n_jobs=-1)\n",
    "    \n",
    "    knn.fit(X_train, y_train)\n",
    "    train_predictions = knn.predict(X_train)\n",
    "    test_predictions = knn.predict(X_test)\n",
    "    \n",
    "    train_errors.append(mean_squared_error(y_train, train_predictions))\n",
    "    test_errors.append(mean_squared_error(y_test, test_predictions))\n",
    "    \n",
    "    train_variances.append(train_predictions.var())\n",
    "    test_variances.append(test_predictions.var())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<div id=\"altair-viz-0245c071f6ec427d98efdea03e65a06c\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "  (function(spec, embedOpt){\n",
       "    let outputDiv = document.currentScript.previousElementSibling;\n",
       "    if (outputDiv.id !== \"altair-viz-0245c071f6ec427d98efdea03e65a06c\") {\n",
       "      outputDiv = document.getElementById(\"altair-viz-0245c071f6ec427d98efdea03e65a06c\");\n",
       "    }\n",
       "    const paths = {\n",
       "      \"vega\": \"https://cdn.jsdelivr.net/npm//vega@5?noext\",\n",
       "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm//vega-lib?noext\",\n",
       "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm//vega-lite@4.8.1?noext\",\n",
       "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm//vega-embed@6?noext\",\n",
       "    };\n",
       "\n",
       "    function loadScript(lib) {\n",
       "      return new Promise(function(resolve, reject) {\n",
       "        var s = document.createElement('script');\n",
       "        s.src = paths[lib];\n",
       "        s.async = true;\n",
       "        s.onload = () => resolve(paths[lib]);\n",
       "        s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
       "        document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "      });\n",
       "    }\n",
       "\n",
       "    function showError(err) {\n",
       "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
       "      throw err;\n",
       "    }\n",
       "\n",
       "    function displayChart(vegaEmbed) {\n",
       "      vegaEmbed(outputDiv, spec, embedOpt)\n",
       "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
       "    }\n",
       "\n",
       "    if(typeof define === \"function\" && define.amd) {\n",
       "      requirejs.config({paths});\n",
       "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
       "    } else if (typeof vegaEmbed === \"function\") {\n",
       "      displayChart(vegaEmbed);\n",
       "    } else {\n",
       "      loadScript(\"vega\")\n",
       "        .then(() => loadScript(\"vega-lite\"))\n",
       "        .then(() => loadScript(\"vega-embed\"))\n",
       "        .catch(showError)\n",
       "        .then(() => displayChart(vegaEmbed));\n",
       "    }\n",
       "  })({\"config\": {\"view\": {\"continuousWidth\": 400, \"continuousHeight\": 300}}, \"data\": {\"name\": \"data-46807cfb66c3c2af5425d4cae582bf46\"}, \"mark\": {\"type\": \"circle\", \"size\": 60}, \"encoding\": {\"color\": {\"type\": \"nominal\", \"field\": \"Metric\"}, \"tooltip\": [{\"type\": \"quantitative\", \"field\": \"K\"}, {\"type\": \"quantitative\", \"field\": \"Value\"}, {\"type\": \"nominal\", \"field\": \"Metric\"}], \"x\": {\"type\": \"quantitative\", \"field\": \"K\"}, \"y\": {\"type\": \"quantitative\", \"field\": \"Value\"}}, \"selection\": {\"selector001\": {\"type\": \"interval\", \"bind\": \"scales\", \"encodings\": [\"x\", \"y\"]}}, \"title\": \"Bias-Variance Tradeoff For K-Nearest Neighbors\", \"$schema\": \"https://vega.github.io/schema/vega-lite/v4.8.1.json\", \"datasets\": {\"data-46807cfb66c3c2af5425d4cae582bf46\": [{\"K\": 1, \"Value\": 0.0, \"Metric\": \"Train Bias\"}, {\"K\": 2, \"Value\": 11.856334808259588, \"Metric\": \"Train Bias\"}, {\"K\": 3, \"Value\": 16.63715830875123, \"Metric\": \"Train Bias\"}, {\"K\": 4, \"Value\": 19.731666666666666, \"Metric\": \"Train Bias\"}, {\"K\": 5, \"Value\": 20.927918584070795, \"Metric\": \"Train Bias\"}, {\"K\": 6, \"Value\": 23.158055555555556, \"Metric\": \"Train Bias\"}, {\"K\": 7, \"Value\": 26.31987839383541, \"Metric\": \"Train Bias\"}, {\"K\": 8, \"Value\": 29.73313928834808, \"Metric\": \"Train Bias\"}, {\"K\": 9, \"Value\": 30.665741287009727, \"Metric\": \"Train Bias\"}, {\"K\": 10, \"Value\": 32.40738289085546, \"Metric\": \"Train Bias\"}, {\"K\": 11, \"Value\": 33.89694799970745, \"Metric\": \"Train Bias\"}, {\"K\": 12, \"Value\": 34.69717100950508, \"Metric\": \"Train Bias\"}, {\"K\": 13, \"Value\": 35.570124801452245, \"Metric\": \"Train Bias\"}, {\"K\": 14, \"Value\": 36.85645415688399, \"Metric\": \"Train Bias\"}, {\"K\": 15, \"Value\": 37.83708816781384, \"Metric\": \"Train Bias\"}, {\"K\": 16, \"Value\": 39.074728521386426, \"Metric\": \"Train Bias\"}, {\"K\": 17, \"Value\": 40.2422408671954, \"Metric\": \"Train Bias\"}, {\"K\": 18, \"Value\": 41.061846116027525, \"Metric\": \"Train Bias\"}, {\"K\": 19, \"Value\": 41.621266638884116, \"Metric\": \"Train Bias\"}, {\"K\": 20, \"Value\": 41.77643584070796, \"Metric\": \"Train Bias\"}, {\"K\": 21, \"Value\": 42.654833945377554, \"Metric\": \"Train Bias\"}, {\"K\": 22, \"Value\": 42.82924778761062, \"Metric\": \"Train Bias\"}, {\"K\": 23, \"Value\": 43.54897095315366, \"Metric\": \"Train Bias\"}, {\"K\": 24, \"Value\": 44.486234380121275, \"Metric\": \"Train Bias\"}, {\"K\": 25, \"Value\": 45.05033373451327, \"Metric\": \"Train Bias\"}, {\"K\": 26, \"Value\": 46.00042973590966, \"Metric\": \"Train Bias\"}, {\"K\": 27, \"Value\": 46.69888071508633, \"Metric\": \"Train Bias\"}, {\"K\": 28, \"Value\": 47.317606443019685, \"Metric\": \"Train Bias\"}, {\"K\": 29, \"Value\": 48.128818094767084, \"Metric\": \"Train Bias\"}, {\"K\": 30, \"Value\": 48.656432972795805, \"Metric\": \"Train Bias\"}, {\"K\": 31, \"Value\": 48.74491904020824, \"Metric\": \"Train Bias\"}, {\"K\": 32, \"Value\": 49.382531370990044, \"Metric\": \"Train Bias\"}, {\"K\": 33, \"Value\": 49.82091366873346, \"Metric\": \"Train Bias\"}, {\"K\": 34, \"Value\": 50.290503235651364, \"Metric\": \"Train Bias\"}, {\"K\": 35, \"Value\": 50.65203373668052, \"Metric\": \"Train Bias\"}, {\"K\": 36, \"Value\": 51.085024240686124, \"Metric\": \"Train Bias\"}, {\"K\": 37, \"Value\": 51.71686929934, \"Metric\": \"Train Bias\"}, {\"K\": 38, \"Value\": 52.03054811282983, \"Metric\": \"Train Bias\"}, {\"K\": 39, \"Value\": 52.12902268923372, \"Metric\": \"Train Bias\"}, {\"K\": 40, \"Value\": 52.56150271017699, \"Metric\": \"Train Bias\"}, {\"K\": 41, \"Value\": 52.81090399554978, \"Metric\": \"Train Bias\"}, {\"K\": 42, \"Value\": 53.05738839390231, \"Metric\": \"Train Bias\"}, {\"K\": 43, \"Value\": 53.32906815611085, \"Metric\": \"Train Bias\"}, {\"K\": 44, \"Value\": 53.4782646761257, \"Metric\": \"Train Bias\"}, {\"K\": 45, \"Value\": 53.684756123675285, \"Metric\": \"Train Bias\"}, {\"K\": 46, \"Value\": 54.32980798077298, \"Metric\": \"Train Bias\"}, {\"K\": 47, \"Value\": 54.38111577603555, \"Metric\": \"Train Bias\"}, {\"K\": 48, \"Value\": 54.622626021693705, \"Metric\": \"Train Bias\"}, {\"K\": 49, \"Value\": 54.88232503909998, \"Metric\": \"Train Bias\"}, {\"K\": 1, \"Value\": 54.347245508982034, \"Metric\": \"Test Bias\"}, {\"K\": 2, \"Value\": 54.02017964071856, \"Metric\": \"Test Bias\"}, {\"K\": 3, \"Value\": 53.00035262807718, \"Metric\": \"Test Bias\"}, {\"K\": 4, \"Value\": 54.66096556886229, \"Metric\": \"Test Bias\"}, {\"K\": 5, \"Value\": 53.115176047904185, \"Metric\": \"Test Bias\"}, {\"K\": 6, \"Value\": 52.02358948769128, \"Metric\": \"Test Bias\"}, {\"K\": 7, \"Value\": 53.46803128437004, \"Metric\": \"Test Bias\"}, {\"K\": 8, \"Value\": 54.16697230538922, \"Metric\": \"Test Bias\"}, {\"K\": 9, \"Value\": 55.60008945072818, \"Metric\": \"Test Bias\"}, {\"K\": 10, \"Value\": 57.015569461077845, \"Metric\": \"Test Bias\"}, {\"K\": 11, \"Value\": 56.81514673133072, \"Metric\": \"Test Bias\"}, {\"K\": 12, \"Value\": 59.50465485695276, \"Metric\": \"Test Bias\"}, {\"K\": 13, \"Value\": 59.72160755412253, \"Metric\": \"Test Bias\"}, {\"K\": 14, \"Value\": 61.18137205181473, \"Metric\": \"Test Bias\"}, {\"K\": 15, \"Value\": 62.48385389221556, \"Metric\": \"Test Bias\"}, {\"K\": 16, \"Value\": 63.601233158682646, \"Metric\": \"Test Bias\"}, {\"K\": 17, \"Value\": 64.20520046412366, \"Metric\": \"Test Bias\"}, {\"K\": 18, \"Value\": 64.31396946847048, \"Metric\": \"Test Bias\"}, {\"K\": 19, \"Value\": 64.96260304875014, \"Metric\": \"Test Bias\"}, {\"K\": 20, \"Value\": 65.85623113772456, \"Metric\": \"Test Bias\"}, {\"K\": 21, \"Value\": 66.55390199193451, \"Metric\": \"Test Bias\"}, {\"K\": 22, \"Value\": 67.07941975553028, \"Metric\": \"Test Bias\"}, {\"K\": 23, \"Value\": 66.08435461779655, \"Metric\": \"Test Bias\"}, {\"K\": 24, \"Value\": 66.91843479707252, \"Metric\": \"Test Bias\"}, {\"K\": 25, \"Value\": 67.49119444311377, \"Metric\": \"Test Bias\"}, {\"K\": 26, \"Value\": 68.38811598696098, \"Metric\": \"Test Bias\"}, {\"K\": 27, \"Value\": 69.02284599525228, \"Metric\": \"Test Bias\"}, {\"K\": 28, \"Value\": 70.07349757118416, \"Metric\": \"Test Bias\"}, {\"K\": 29, \"Value\": 70.50368032069036, \"Metric\": \"Test Bias\"}, {\"K\": 30, \"Value\": 71.43774537591484, \"Metric\": \"Test Bias\"}, {\"K\": 31, \"Value\": 71.87418289331846, \"Metric\": \"Test Bias\"}, {\"K\": 32, \"Value\": 72.57700166074102, \"Metric\": \"Test Bias\"}, {\"K\": 33, \"Value\": 72.772099602448, \"Metric\": \"Test Bias\"}, {\"K\": 34, \"Value\": 72.70509163334232, \"Metric\": \"Test Bias\"}, {\"K\": 35, \"Value\": 73.00067505804718, \"Metric\": \"Test Bias\"}, {\"K\": 36, \"Value\": 73.37565540215864, \"Metric\": \"Test Bias\"}, {\"K\": 37, \"Value\": 73.63905407592412, \"Metric\": \"Test Bias\"}, {\"K\": 38, \"Value\": 73.78247059896827, \"Metric\": \"Test Bias\"}, {\"K\": 39, \"Value\": 74.06564921439173, \"Metric\": \"Test Bias\"}, {\"K\": 40, \"Value\": 74.53722327844312, \"Metric\": \"Test Bias\"}, {\"K\": 41, \"Value\": 74.90132456087231, \"Metric\": \"Test Bias\"}, {\"K\": 42, \"Value\": 74.9903081252461, \"Metric\": \"Test Bias\"}, {\"K\": 43, \"Value\": 75.28727893051108, \"Metric\": \"Test Bias\"}, {\"K\": 44, \"Value\": 75.4512297100015, \"Metric\": \"Test Bias\"}, {\"K\": 45, \"Value\": 75.48957093220966, \"Metric\": \"Test Bias\"}, {\"K\": 46, \"Value\": 75.53544785664965, \"Metric\": \"Test Bias\"}, {\"K\": 47, \"Value\": 75.452495371412, \"Metric\": \"Test Bias\"}, {\"K\": 48, \"Value\": 75.47738681491184, \"Metric\": \"Test Bias\"}, {\"K\": 49, \"Value\": 75.3496933164076, \"Metric\": \"Test Bias\"}, {\"K\": 1, \"Value\": 78.28222744320013, \"Metric\": \"Train Variance\"}, {\"K\": 2, \"Value\": 69.85311239895233, \"Metric\": \"Train Variance\"}, {\"K\": 3, \"Value\": 59.87599916464353, \"Metric\": \"Train Variance\"}, {\"K\": 4, \"Value\": 55.71632280001044, \"Metric\": \"Train Variance\"}, {\"K\": 5, \"Value\": 50.69593236397177, \"Metric\": \"Train Variance\"}, {\"K\": 6, \"Value\": 46.497385701675256, \"Metric\": \"Train Variance\"}, {\"K\": 7, \"Value\": 42.97552070286438, \"Metric\": \"Train Variance\"}, {\"K\": 8, \"Value\": 40.53569569530373, \"Metric\": \"Train Variance\"}, {\"K\": 9, \"Value\": 38.932358808804885, \"Metric\": \"Train Variance\"}, {\"K\": 10, \"Value\": 37.414601223449154, \"Metric\": \"Train Variance\"}, {\"K\": 11, \"Value\": 36.67564083728088, \"Metric\": \"Train Variance\"}, {\"K\": 12, \"Value\": 35.1730393644813, \"Metric\": \"Train Variance\"}, {\"K\": 13, \"Value\": 33.30481642109792, \"Metric\": \"Train Variance\"}, {\"K\": 14, \"Value\": 32.04017776896959, \"Metric\": \"Train Variance\"}, {\"K\": 15, \"Value\": 31.016236608530114, \"Metric\": \"Train Variance\"}, {\"K\": 16, \"Value\": 29.785239849548823, \"Metric\": \"Train Variance\"}, {\"K\": 17, \"Value\": 28.698366392149822, \"Metric\": \"Train Variance\"}, {\"K\": 18, \"Value\": 27.528853009705756, \"Metric\": \"Train Variance\"}, {\"K\": 19, \"Value\": 26.461411245268067, \"Metric\": \"Train Variance\"}, {\"K\": 20, \"Value\": 25.677873890324655, \"Metric\": \"Train Variance\"}, {\"K\": 21, \"Value\": 24.98423765978171, \"Metric\": \"Train Variance\"}, {\"K\": 22, \"Value\": 24.496750338590488, \"Metric\": \"Train Variance\"}, {\"K\": 23, \"Value\": 24.006448495258738, \"Metric\": \"Train Variance\"}, {\"K\": 24, \"Value\": 22.70520665585247, \"Metric\": \"Train Variance\"}, {\"K\": 25, \"Value\": 22.229490406522743, \"Metric\": \"Train Variance\"}, {\"K\": 26, \"Value\": 21.60656217759882, \"Metric\": \"Train Variance\"}, {\"K\": 27, \"Value\": 21.105771601506557, \"Metric\": \"Train Variance\"}, {\"K\": 28, \"Value\": 20.642622875670224, \"Metric\": \"Train Variance\"}, {\"K\": 29, \"Value\": 20.082937750102662, \"Metric\": \"Train Variance\"}, {\"K\": 30, \"Value\": 19.935082284931966, \"Metric\": \"Train Variance\"}, {\"K\": 31, \"Value\": 19.511759119400853, \"Metric\": \"Train Variance\"}, {\"K\": 32, \"Value\": 19.1292318311669, \"Metric\": \"Train Variance\"}, {\"K\": 33, \"Value\": 18.727600560736544, \"Metric\": \"Train Variance\"}, {\"K\": 34, \"Value\": 18.572317146013564, \"Metric\": \"Train Variance\"}, {\"K\": 35, \"Value\": 18.236290351579587, \"Metric\": \"Train Variance\"}, {\"K\": 36, \"Value\": 17.557198571648954, \"Metric\": \"Train Variance\"}, {\"K\": 37, \"Value\": 17.118879394069605, \"Metric\": \"Train Variance\"}, {\"K\": 38, \"Value\": 16.635962045563716, \"Metric\": \"Train Variance\"}, {\"K\": 39, \"Value\": 16.371172448619355, \"Metric\": \"Train Variance\"}, {\"K\": 40, \"Value\": 15.950374092202475, \"Metric\": \"Train Variance\"}, {\"K\": 41, \"Value\": 15.757188805401386, \"Metric\": \"Train Variance\"}, {\"K\": 42, \"Value\": 15.655138239399045, \"Metric\": \"Train Variance\"}, {\"K\": 43, \"Value\": 15.524311932599558, \"Metric\": \"Train Variance\"}, {\"K\": 44, \"Value\": 15.290445490815433, \"Metric\": \"Train Variance\"}, {\"K\": 45, \"Value\": 14.83160366589996, \"Metric\": \"Train Variance\"}, {\"K\": 46, \"Value\": 14.528797083815734, \"Metric\": \"Train Variance\"}, {\"K\": 47, \"Value\": 14.109564284578372, \"Metric\": \"Train Variance\"}, {\"K\": 48, \"Value\": 13.946747010204595, \"Metric\": \"Train Variance\"}, {\"K\": 49, \"Value\": 13.697918884798547, \"Metric\": \"Train Variance\"}, {\"K\": 1, \"Value\": 71.52597654989422, \"Metric\": \"Test Variance\"}, {\"K\": 2, \"Value\": 56.660731829753665, \"Metric\": \"Test Variance\"}, {\"K\": 3, \"Value\": 48.4135893482496, \"Metric\": \"Test Variance\"}, {\"K\": 4, \"Value\": 44.421502877478574, \"Metric\": \"Test Variance\"}, {\"K\": 5, \"Value\": 38.82984743805802, \"Metric\": \"Test Variance\"}, {\"K\": 6, \"Value\": 36.0366236190294, \"Metric\": \"Test Variance\"}, {\"K\": 7, \"Value\": 33.907255321935864, \"Metric\": \"Test Variance\"}, {\"K\": 8, \"Value\": 31.817234572770634, \"Metric\": \"Test Variance\"}, {\"K\": 9, \"Value\": 29.976890096498074, \"Metric\": \"Test Variance\"}, {\"K\": 10, \"Value\": 28.638341001828678, \"Metric\": \"Test Variance\"}, {\"K\": 11, \"Value\": 28.307656017701817, \"Metric\": \"Test Variance\"}, {\"K\": 12, \"Value\": 27.74276135852049, \"Metric\": \"Test Variance\"}, {\"K\": 13, \"Value\": 27.28588544485632, \"Metric\": \"Test Variance\"}, {\"K\": 14, \"Value\": 26.37966861340255, \"Metric\": \"Test Variance\"}, {\"K\": 15, \"Value\": 25.5251211716288, \"Metric\": \"Test Variance\"}, {\"K\": 16, \"Value\": 24.876837025798704, \"Metric\": \"Test Variance\"}, {\"K\": 17, \"Value\": 23.9881693753574, \"Metric\": \"Test Variance\"}, {\"K\": 18, \"Value\": 23.52702184232112, \"Metric\": \"Test Variance\"}, {\"K\": 19, \"Value\": 22.866675503969084, \"Metric\": \"Test Variance\"}, {\"K\": 20, \"Value\": 21.953873842733692, \"Metric\": \"Test Variance\"}, {\"K\": 21, \"Value\": 20.981504966766128, \"Metric\": \"Test Variance\"}, {\"K\": 22, \"Value\": 20.338398161365202, \"Metric\": \"Test Variance\"}, {\"K\": 23, \"Value\": 20.14063974244102, \"Metric\": \"Test Variance\"}, {\"K\": 24, \"Value\": 19.311541037187105, \"Metric\": \"Test Variance\"}, {\"K\": 25, \"Value\": 18.82914610921869, \"Metric\": \"Test Variance\"}, {\"K\": 26, \"Value\": 18.09899171292111, \"Metric\": \"Test Variance\"}, {\"K\": 27, \"Value\": 17.68302371920116, \"Metric\": \"Test Variance\"}, {\"K\": 28, \"Value\": 17.533680105571577, \"Metric\": \"Test Variance\"}, {\"K\": 29, \"Value\": 17.10545898341945, \"Metric\": \"Test Variance\"}, {\"K\": 30, \"Value\": 16.749908385225556, \"Metric\": \"Test Variance\"}, {\"K\": 31, \"Value\": 16.220289358785156, \"Metric\": \"Test Variance\"}, {\"K\": 32, \"Value\": 15.992423118069624, \"Metric\": \"Test Variance\"}, {\"K\": 33, \"Value\": 16.017665715401158, \"Metric\": \"Test Variance\"}, {\"K\": 34, \"Value\": 15.999488112848752, \"Metric\": \"Test Variance\"}, {\"K\": 35, \"Value\": 15.732539823396104, \"Metric\": \"Test Variance\"}, {\"K\": 36, \"Value\": 15.05509088221871, \"Metric\": \"Test Variance\"}, {\"K\": 37, \"Value\": 14.68732415085673, \"Metric\": \"Test Variance\"}, {\"K\": 38, \"Value\": 14.435490426581278, \"Metric\": \"Test Variance\"}, {\"K\": 39, \"Value\": 13.998494504218133, \"Metric\": \"Test Variance\"}, {\"K\": 40, \"Value\": 13.637110291243859, \"Metric\": \"Test Variance\"}, {\"K\": 41, \"Value\": 13.366901498630297, \"Metric\": \"Test Variance\"}, {\"K\": 42, \"Value\": 13.054791750565432, \"Metric\": \"Test Variance\"}, {\"K\": 43, \"Value\": 13.023484220775476, \"Metric\": \"Test Variance\"}, {\"K\": 44, \"Value\": 12.8742890355035, \"Metric\": \"Test Variance\"}, {\"K\": 45, \"Value\": 12.489256640588863, \"Metric\": \"Test Variance\"}, {\"K\": 46, \"Value\": 12.183477793176992, \"Metric\": \"Test Variance\"}, {\"K\": 47, \"Value\": 11.877751154454524, \"Metric\": \"Test Variance\"}, {\"K\": 48, \"Value\": 11.70389548777943, \"Metric\": \"Test Variance\"}, {\"K\": 49, \"Value\": 11.510296232211923, \"Metric\": \"Test Variance\"}]}}, {\"mode\": \"vega-lite\"});\n",
       "</script>"
      ],
      "text/plain": [
       "alt.Chart(...)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import altair as alt\n",
    "\n",
    "def _build_scatter_data(x, y, name):\n",
    "    return list(zip(x, y, [name]*len(y)))\n",
    "\n",
    "plot_data = (\n",
    "    _build_scatter_data(neighbor_list, train_errors, 'Train Bias') +\n",
    "    _build_scatter_data(neighbor_list, test_errors, 'Test Bias') +\n",
    "    _build_scatter_data(neighbor_list, train_variances, 'Train Variance') +\n",
    "    _build_scatter_data(neighbor_list, test_variances, 'Test Variance')\n",
    ")\n",
    "    \n",
    "source = pd.DataFrame(plot_data, columns=['K', 'Value', 'Metric'])\n",
    "\n",
    "chart = alt.Chart(source, title='Bias-Variance Tradeoff For K-Nearest Neighbors').mark_circle(size=60).encode(\n",
    "    x='K',\n",
    "    y='Value',\n",
    "    color='Metric',\n",
    "    tooltip=['K', 'Value', 'Metric']\n",
    ").interactive()\n",
    "\n",
    "chart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observations:\n",
    "\n",
    "* For K=1, as expected, we get 0 training error.\n",
    "* K=1 has more test/generalization error than K=2.\n",
    "* Bias and variance are inversely proportional.\n",
    "* Variance increases and bias decreases as model complexity increases."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 7.3 - The Bias-Variance Decomposition\n",
    "\n",
    "Assume that assume that $Y = f(X)+\\epsilon$, where $E(\\epsilon) = 0$ and $Var(\\epsilon) = \\sigma_\\epsilon^2$.\n",
    "\n",
    "Expected prediction error of a regression fit can be analytically decomposed into several parts:\n",
    "\n",
    "* Irreducible Error - variance of the target around its true mean $f(x_0)$, and cannot be avoided no matter how well we estimate $f(x_0)$, unless $\\sigma_\\epsilon^2 = 0$.\n",
    "* Bias^2 - the amount by which the average of our estimate differs from the true mean.\n",
    "* Variance - the expected squared deviation of $f(x_0)$ around its mean. Typically the more complex we make the model f, the lower the (squared) bias but the higher the variance.\n",
    "\n",
    "Variance reduction techniques can often introduce *estimation bias* because they restrict the mean to be further away from the actual best fit mean. A least-squares fit, for example, will have 0 *estimation bias*.  A Ridge, while introducting estimation bias, might actually be better because of the lowered variance in predictions.  I like to think of estimation bias being the result of a specific algorithm not being an unbiased estimator of the mean.  i.e. $E(\\theta) \\neq \\theta$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 7.4 - Optimism of the Training Error Rate\n",
    "* A fitting method typically adapts to the training data, and hence the apparent or training error $\\bar{err}$ will be an overly *optimistic* estimate of the generalization error $Err_T$.\n",
    "* In-sample error - average error given that the new \"test\" observations are at the same points as the training points but with newly observed response values.\n",
    "* Extra-sample error - the generalization error - test input vectors need not coincide with the training input vectors.\n",
    "* Optimism of the training error is defined as the difference between the in-sample error and the training error.\n",
    "* Optimism increases linearly with the number or inputs of basis functions we use, but decreases as the training sample size increases.\n",
    "* Methods like AIC, BIC, and others work by estimating the optimism and adding it to the training error.\n",
    "* Cross-validation and other methods like bootstrapping work by estimating the extra-sample error.\n",
    "* In-sample error is usually not of direct interest but for comparison between models it can be convenient and effective."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute AIC/BIC for different alpha's\n",
    "# Degrees of freedom = those with non-zero coefficients (see section 3.5, pg 79)\n",
    "from sklearn.linear_model import Lasso\n",
    "import numpy as np\n",
    "\n",
    "alphas = np.linspace(0.01, 0.20, 19)\n",
    "aics = []\n",
    "bics = []\n",
    "degrees_of_freedom =  []\n",
    "train_errors = []\n",
    "train_variances = []\n",
    "test_errors = []\n",
    "test_variances = []\n",
    "\n",
    "for alpha in alphas:\n",
    "    lasso = Lasso(alpha=alpha, fit_intercept=True, normalize=True, precompute=False, copy_X=True, \n",
    "                  max_iter=1000, tol=0.0001, warm_start=False, positive=False, random_state=None, \n",
    "                  selection='cyclic')\n",
    "\n",
    "    lasso.fit(X_train, y_train)\n",
    "    train_predictions = lasso.predict(X_train)\n",
    "    test_predictions = lasso.predict(X_test)\n",
    "\n",
    "    # compute AIC/BIC\n",
    "    resid = y_train - train_predictions\n",
    "    sse = sum(resid**2)\n",
    "    k = len(lasso.coef_[lasso.coef_ != 0]) # of variables\n",
    "    n = X_train.shape[0] # of observations\n",
    "    aic = 2*k + n*np.log(sse/n)\n",
    "    bic = k*np.log(n) + n*np.log(sse/n)\n",
    "    \n",
    "    aics.append(aic)\n",
    "    bics.append(bic)\n",
    "    degrees_of_freedom.append(k)\n",
    "    \n",
    "    # Compute train/test errors and variances\n",
    "    train_errors.append(mean_squared_error(y_train, train_predictions))\n",
    "    test_errors.append(mean_squared_error(y_test, test_predictions))\n",
    "    \n",
    "    train_variances.append(train_predictions.var())\n",
    "    test_variances.append(test_predictions.var())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<div id=\"altair-viz-dd50fdbea54b4c888ae00e41fe790a7c\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "  (function(spec, embedOpt){\n",
       "    let outputDiv = document.currentScript.previousElementSibling;\n",
       "    if (outputDiv.id !== \"altair-viz-dd50fdbea54b4c888ae00e41fe790a7c\") {\n",
       "      outputDiv = document.getElementById(\"altair-viz-dd50fdbea54b4c888ae00e41fe790a7c\");\n",
       "    }\n",
       "    const paths = {\n",
       "      \"vega\": \"https://cdn.jsdelivr.net/npm//vega@5?noext\",\n",
       "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm//vega-lib?noext\",\n",
       "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm//vega-lite@4.8.1?noext\",\n",
       "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm//vega-embed@6?noext\",\n",
       "    };\n",
       "\n",
       "    function loadScript(lib) {\n",
       "      return new Promise(function(resolve, reject) {\n",
       "        var s = document.createElement('script');\n",
       "        s.src = paths[lib];\n",
       "        s.async = true;\n",
       "        s.onload = () => resolve(paths[lib]);\n",
       "        s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
       "        document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "      });\n",
       "    }\n",
       "\n",
       "    function showError(err) {\n",
       "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
       "      throw err;\n",
       "    }\n",
       "\n",
       "    function displayChart(vegaEmbed) {\n",
       "      vegaEmbed(outputDiv, spec, embedOpt)\n",
       "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
       "    }\n",
       "\n",
       "    if(typeof define === \"function\" && define.amd) {\n",
       "      requirejs.config({paths});\n",
       "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
       "    } else if (typeof vegaEmbed === \"function\") {\n",
       "      displayChart(vegaEmbed);\n",
       "    } else {\n",
       "      loadScript(\"vega\")\n",
       "        .then(() => loadScript(\"vega-lite\"))\n",
       "        .then(() => loadScript(\"vega-embed\"))\n",
       "        .catch(showError)\n",
       "        .then(() => displayChart(vegaEmbed));\n",
       "    }\n",
       "  })({\"config\": {\"view\": {\"continuousWidth\": 400, \"continuousHeight\": 300}}, \"data\": {\"name\": \"data-262dde68b33aae495e172985a8e36363\"}, \"mark\": {\"type\": \"circle\", \"size\": 60}, \"encoding\": {\"color\": {\"type\": \"nominal\", \"field\": \"Metric\"}, \"tooltip\": [{\"type\": \"quantitative\", \"field\": \"Alpha\"}, {\"type\": \"quantitative\", \"field\": \"Value\"}, {\"type\": \"nominal\", \"field\": \"Metric\"}], \"x\": {\"type\": \"quantitative\", \"field\": \"Alpha\"}, \"y\": {\"type\": \"quantitative\", \"field\": \"Value\"}}, \"selection\": {\"selector002\": {\"type\": \"interval\", \"bind\": \"scales\", \"encodings\": [\"x\", \"y\"]}}, \"title\": \"Bias-Variance Tradeoff For Lasso\", \"$schema\": \"https://vega.github.io/schema/vega-lite/v4.8.1.json\", \"datasets\": {\"data-262dde68b33aae495e172985a8e36363\": [{\"Alpha\": 0.01, \"Value\": 21.70884454626817, \"Metric\": \"Train Bias\"}, {\"Alpha\": 0.020555555555555556, \"Value\": 23.597096328314134, \"Metric\": \"Train Bias\"}, {\"Alpha\": 0.03111111111111111, \"Value\": 25.221053315863834, \"Metric\": \"Train Bias\"}, {\"Alpha\": 0.04166666666666667, \"Value\": 25.890721776954283, \"Metric\": \"Train Bias\"}, {\"Alpha\": 0.052222222222222225, \"Value\": 26.719782113956352, \"Metric\": \"Train Bias\"}, {\"Alpha\": 0.06277777777777778, \"Value\": 27.69926934529512, \"Metric\": \"Train Bias\"}, {\"Alpha\": 0.07333333333333333, \"Value\": 28.78107748272529, \"Metric\": \"Train Bias\"}, {\"Alpha\": 0.08388888888888889, \"Value\": 30.03066502806311, \"Metric\": \"Train Bias\"}, {\"Alpha\": 0.09444444444444444, \"Value\": 31.171696854467598, \"Metric\": \"Train Bias\"}, {\"Alpha\": 0.105, \"Value\": 32.321380408483996, \"Metric\": \"Train Bias\"}, {\"Alpha\": 0.11555555555555555, \"Value\": 33.592553993854985, \"Metric\": \"Train Bias\"}, {\"Alpha\": 0.12611111111111112, \"Value\": 34.985370963607984, \"Metric\": \"Train Bias\"}, {\"Alpha\": 0.1366666666666667, \"Value\": 36.49986609550822, \"Metric\": \"Train Bias\"}, {\"Alpha\": 0.14722222222222223, \"Value\": 38.135980344356625, \"Metric\": \"Train Bias\"}, {\"Alpha\": 0.1577777777777778, \"Value\": 39.893826685461704, \"Metric\": \"Train Bias\"}, {\"Alpha\": 0.16833333333333333, \"Value\": 41.50601342354824, \"Metric\": \"Train Bias\"}, {\"Alpha\": 0.1788888888888889, \"Value\": 43.0819096344238, \"Metric\": \"Train Bias\"}, {\"Alpha\": 0.18944444444444447, \"Value\": 44.753655479488565, \"Metric\": \"Train Bias\"}, {\"Alpha\": 0.2, \"Value\": 46.52122949781151, \"Metric\": \"Train Bias\"}, {\"Alpha\": 0.01, \"Value\": 27.738491248060893, \"Metric\": \"Test Bias\"}, {\"Alpha\": 0.020555555555555556, \"Value\": 29.368305759545816, \"Metric\": \"Test Bias\"}, {\"Alpha\": 0.03111111111111111, \"Value\": 30.6698058202675, \"Metric\": \"Test Bias\"}, {\"Alpha\": 0.04166666666666667, \"Value\": 31.24730118898577, \"Metric\": \"Test Bias\"}, {\"Alpha\": 0.052222222222222225, \"Value\": 31.990366476567747, \"Metric\": \"Test Bias\"}, {\"Alpha\": 0.06277777777777778, \"Value\": 32.97882268686714, \"Metric\": \"Test Bias\"}, {\"Alpha\": 0.07333333333333333, \"Value\": 34.162741515610136, \"Metric\": \"Test Bias\"}, {\"Alpha\": 0.08388888888888889, \"Value\": 35.551128475435696, \"Metric\": \"Test Bias\"}, {\"Alpha\": 0.09444444444444444, \"Value\": 37.035795235946466, \"Metric\": \"Test Bias\"}, {\"Alpha\": 0.105, \"Value\": 38.6241815680332, \"Metric\": \"Test Bias\"}, {\"Alpha\": 0.11555555555555555, \"Value\": 40.35328236310028, \"Metric\": \"Test Bias\"}, {\"Alpha\": 0.12611111111111112, \"Value\": 42.223375962401, \"Metric\": \"Test Bias\"}, {\"Alpha\": 0.1366666666666667, \"Value\": 44.23451585199178, \"Metric\": \"Test Bias\"}, {\"Alpha\": 0.14722222222222223, \"Value\": 46.38672777351354, \"Metric\": \"Test Bias\"}, {\"Alpha\": 0.1577777777777778, \"Value\": 48.67993709529927, \"Metric\": \"Test Bias\"}, {\"Alpha\": 0.16833333333333333, \"Value\": 50.84568299385987, \"Metric\": \"Test Bias\"}, {\"Alpha\": 0.1788888888888889, \"Value\": 53.00155993078373, \"Metric\": \"Test Bias\"}, {\"Alpha\": 0.18944444444444447, \"Value\": 55.28111961831806, \"Metric\": \"Test Bias\"}, {\"Alpha\": 0.2, \"Value\": 57.68431773846198, \"Metric\": \"Test Bias\"}, {\"Alpha\": 0.01, \"Value\": 51.45458841303118, \"Metric\": \"Train Variance\"}, {\"Alpha\": 0.020555555555555556, \"Value\": 46.765482011404266, \"Metric\": \"Train Variance\"}, {\"Alpha\": 0.03111111111111111, \"Value\": 43.08272856222619, \"Metric\": \"Train Variance\"}, {\"Alpha\": 0.04166666666666667, \"Value\": 39.798185358055925, \"Metric\": \"Train Variance\"}, {\"Alpha\": 0.052222222222222225, \"Value\": 36.701322350907056, \"Metric\": \"Train Variance\"}, {\"Alpha\": 0.06277777777777778, \"Value\": 33.789731783328286, \"Metric\": \"Train Variance\"}, {\"Alpha\": 0.07333333333333333, \"Value\": 31.049853964036288, \"Metric\": \"Train Variance\"}, {\"Alpha\": 0.08388888888888889, \"Value\": 28.477803981841532, \"Metric\": \"Train Variance\"}, {\"Alpha\": 0.09444444444444444, \"Value\": 26.062242422753098, \"Metric\": \"Train Variance\"}, {\"Alpha\": 0.105, \"Value\": 23.770395980525898, \"Metric\": \"Train Variance\"}, {\"Alpha\": 0.11555555555555555, \"Value\": 21.600452852259536, \"Metric\": \"Train Variance\"}, {\"Alpha\": 0.12611111111111112, \"Value\": 19.552201668241374, \"Metric\": \"Train Variance\"}, {\"Alpha\": 0.1366666666666667, \"Value\": 17.62561462301944, \"Metric\": \"Train Variance\"}, {\"Alpha\": 0.14722222222222223, \"Value\": 15.820642743768014, \"Metric\": \"Train Variance\"}, {\"Alpha\": 0.1577777777777778, \"Value\": 14.13740054684871, \"Metric\": \"Train Variance\"}, {\"Alpha\": 0.16833333333333333, \"Value\": 12.570427087294574, \"Metric\": \"Train Variance\"}, {\"Alpha\": 0.1788888888888889, \"Value\": 11.10047683016811, \"Metric\": \"Train Variance\"}, {\"Alpha\": 0.18944444444444447, \"Value\": 9.72632718487875, \"Metric\": \"Train Variance\"}, {\"Alpha\": 0.2, \"Value\": 8.447993065344157, \"Metric\": \"Train Variance\"}, {\"Alpha\": 0.01, \"Value\": 64.8643185480961, \"Metric\": \"Test Variance\"}, {\"Alpha\": 0.020555555555555556, \"Value\": 59.08157905121369, \"Metric\": \"Test Variance\"}, {\"Alpha\": 0.03111111111111111, \"Value\": 53.59375785084431, \"Metric\": \"Test Variance\"}, {\"Alpha\": 0.04166666666666667, \"Value\": 49.03689721675581, \"Metric\": \"Test Variance\"}, {\"Alpha\": 0.052222222222222225, \"Value\": 44.73900428591878, \"Metric\": \"Test Variance\"}, {\"Alpha\": 0.06277777777777778, \"Value\": 40.80202565822875, \"Metric\": \"Test Variance\"}, {\"Alpha\": 0.07333333333333333, \"Value\": 37.29195839647929, \"Metric\": \"Test Variance\"}, {\"Alpha\": 0.08388888888888889, \"Value\": 33.98242811396847, \"Metric\": \"Test Variance\"}, {\"Alpha\": 0.09444444444444444, \"Value\": 31.060948365918797, \"Metric\": \"Test Variance\"}, {\"Alpha\": 0.105, \"Value\": 28.368940056702748, \"Metric\": \"Test Variance\"}, {\"Alpha\": 0.11555555555555555, \"Value\": 25.815374453188753, \"Metric\": \"Test Variance\"}, {\"Alpha\": 0.12611111111111112, \"Value\": 23.399971770139082, \"Metric\": \"Test Variance\"}, {\"Alpha\": 0.1366666666666667, \"Value\": 21.122699501106123, \"Metric\": \"Test Variance\"}, {\"Alpha\": 0.14722222222222223, \"Value\": 18.983421354855288, \"Metric\": \"Test Variance\"}, {\"Alpha\": 0.1577777777777778, \"Value\": 16.982450276939282, \"Metric\": \"Test Variance\"}, {\"Alpha\": 0.16833333333333333, \"Value\": 15.064025986272288, \"Metric\": \"Test Variance\"}, {\"Alpha\": 0.1788888888888889, \"Value\": 13.240937573941798, \"Metric\": \"Test Variance\"}, {\"Alpha\": 0.18944444444444447, \"Value\": 11.539676659697857, \"Metric\": \"Test Variance\"}, {\"Alpha\": 0.2, \"Value\": 9.960271461502012, \"Metric\": \"Test Variance\"}]}}, {\"mode\": \"vega-lite\"});\n",
       "</script>"
      ],
      "text/plain": [
       "alt.Chart(...)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plot_data = (\n",
    "    _build_scatter_data(alphas, train_errors, 'Train Bias') +\n",
    "    _build_scatter_data(alphas, test_errors, 'Test Bias') +\n",
    "    _build_scatter_data(alphas, train_variances, 'Train Variance') +\n",
    "    _build_scatter_data(alphas, test_variances, 'Test Variance')\n",
    ")\n",
    "    \n",
    "source = pd.DataFrame(plot_data, columns=['Alpha', 'Value', 'Metric'])\n",
    "\n",
    "chart = alt.Chart(source, title='Bias-Variance Tradeoff For Lasso').mark_circle(size=60).encode(\n",
    "    x='Alpha',\n",
    "    y='Value',\n",
    "    color='Metric',\n",
    "    tooltip=['Alpha', 'Value', 'Metric']\n",
    ").interactive()\n",
    "\n",
    "chart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 7.5 - Estimates of In-Sample Prediction Error\n",
    "* Akaike information criteria (AIC) - estimate of in-sample error when a log-likelihood loss function is used. To use it for model selection, we simply choose the model with the smallest AIC. Typically this can be used to determine how complex a model should be."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<div id=\"altair-viz-fef9f709260c485880a3c56563007304\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "  (function(spec, embedOpt){\n",
       "    let outputDiv = document.currentScript.previousElementSibling;\n",
       "    if (outputDiv.id !== \"altair-viz-fef9f709260c485880a3c56563007304\") {\n",
       "      outputDiv = document.getElementById(\"altair-viz-fef9f709260c485880a3c56563007304\");\n",
       "    }\n",
       "    const paths = {\n",
       "      \"vega\": \"https://cdn.jsdelivr.net/npm//vega@5?noext\",\n",
       "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm//vega-lib?noext\",\n",
       "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm//vega-lite@4.8.1?noext\",\n",
       "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm//vega-embed@6?noext\",\n",
       "    };\n",
       "\n",
       "    function loadScript(lib) {\n",
       "      return new Promise(function(resolve, reject) {\n",
       "        var s = document.createElement('script');\n",
       "        s.src = paths[lib];\n",
       "        s.async = true;\n",
       "        s.onload = () => resolve(paths[lib]);\n",
       "        s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
       "        document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "      });\n",
       "    }\n",
       "\n",
       "    function showError(err) {\n",
       "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
       "      throw err;\n",
       "    }\n",
       "\n",
       "    function displayChart(vegaEmbed) {\n",
       "      vegaEmbed(outputDiv, spec, embedOpt)\n",
       "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
       "    }\n",
       "\n",
       "    if(typeof define === \"function\" && define.amd) {\n",
       "      requirejs.config({paths});\n",
       "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
       "    } else if (typeof vegaEmbed === \"function\") {\n",
       "      displayChart(vegaEmbed);\n",
       "    } else {\n",
       "      loadScript(\"vega\")\n",
       "        .then(() => loadScript(\"vega-lite\"))\n",
       "        .then(() => loadScript(\"vega-embed\"))\n",
       "        .catch(showError)\n",
       "        .then(() => displayChart(vegaEmbed));\n",
       "    }\n",
       "  })({\"config\": {\"view\": {\"continuousWidth\": 400, \"continuousHeight\": 300}}, \"data\": {\"name\": \"data-9eeb08980eba107b09e19fab27fe8093\"}, \"mark\": {\"type\": \"circle\", \"size\": 60}, \"encoding\": {\"tooltip\": [{\"type\": \"quantitative\", \"field\": \"Alpha\"}, {\"type\": \"quantitative\", \"field\": \"AIC\"}], \"x\": {\"type\": \"quantitative\", \"field\": \"Alpha\"}, \"y\": {\"type\": \"quantitative\", \"field\": \"AIC\"}}, \"selection\": {\"selector003\": {\"type\": \"interval\", \"bind\": \"scales\", \"encodings\": [\"x\", \"y\"]}}, \"title\": \"AIC For Lasso\", \"$schema\": \"https://vega.github.io/schema/vega-lite/v4.8.1.json\", \"datasets\": {\"data-9eeb08980eba107b09e19fab27fe8093\": [{\"Alpha\": 0.01, \"AIC\": 1107.4329999176139, \"Metric\": \"AIC\"}, {\"Alpha\": 0.020555555555555556, \"AIC\": 1124.0549242577576, \"Metric\": \"AIC\"}, {\"Alpha\": 0.03111111111111111, \"AIC\": 1129.1392137700568, \"Metric\": \"AIC\"}, {\"Alpha\": 0.04166666666666667, \"AIC\": 1132.1969043363788, \"Metric\": \"AIC\"}, {\"Alpha\": 0.052222222222222225, \"AIC\": 1142.8820223765993, \"Metric\": \"AIC\"}, {\"Alpha\": 0.06277777777777778, \"AIC\": 1149.2606464366422, \"Metric\": \"AIC\"}, {\"Alpha\": 0.07333333333333333, \"AIC\": 1162.2484496698721, \"Metric\": \"AIC\"}, {\"Alpha\": 0.08388888888888889, \"AIC\": 1176.6562506525859, \"Metric\": \"AIC\"}, {\"Alpha\": 0.09444444444444444, \"AIC\": 1183.4720703323494, \"Metric\": \"AIC\"}, {\"Alpha\": 0.105, \"AIC\": 1195.7501121345247, \"Metric\": \"AIC\"}, {\"Alpha\": 0.11555555555555555, \"AIC\": 1208.8272038159166, \"Metric\": \"AIC\"}, {\"Alpha\": 0.12611111111111112, \"AIC\": 1222.5992708802137, \"Metric\": \"AIC\"}, {\"Alpha\": 0.1366666666666667, \"AIC\": 1236.9656129984467, \"Metric\": \"AIC\"}, {\"Alpha\": 0.14722222222222223, \"AIC\": 1251.83063096735, \"Metric\": \"AIC\"}, {\"Alpha\": 0.1577777777777778, \"AIC\": 1267.1071200992585, \"Metric\": \"AIC\"}, {\"Alpha\": 0.16833333333333333, \"AIC\": 1274.7111901902106, \"Metric\": \"AIC\"}, {\"Alpha\": 0.1788888888888889, \"AIC\": 1287.3439778749503, \"Metric\": \"AIC\"}, {\"Alpha\": 0.18944444444444447, \"AIC\": 1300.2496906329213, \"Metric\": \"AIC\"}, {\"Alpha\": 0.2, \"AIC\": 1313.381068751946, \"Metric\": \"AIC\"}]}}, {\"mode\": \"vega-lite\"});\n",
       "</script>"
      ],
      "text/plain": [
       "alt.Chart(...)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plot_data = _build_scatter_data(alphas, bics, 'AIC')\n",
    "    \n",
    "source = pd.DataFrame(plot_data, columns=['Alpha', 'AIC', 'Metric'])\n",
    "\n",
    "chart = alt.Chart(source, title='AIC For Lasso').mark_circle(size=60).encode(\n",
    "    x='Alpha',\n",
    "    y='AIC',\n",
    "    tooltip=['Alpha', 'AIC']\n",
    ").interactive()\n",
    "\n",
    "chart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 7.6 - The Effective Number of Parameters\n",
    "* The concept of # of parameters can be generalized, for example, to cases with regularization where we are shrinking estimates.\n",
    "* A linear fitting method allows us to write $\\hat{y} = Sy$ where $S$ is an $NxN$ matrix depending on the input vectors $x_i$ but not on $y_i$.\n",
    "* The effective # of parameters is then defined as df(S) = trace(S), the sum of the diagonal elements of $S$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 7.7 - The Bayesian Approach and BIC\n",
    "* The Bayesian information criteria (BIC), like AIC, is applicable in settings where the fitting is carried out by maximization of a log-likelihood.\n",
    "* Arises from the Bayesian approach to model selection - compute a posterior probability that the given model is the correct one, given the data.\n",
    "* Select the model with the minimum BIC.\n",
    "* Takes into account both the statistical goodness of fit and the number of parameters that have to be estimated to achieve this particular degree of fit, by imposing a penalty for increasing the number of parameters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<div id=\"altair-viz-fb0e94f60f424596b872ff34c8cd7dfa\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "  (function(spec, embedOpt){\n",
       "    let outputDiv = document.currentScript.previousElementSibling;\n",
       "    if (outputDiv.id !== \"altair-viz-fb0e94f60f424596b872ff34c8cd7dfa\") {\n",
       "      outputDiv = document.getElementById(\"altair-viz-fb0e94f60f424596b872ff34c8cd7dfa\");\n",
       "    }\n",
       "    const paths = {\n",
       "      \"vega\": \"https://cdn.jsdelivr.net/npm//vega@5?noext\",\n",
       "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm//vega-lib?noext\",\n",
       "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm//vega-lite@4.8.1?noext\",\n",
       "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm//vega-embed@6?noext\",\n",
       "    };\n",
       "\n",
       "    function loadScript(lib) {\n",
       "      return new Promise(function(resolve, reject) {\n",
       "        var s = document.createElement('script');\n",
       "        s.src = paths[lib];\n",
       "        s.async = true;\n",
       "        s.onload = () => resolve(paths[lib]);\n",
       "        s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
       "        document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "      });\n",
       "    }\n",
       "\n",
       "    function showError(err) {\n",
       "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
       "      throw err;\n",
       "    }\n",
       "\n",
       "    function displayChart(vegaEmbed) {\n",
       "      vegaEmbed(outputDiv, spec, embedOpt)\n",
       "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
       "    }\n",
       "\n",
       "    if(typeof define === \"function\" && define.amd) {\n",
       "      requirejs.config({paths});\n",
       "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
       "    } else if (typeof vegaEmbed === \"function\") {\n",
       "      displayChart(vegaEmbed);\n",
       "    } else {\n",
       "      loadScript(\"vega\")\n",
       "        .then(() => loadScript(\"vega-lite\"))\n",
       "        .then(() => loadScript(\"vega-embed\"))\n",
       "        .catch(showError)\n",
       "        .then(() => displayChart(vegaEmbed));\n",
       "    }\n",
       "  })({\"config\": {\"view\": {\"continuousWidth\": 400, \"continuousHeight\": 300}}, \"data\": {\"name\": \"data-ec8f9ced4c18e7578186da469aefcf84\"}, \"mark\": {\"type\": \"circle\", \"size\": 60}, \"encoding\": {\"tooltip\": [{\"type\": \"quantitative\", \"field\": \"Alpha\"}, {\"type\": \"quantitative\", \"field\": \"BIC\"}], \"x\": {\"type\": \"quantitative\", \"field\": \"Alpha\"}, \"y\": {\"type\": \"quantitative\", \"field\": \"BIC\"}}, \"selection\": {\"selector004\": {\"type\": \"interval\", \"bind\": \"scales\", \"encodings\": [\"x\", \"y\"]}}, \"title\": \"BIC For Lasso\", \"$schema\": \"https://vega.github.io/schema/vega-lite/v4.8.1.json\", \"datasets\": {\"data-ec8f9ced4c18e7578186da469aefcf84\": [{\"Alpha\": 0.01, \"BIC\": 1107.4329999176139, \"Metric\": \"BIC\"}, {\"Alpha\": 0.020555555555555556, \"BIC\": 1124.0549242577576, \"Metric\": \"BIC\"}, {\"Alpha\": 0.03111111111111111, \"BIC\": 1129.1392137700568, \"Metric\": \"BIC\"}, {\"Alpha\": 0.04166666666666667, \"BIC\": 1132.1969043363788, \"Metric\": \"BIC\"}, {\"Alpha\": 0.052222222222222225, \"BIC\": 1142.8820223765993, \"Metric\": \"BIC\"}, {\"Alpha\": 0.06277777777777778, \"BIC\": 1149.2606464366422, \"Metric\": \"BIC\"}, {\"Alpha\": 0.07333333333333333, \"BIC\": 1162.2484496698721, \"Metric\": \"BIC\"}, {\"Alpha\": 0.08388888888888889, \"BIC\": 1176.6562506525859, \"Metric\": \"BIC\"}, {\"Alpha\": 0.09444444444444444, \"BIC\": 1183.4720703323494, \"Metric\": \"BIC\"}, {\"Alpha\": 0.105, \"BIC\": 1195.7501121345247, \"Metric\": \"BIC\"}, {\"Alpha\": 0.11555555555555555, \"BIC\": 1208.8272038159166, \"Metric\": \"BIC\"}, {\"Alpha\": 0.12611111111111112, \"BIC\": 1222.5992708802137, \"Metric\": \"BIC\"}, {\"Alpha\": 0.1366666666666667, \"BIC\": 1236.9656129984467, \"Metric\": \"BIC\"}, {\"Alpha\": 0.14722222222222223, \"BIC\": 1251.83063096735, \"Metric\": \"BIC\"}, {\"Alpha\": 0.1577777777777778, \"BIC\": 1267.1071200992585, \"Metric\": \"BIC\"}, {\"Alpha\": 0.16833333333333333, \"BIC\": 1274.7111901902106, \"Metric\": \"BIC\"}, {\"Alpha\": 0.1788888888888889, \"BIC\": 1287.3439778749503, \"Metric\": \"BIC\"}, {\"Alpha\": 0.18944444444444447, \"BIC\": 1300.2496906329213, \"Metric\": \"BIC\"}, {\"Alpha\": 0.2, \"BIC\": 1313.381068751946, \"Metric\": \"BIC\"}]}}, {\"mode\": \"vega-lite\"});\n",
       "</script>"
      ],
      "text/plain": [
       "alt.Chart(...)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plot_data = _build_scatter_data(alphas, bics, 'BIC')\n",
    "    \n",
    "source = pd.DataFrame(plot_data, columns=['Alpha', 'BIC', 'Metric'])\n",
    "\n",
    "chart = alt.Chart(source, title='BIC For Lasso').mark_circle(size=60).encode(\n",
    "    x='Alpha',\n",
    "    y='BIC',\n",
    "    tooltip=['Alpha', 'BIC']\n",
    ").interactive()\n",
    "\n",
    "chart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 7.8 - Minimum Description Length\n",
    "* You can use concepts from messaging coding/compression theory to derive BIC in a different manner. From a high level view it reflects the minimum amount of encoding necessary to describe your data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 7.9 - Vapnik-Chervonenkis Dimension\n",
    "* In-sample error estimates need a complexity/degrees of freedom parameter, and this can not be estimated for all non-linear models using effective # of parameter methods from 7.6.  The VC theory provides such a general measure of complexity and gives a bound on optimism.\n",
    "* Way of measuring the complexity of a class of functions by assessing how \"wiggly\" its memberes can be. For example this is important when determining how accurately a boundary function can separate classes.\n",
    "* VC is defined to be the largest number of points that can be shattered by the class of functions.\n",
    "* Vapnik's structural risk minimization (SRM) approach fits a nested sequence of models of increasing VC dimensions and then chooses the model with the smallest value for upper bound of optimism."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 7.10 - Cross Validation\n",
    "* Directly estimates the extra-sample error (the average generalization error).\n",
    "* K-Fold - Split into K equal sized samples and train on K-1 of the samples and predict on the remaining sample. Repeat this K times. Combine the K estimates.\n",
    "* If K = N, the number of rows, this is known as *leave-one-out* cross validation.\n",
    "* Cross validation only effectively estimates the average error (generalization error).\n",
    "* 5 or 10 fold validation is recommended as the right tradeoff between bias, variance, and computational constraints.\n",
    "* Generalized cross-validation technique gives an approximation to leave-one-out method.\n",
    "* Be careful of order of operations when doing cross validation. For example, if selecting variables based on correlation w/ response, make sure it is not done before the K-Fold validation. In general, CV must be applied to ALL modeling steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import KFold\n",
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn.metrics import auc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "kf = KFold(n_splits=5, shuffle=True, random_state=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold: 1 \n",
      " Train MSE: 62.150713996905935, Test MSE: 94.6636065833333 \n",
      " Train Variance: 6.890965279009102, Test Variance: 5.3116149893310265\n",
      "\n",
      "Fold: 2 \n",
      " Train MSE: 72.81067819691359, Test MSE: 59.03369190841583 \n",
      " Train Variance: 6.015169766660568, Test Variance: 6.030122304234878\n",
      "\n",
      "Fold: 3 \n",
      " Train MSE: 67.72803781481481, Test MSE: 74.01915098762376 \n",
      " Train Variance: 7.674340321969212, Test Variance: 7.904632889177528\n",
      "\n",
      "Fold: 4 \n",
      " Train MSE: 73.47340186172839, Test MSE: 58.45413314603961 \n",
      " Train Variance: 5.4428992285871045, Test Variance: 5.957576775855306\n",
      "\n",
      "Fold: 5 \n",
      " Train MSE: 69.77699731604939, Test MSE: 66.16521578960396 \n",
      " Train Variance: 6.981706877177259, Test Variance: 7.537848432261543\n",
      "\n"
     ]
    }
   ],
   "source": [
    "X = boston.data\n",
    "y = boston.target\n",
    "\n",
    "model_objects = []\n",
    "\n",
    "fold = 1\n",
    "for train_index, test_index in kf.split(X):\n",
    "    X_train, X_test = X[train_index], X[test_index]\n",
    "    y_train, y_test = y[train_index], y[test_index]\n",
    "    \n",
    "    knn = KNeighborsRegressor(n_neighbors=200, weights='uniform', algorithm='auto', leaf_size=30, p=5, \n",
    "                              metric='minkowski', metric_params=None, n_jobs=-1)\n",
    "    \n",
    "    knn.fit(X_train, y_train)\n",
    "    model_objects.append(knn)\n",
    "    train_predictions = knn.predict(X_train)\n",
    "    test_predictions = knn.predict(X_test)\n",
    "    \n",
    "    print('Fold: {} \\n Train MSE: {}, Test MSE: {} \\n Train Variance: {}, Test Variance: {}\\n'.format(\n",
    "        fold,\n",
    "        mean_squared_error(y_train, train_predictions),\n",
    "        mean_squared_error(y_test, test_predictions),\n",
    "        train_predictions.var(),\n",
    "        test_predictions.var()\n",
    "    ))\n",
    "    \n",
    "    fold += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = []\n",
    "\n",
    "for knn in model_objects:\n",
    "    predictions.append(knn.predict(X))\n",
    "    \n",
    "ensembled_predictions = np.array(predictions).mean(axis = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE: 69.31600944288537, Variance: 6.566750649521784\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print('MSE: {}, Variance: {}\\n'.format(\n",
    "    mean_squared_error(y, ensembled_predictions),\n",
    "    ensembled_predictions.var()\n",
    "))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=None, error_score=nan,\n",
       "             estimator=KNeighborsRegressor(algorithm='auto', leaf_size=30,\n",
       "                                           metric='minkowski',\n",
       "                                           metric_params=None, n_jobs=None,\n",
       "                                           n_neighbors=5, p=2,\n",
       "                                           weights='uniform'),\n",
       "             iid='deprecated', n_jobs=None,\n",
       "             param_grid={'n_neighbors': range(1, 200)}, pre_dispatch='2*n_jobs',\n",
       "             refit=True, return_train_score=False, scoring=None, verbose=0)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn = KNeighborsRegressor()\n",
    "gsc = GridSearchCV(estimator=knn, param_grid = {'n_neighbors': range(1, 200)})\n",
    "gsc.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_knn = gsc.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "58"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_knn.n_neighbors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.1630466679885652"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsc.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 7.11 - Bootstrap Methods\n",
    "* Randomly draw datasets with replacement from the training data, each size the same as the original training set. Do this B times.\n",
    "* Then, refit the model to each of the bootstrap datasets and examine the fits over the B replications.\n",
    "* Approaches for estimating error:\n",
    "    * Predict how well it fits the original non-bootstrapped training set and keep track of the error (average each bootstrap model error). This doesn't provide a great estimate because of its tendency to favor overfit models.\n",
    "    * Keep track of predictions only from bootstrap samples not containing that prediction. That is, when predicting on the training set only use data points that do not exist in the bootstrap sample. This is like CV and called the leave-one-out bootstrap method. Its training set bias will behave like two-fold CV. An unbiased version is 0.368 * training error + 0.632 * leave_one_out_error. This formula has a complicated derivation.\n",
    "* For many adaptive, non-linear methods, estimators like AIC are impractical, leaving us with CV and bootstrap.\n",
    "* **With methods like fitting trees, cross-validation and bootstrap can understimate the true error by 10%, because the search for best tree is strongly affected by the validation set. In these situations only a separate test set will provide an unbiased estimate of the test error.**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.utils import resample\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample: 1 \n",
      " MSE: 58.46088671936759, Variance: 18.71067635307535\n",
      "\n",
      "Sample: 2 \n",
      " MSE: 57.20755137549407, Variance: 19.748870403115188\n",
      "\n",
      "Sample: 3 \n",
      " MSE: 56.701366, Variance: 16.487710413301258\n",
      "\n",
      "Sample: 4 \n",
      " MSE: 56.85304129644268, Variance: 22.339110184505305\n",
      "\n",
      "Sample: 5 \n",
      " MSE: 60.155956079051386, Variance: 24.089078261931913\n",
      "\n",
      "Sample: 6 \n",
      " MSE: 57.67518415019763, Variance: 26.058396206486584\n",
      "\n",
      "Sample: 7 \n",
      " MSE: 57.41876413438735, Variance: 20.010678020231527\n",
      "\n",
      "Sample: 8 \n",
      " MSE: 57.82781815019762, Variance: 23.491621240013124\n",
      "\n",
      "Sample: 9 \n",
      " MSE: 59.29024582608696, Variance: 21.558376858894846\n",
      "\n",
      "Sample: 10 \n",
      " MSE: 57.91658765217392, Variance: 17.722285617397553\n",
      "\n",
      "Avg. MSE: 57.950740138339924\n"
     ]
    }
   ],
   "source": [
    "B = 10\n",
    "errors = []\n",
    "\n",
    "for i in range(B):\n",
    "    X_bootstrap, y_bootstrap = resample(X, y)\n",
    "    \n",
    "    knn = KNeighborsRegressor(n_neighbors=50, weights='uniform', algorithm='auto', leaf_size=30, p=2, \n",
    "                          metric='minkowski', metric_params=None, n_jobs=-1)\n",
    "    \n",
    "    knn.fit(X_bootstrap, y_bootstrap)\n",
    "    predictions = knn.predict(X)\n",
    "    e = mean_squared_error(y, predictions)\n",
    "    errors.append(e)\n",
    "    \n",
    "    print('Sample: {} \\n MSE: {}, Variance: {}\\n'.format(\n",
    "        i+1,\n",
    "        e,\n",
    "        predictions.var()\n",
    "    ))\n",
    "    \n",
    "print('Avg. MSE: {}'.format(np.array(errors).mean()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TO DO - implement error estimate for samples not in bootstrap (leave-one-out bootstrap)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 7.12 - Conditional or Expected Test Error\n",
    "* K-Fold validation does not estimate training error well.\n",
    "* K-Fold validation DOES estimate expected error (generalization error) well and in an unbiased manner, even though the variation in test error for different training sets (folds) is substantial.\n",
    "* Estimation of test error for a particular training set is not easy in general, given just the data from that same training set. Instead, CV and related methods may provide reasonable estimates of the *expected* error."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}