4_logistic_regression/3-PCA_and_Logistic_Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 主成分分析和逻辑回归"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "主成分分析所做的是无监督下的降维工作，而逻辑回归是预测的工作。\n",
    "\n",
    "我们用GridSearchCV设置PCA的维数。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQAAAADFCAYAAAC7ICzVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAAAnn0lEQVR4nO2deXhV1bXAfys3N/NEyAAJIZEwRHBARUCQklZrtbVarCIoVq2I9FVr7Xut03s4tQ5tfbWDT6soqLziVC2oLQURrD4GBUGQIYBACAmQgSFzcpOs98c5CQlkOteEm3vv/n3f+e45++x19tp3WHcPa68tqorBYAhOQnytgMFg8B3GABgMQYwxAAZDEGMMgMEQxBgDYDAEMcYAGAxBTKivFfiqJCUlaVZWlq/VMBj6NOvXry9V1eQT0/3eAGRlZbFu3Tpfq2Ew9GlEJL+9dNMFMBgCgBc/3sOLH+9xLGcMgMEQAKz6spRVX5Y6lvP7LoDBYIC5N57vlVxQtACq6xswax4MhpMJ+BbArJfX8cH2Ypb8dBJDU2I7zVteXk5xcTEej+cUaWcw9AyHaxrZVxPK9Ik5juQC3gCEhYbQ0KSs2X24UwNQXl7OoUOHSE9PJzIyEhE5hVoaDN6jquw+eBiKiigvLycuLq7bsgHfBRg3pD8Aa3aXdZqvuLiY9PR0oqKizI/f4FeICNkD+3PO6dkUFxc7kg14A3DBkEQA1u453Ok4gMfjITIy8lSpZTD0OJGRkY67rwFvALKTY0iKCaOkoo7dpVWd5jX//AZ/pbiilpLKOsdyAW8ARIRxp1ndgLW7D/tYG4O/k5uby9y5c32txknU1jdSW9/oWC7gDQDAuJZuQOfjAIbAYeXKlQwaNMjXapwyBvePZnD/aMdyQWEAxrcaCDT+AAbDcYLCAAxLiSExOoxD5XXkl1X7Wh2vyMrK4re//S1nnXUW8fHxXHvttdTW1nYpt2jRIkaPHk1cXBzZ2dksWbIEgKKiIq644goSExMZOnQozz//fIvMgw8+yDXXXMOMGTOIjY3lzDPPZMeOHTz22GOkpKSQkZHB0qVLW/Ln5uZy7733MnbsWOLi4rjyyis5fPh4d2vx4sWMGjWKhIQEcnNz2bZtW7fr9e677zJ69GgSEhKYMGECmzZt6lK2qqqKyy67jKKiImJiYoiJiaGoqIhPPvmEMWPGEBcXR2pqKj/72c8cv28A+fn5TJw4kdjYWC655BJKS4+74HZW1yeeeIL09HRiY2MZMWIEy5cvB6CpqYnHH3+c7Oxs+vfvz9SpU1vev7179yIivPTSSwwePJikpCR+9atfnaTvofJaDpV3/X04CVX16+O8887T7nDby+s08+53deHa/Hbvb926tVvP8RWZmZl6/vnna2FhoZaVlWlOTo4+88wzncqsXbtW4+LidOnSpdrY2Kj79+/Xbdu2qarqpEmT9Ec/+pHW1NTohg0bNCkpSZcvX66qqg888ICGh4frkiVL1OPx6A033KBZWVn6y1/+Uuvr6/W5557TrKyslnImT56saWlpunnzZq2srNSrrrpKr7/+elVVzcvL06ioKF26dKnW19frE088odnZ2VpXV9dlvT777DNNTk7WNWvWaENDg86fP18zMzO1tra2S9kVK1Zoenp6m/dj/Pjx+vLLL6uqakVFha5evdrx+zZ58mQdMmSI5uXlaXV1tU6ePFnvvvvuLuu6fft2HTRokBYWFqqq6p49e3TXrl2qqvrUU0/puHHjtKCgQGtra3XWrFk6bdq0lnyAzpw5U6urq3Xjxo0aFhZ20vc1v6xK88uqOvweA+u0nd+Pz3/AX/XorgGY9/Fuzbz7Xf3pqxvavd/eGzf12VX6+qf7VFW1vqFRpz67St/6rEBVVavrGnTqs6t08UbrAz1WU69Tn12l/9hcpKqqZZV1OvXZVbpsy0FVVT1UXqNTn12lK7YfUlXVwiPV3dK7mczMTH3llVdarn/+85/rbbfd1qnMrFmz9Kc//elJ6fv27dOQkBAtLy9vSbvnnnv0xhtvVFXLAFx88cUt9xYvXqzR0dHa0NCgqqrl5eUK6JEjR1RV2/wIVFW3bNmibrdbGxoa9OGHH9Zrrrmm5V5jY6OmpaXpihUruqzX7Nmz9T//8z/b6D58+HBduXJll7LtGYBJkybpnDlztKSkpJN3reP3rbmujzzySMv1008/rd/61rdUVTut686dOzU5OVmXLVum9fX1bZ6Zk5Oj77//fst1UVGRhoaGqsfjaTEABQUFLffPP/98XbhwYbv6OTUAQdEFgLYOQeqn4wADBgxoOY+KiqKysrLT/AUFBWRnZ5+UXlRURGJiIrGxxz0jMzMzKSwsbLlOTU1tOY+MjCQpKQmXy9VyDbQpPyMjo82zPB4PpaWlFBUVkZmZ2XIvJCSEjIyMNmV1VK/8/HyefPJJEhISWo6CggKKioq8ek9eeOEFduzYQU5ODueffz7vvvtuu/k6et+6KrOzug4dOpSnnnqKBx98kJSUFKZNm9ZSj/z8fKZMmdJSx9NPPx2Xy8WhQ4e8qqcTAt4VuJkRqbEkRLk5cKyW/UdqyEiM6lLmtdsuaDl3u0LaXEeGudpcx0W421wnRoe1uU6JjWhznZbQ+05HGRkZfPnllyelp6WlcfjwYSoqKlqMwL59+0hPT/e6rIKCgpbzffv24Xa7SUpKIi0tjc2bN7fcU1UKCgq6VVZGRgb3338/999/v2N92vPpGDZsGAsXLqSpqYm33nqLq6++mrKyMqKj246ed/S+dUVXdb3uuuu47rrrKC8v57bbbuPuu+/mlVdeISMjgxdffJGJEyee9My9e/d2q+yDx7zo/xMkg4AAISHCWYMSANh6oNy3ypwibrnlFubNm8fy5ctpamqisLCQ7du3k5GRwYQJE7j33nupra1l06ZNvPDCC8yYMcPrshYsWMDWrVuprq5mzpw5XH311bhcLqZOncp7773H8uXL8Xg8PPnkk4SHhzNhwoQun3nrrbfy7LPPsnbtWlSVqqoq3nvvPSoqKrqUTU1NpaysjGPHjrXRsaSkhJCQEBISEgDrX/pEOnrfuqKzuubl5fHBBx9QV1dHREQEkZGRLWXPnj2b+++/n/x8K2hPSUkJixYt6rK81ngam/A0NjmSgSAyAAAjUmMAyDvY9RcoEBg7dizz5s3jrrvuIj4+nsmTJ7d8yRYuXMjevXtJS0tjypQpPPTQQ1x88cVel3XDDTdw0003MWDAAGpra/nDH/4AwIgRI1iwYAF33HEHSUlJvPPOO7zzzjuEhYV1+cwxY8bw/PPPc/vtt9OvXz+GDh3K/Pnzu6VPTk4O06dPZ8iQISQkJFBUVMSSJUsYNWoUMTEx3Hnnnbz66qvtun939r51Rmd1raur45577iEpKYkBAwZQXFzMY489BsCdd97JFVdcwSWXXEJsbCzjx49n7dq13apnMxmJUd1q1Z6I+Gt/uJkxY8Zod2MCvrGugJ+/uYnvnDWQp687t829bdu2cfrpp/eGigFPbm4uM2bMYObMmb5WJejp6HssIutVdcyJ6UEzBgCQM8BaJrkjSFoAhuDhwLEar+SCqgswNCUGEdhTWkVdg3O/6b7Io48+2uLs0vq47LLLfK2a4RTS2KQ0NjlvzQdVCyAyzEVmYhR7y6rZU1rV0iLwZ+677z7uu+8+n+qwcuVKn5ZvgEH9rP7/toPO5Hq1BSAiL4pIsYh80SrtQREpFJGN9vHtVvfuFZFdIpInIt/qDZ1GDLCmvYJlINBg6IweMwAi0q+d5PnApe2k/05VR9vH3235kcA0YJQt8z8i4uop/ZoZkdqxAfD3AVFD8HLgaA1FR5yvc+nJFsDyExNU9V9AdxfhXwm8qqp1qroH2AWM7UH9ABhutwB2HGprANxuNzU13g2kGAy+pkmhvq4Ot9vtSK4nDYCTcDq3i8gmu4vQ3HJIBwpa5dlvp/UoLS2AEwxASkoKhYWFVFdXm5aAwa9QVfqFK56KUlJSUhzJ9uQgYHd/Nc8Aj9j5HwGeBH7opCARmQXMAhg8eLATUbKSoglzhVBwuIbKugZiwq23oDmSalFRkQkLbvA73G43qampjiICgw9mAVS1ZYWDiDwPNK/IKAQyWmUdZKe194zngOfAcgRyUr7bFcKQ5Gi2H6xg56EKzhl8fOgiLi7O8RtoMPQFHnpnC2wr4IHvjnIkd8q7ACIysNXlFKB5hmAxME1EwkXkNGAY8EkP6tfCiA7GAQyGYMNRC0BELgSGqeo8EUkGYuwBO4CL2sm/EMgFkkRkP/AAkCsio7G6AHuB2wBUdYuIvA5sBRqAH6tqr3jrDG+ZCeiZJZUGg69x+s/fTLcNgIg8AIwBRgDzADewAJgIoKonjfar6vR2HvVCR2Wo6q+Ak+Md9TA5pgVgMADOugBTgCuAKgBVLQI632yvj9LcAthunIEMAcJ//e0L/utvX3Sd8QScGIB6O7SQAoiI8xjEfYT0hEiiw1yUVtZxtLre1+oYDF+ZCHcIEW7nQ3pOxgBeF5E/AwkicivW1N3zXcj0SUJChIEJkewqruRQeR0JUV2vTTcY+jL3f2ekV3LdNhmq+lvgTeCvWOMAc1T1j16V2gdIirF+9CUVzrdTMhgCBSeDgKcBH6nqMvs6UkSyVHVvbynXmyTHRgBQUuldLDWDoS9x71vWfgmPXXWWIzknnYY3gNZBxxrtNL8kOSYcMC0AQ2CQEBXmVVfWyRhAqKq2jJipar2I+G3nOTnWMgCllWYQ0OD/3H1pjldyTloAJSJyRfOFiFwJlHaSv0/TbABMC8AQzDhpAcwG/ldE/oTl9lsA/KBXtDoFGANgCCT+443PAfjtNWc7kuu2AVDVL4HxIhJjX/u1H62ZBTAEEmnxEV7JOZkFCAe+D2QBoc07r6jqw16V7GNaWgCVxgAY/J+fXTLCKzknXYBFwDFgPeD3v5r+0eGECByprsfT2ITbFVQBkg0GwJkBGKSq7cX380tcIUJidDillXUcrqonNc67JpTB0Bf46asbAHhq2jmO5Jz87a0SkTMdPb2PYwYCDYHCkOQYhiTHOJZz0gK4ELhJRPZgdQEEUFV15nrUh0iODWfbAWMADP7PTy4a5pWcEwMQcFvNmJkAQ7DjZBowH0BEUoCA6DCbmQBDoHD7Xz4D4E8nbHrbFU6mAa/AiuCbBhQDmcA2rI08/BKzHsAQKIxM8y6YrZMuwCPAeOB9VT1HRL4OzPCq1D6CaQEYAoV/yx3qlZyTWQCPqpYBISISoqorsGIE+i1mFsAQ7DhpARy13YD/hbUmoBg7PqC/ktK8ItAYAIOfM/uV9QA8e8N5juScGIArgVrgLuB6IB7wSzfgZpLMGIAhQDg3M8ErOSezAK3/7V/yqrQ+RnykG7dLqKhroNbTSIS7xzcjNhhOCbO+lu2VXJdjACLysf1aISLlrY4KESn3qtQ+goiYmQBDUNOlAVDVC+3XWFWNa3XEqqrfb6RnZgIMgcDMlz5l5kufOpbrVhdARFzAFlX1Lu5QH8bMBBgCgQnZSV7JdcsAqGqjiOSJyGBV3edVSX0UYwAMgcAPLzzNKzknswD9gC0i8gmtpv9U9YqORfo+ZibAEMw4MQD/1Wta+BAzBmAIBG588RMAXvrhWEdyTqYBP3Smkn/QPAtgnIEM/szFp6d4JddtV2ARGS8in4pIpYjUi0hjV9OAIvKiiBSLyBet0hJFZJmI7LRf+9npIiJ/EJFdIrJJRJwta/IS0wIwBAI3XJDFDRdkOZZzshbgT8B0YCcQCcwEnu5CZj5wYhixe4DlqjoMWG5fgxVvYJh9zAKecaCb15hBQEMw4ygSpqruAlyq2qiq8zj5x31i/n8Bh09IvpLjnoQvAd9rlf6yWqzB2oV4oBP9vKG1AbB2PzcY/I/r567h+rlrHMs5GQSstrcC2ygivwYO4NCA2KSq6gH7/CCQap+nY2020sx+O+0AvUhUWCixEaFU1DZwpNpDYrTf7nZmCGIuPyvNKzknP+Ab7Py3Y00DZmDtE+A1av3lOv7bFZFZIrJORNaVlJR8FRUASE+IBKDwSM1XfpbB4Aumjx3M9LGDHcs5MQDnYf1my1X1IVX9md0lcMqh5qa9/VpspxdiGZVmBtlpJ6Gqz6nqGFUdk5yc7IUKbWkxAEeNATAEF04MwHeBHSLyiohcLiJOug+tWQzcaJ/fiLXhSHP6D+zZgPHAsVZdhV4lvZ8xAAb/5to/r+baP692LOfED+BmEXFjjdZPB54WkWWqOrMjGRFZCOQCSSKyH3gAeBx4XURuAfKBqXb2vwPfBnYB1cDNjmvjJWmmC2Dwc64+b5BXco7+xVXVIyL/wOq3R2KN4HdoAFR1ege3LmonrwI/dqJPT9HcBSgyLQCDn3LNmIyuM7WDE0egy0RkPpYfwPeBucAAr0rtY5gugMHf8TQ24WlsciznpAXwA+A14DZVDSivmUFmENDg58yYuxaA1267wJGckzGAjprzAIjIalV1VnofISkmnDBXCIer6qmubyAqzNvxTYPBN0wb610XoCe/6X67W1BIiDAwIYL8smqKjtYyNMX5JosGgy+Zco53g4DeePJ1hF/70RpfAIM/U1PfSE19o2O5njQAfo2ZCjT4MzfN+4Sb5n3iWK4nuwDSg8865ZipQIM/M2N8pldyPWkAbujBZ51yzFSgwZ/57tneLQbq0gCISAWd9O+bQ4Or6hcd5fEHzIIggz9TXusBIC7C7UiuSwOgqrEAIvII1tLcV7Ca+9cDvb5e/1RhBgEN/sytL60DetEPALhCVc9udf2MiHwOzHFUYh9lYII1i3mwvJaGxiZCXWZ81OA/3Dwxyys5J9/yKhG5XkRcIhIiItfj57sDtyY81EVKbDiNTcohEx7M4GdcesZALj3DeYPciQG4Dmvl3iH7uMZOCxjMVKDBXzlcVc/hqnrHck5cgfdixe0LWNL7RbKx4KiZCjT4HT9asB7oxTEAERmOFak3VVXPEJGzsMYFfumoxD6MWRRk8FdunTTEKzknXYDngXsBD4CqbgKmeVVqH6W5C7DfdAEMfsbFI1O5eGRq1xlPwIkBiFLVE30NGxyX2IcxU4EGf6W4opbiilrHck6mAUtFJBvbKUhErqaXQ3afalq8AY9U+1gTg8EZd/xlA9C7fgA/Bp4DckSkENgDzHBUWh8ns38UYaEh7C6t4sCxGgbGR/paJYOhW/woN9sruW53AVR1t6peDCQDOap6oT0zEDBEhYVy8ekpqMLfNhT5Wh2Dodvkjkghd4TzDUKdxAQMF5HrgDuBu0RkjogEhBdga5oDK7y9Yb/ZKszgNxQdrfFq+trJIOAiLD+ABiwPwOYjoJg8PJl+UW52HKpkS1Gnmx8bDH2Gu17byF2vbXQs52QMYJCqdroZaCAQFhrCd89O4+XV+by9oZAz0uN9rZLB0CV3fGOYV3JOWgCrRORMr0rxM6461+oGLNpYRIMXoZYNhlPNhcOSuHBYkmM5JwbgQmC9iOSJyCYR2SwimxyX6AecPSieIUnRlFbW8dGuUl+rYzB0yb6yavaVOZ++dmIALgOGAZdg7RN4uf0acIgIU85JB+Cv6/f7WBuDoWt+/ubn/PzNzx3LdWkARCTOPq3o4AhIppybjitE+McXB/mypNLX6hgMnXLXN4dz1zeHO5brTgvgL/bremCd/bq+1XVAMqhfFFPHZNDYpPz3sh2+Vsdg6JTxQ/ozfkh/x3JdGgBVvdx+PU1Vh9ivzYd3S5D8hJ9cNJSw0BDe23SALwqP+Vodg6FDviyp9Kql6ijulYj0E5GxIvK15sNxiceftdceSNwoIuvstEQRWSYiO+3Xft4+vycYGB/JD+xwy08uzfOlKgZDp9z31mbue2uzYzknnoAzgX8B/wQesl8fdFxiW76uqqNVdYx9fQ+wXFWHAcvta5/yb18fSnSYixV5JXy697Cv1TEY2uUXl47gF5eOcCznpAVwJ3A+kK+qXwfOAY46LrFzrgRess9fAr7Xw893TGJ0GDPtYAumFWDoq5yXmch5mYmO5ZwYgFpVrQVrXYCqbgecm5zjKLBURNaLyCw7LVVVm5cYHwScRzjoBW6ZdBqxEaGs2X2Y9fmmFWDoe+QdrCDvoPNJOScGYL+IJAB/A5aJyCIg33GJx7lQVc/F8i/48YnjCWqtxGl3NY6IzBKRdSKyrqSk5Cuo0D3iItzcNCELgD99sKvXyzMYnDJn0RfMWeR8bx7xZsWbiEwG4oElquo8FOnJz3sQqARuBXJV9YCIDARWqmqnrYwxY8bounW9Pxt5uKqeiY9/QI2nkXfvuNCsETD0KT4vOArA2RkJ7d4XkfWtxtpa6I4jUOKJB7AZ+BiI8UZZEYkWkeYdh6KxvAu/ABYDN9rZbsRagdgnSIwOY8b4wQA8vcK0Agx9i7MzEjr88XdGd1YDrsdqire3+68C3vgCpAJvi0izDn9R1SUi8inwuojcgtW9mOrFs3uNWycN4aXV+SzZcpBdxRUMTYn1tUoGAwBbiiw/lVFpzlqm3dkb8DTvVOr0mbuBs9tJLwMu6unyeoqUuAimjhnEgjX7+I83NvHyLWMdb8ZoMPQGD7+zFejdmICIyFVYqwIV+EhV/+aotADgJ98YxortJWwsOMqMuWt55YfjiI8yRsDgW+Z8d6RXck4cgf4HmI3V//8CmC0iT3tVqh+TEhfBa7eNZ3BiFJv2H2P682u82pLJYOhJRqXFO27+g7NpwG8A31LVeao6D/i2nRZ0DOoXxWu3jee0pGi2Hijn+8+sYm9pwEVHM/gRnxccbZkJcIITA7ALGNzqOsNOC0oGxkfy2qzxjBwYx57SKq56ZhXr84/4Wi1DkPLo37fx6N+3OZZzYgBigW0islJEVgBbgTgRWSwiix2XHACkxEXw+uwLmDw8mcNV9Vz3/BpjBAw+4eErz+DhK89wLNdtRyDb+adDVPVDx6X3AKfKEagzGhqb+MVfN/HWZ4VMGpbEK7eM86k+BsOJdOQI5GQWoERVt57w0FxVXflVlfN3Ql0hPHD5KP6x+SAf7Swl72AFIwYYHwHDqaN5jYrTBUFOugCvi8gvxCJSRP4IPOaotAAmPsrN1DFWNOEXPt7tY20Mwcavl+Tx6yXOV6s6MQDjsAYBVwGfAkXARMclBjA3TzwNEWtbsZKKOl+rYwgiHr3qTB69ynnUficGwAPUAJFABLBHVU3Q/FZkJUXzzdNTqW9s4pXVe32tjiGIyE6OITvZ+dIcJwbgUywDMAaYBEwXkTcclxjgNAcPeWVNvmkFGE4Za3aXsWZ3mWM5JwbgVmAncJ8dtOMOwHkg8gDn/Kx+nD0oniPVHsY9+j7X/nk1f1m7j6Yms9Gooff43bId/M6L6NVODMDNwHhgun1dgRXCy9AKEeG/rx3N5OHJuEKEtXsOc9/bm3nuIzMwaOg9fnP12fzm6pPW13WJk2nAcap6rohsAFDVIyJiVsG0Q3ZyDC/9cCzltR7e/qyQBxZv4cmleUwaluSVv7bB0BWD+0d5JedoEFBEXNhhukQkmQ5Cdhks4iLc3DghixnjB+NpVO56bSO1nkZfq2UIQD7eWcrHO53vY+nEAPwBeBtIEZFfYUUEetRxiUHI/d8eyZCkaHYcquSRd7dy4FgN3oRiMxg64o8f7OSPH+x0LOcoJqCI5GAF7BCs+P3OVx/0MH3BFbg7fF5wlO8/s4oGezAw0u1i4tD+/Paas0mICvOxdgZ/p+hoDQBpCZHt3u/IFdiroKB9CX8xAADvbTrA/FV72F1SRZkdQ2DkwDgWzBxHYrQxAobewxiAPkbB4WpufPETdpdWMSI1lgUzx5EcG+5rtQx+ysq8YgByR6S0e78nFgMZepCMxChenTWe6+euJe9QBbm/WcHpA+PIGRjLuYP7MXl4Mv1jjEEwdI9nVn4JdGwAOsK0AHxMaWUds15ex2f7jrZJF4GzBiUwIbs/5w3uxzmDE4xBMHRIcUUtACmxEe3eN12APk5pZR3bDpSzpaic/9tVytrdh6lvbLvU4jtnDuTRKWeaIKQGxxgD4GdU1zew+ssy1uUf4bP8I2woOEp9QxPpCZH8ftpoxmQ53wjSELi8v/UQABePbH87TTMG4GdEhYVy0empXHS69YHml1Xxk4Ub+Hz/Ma59bg1Xjk7je6PTmZDdn1CXE3cOQyDyvO1q3pEB6AjTAvAj6huaeHJZHn/+8Pi6gv7RYaT3iyQ6LJSEKDdjT0vkGzkpZPaP9qGmhlNNc2j6jqaTTRcggNhbWsXfNhayaGMRezoIRz44MYpB/SJJjg0nNS6CzP5RZPWPJqNfFEmxYUSFmcZfMGEMQACiquSXVXO0xkNVXQNFR2v4cEcJH+4ooaK2oVPZSLfLDmCSwiWjBjAqLQ57r0aDH7LkiwMAXHrGwHbvGwMQRHgam9hVXElJRR0lFXUcOFbD3rJq8suqKDxSQ2lVPfUNbWcYIt0u4iPdxEe6GdQvklFpcYxMi2dwYhSJ0WH0i3YTHuryUY0MXXHtn1cDHe8NaAyAoQVVpbKugc/2HeWfWw6ybOuhbkUvajYMZ6TFkzMwjpwBsaQnRBISYloOvqa81gPQ4Wa1xgAYOqTZIJTXNnC0up7dJVVsKSpn64FyDh2rpayqniPV9TS2E9UoKsxFYnQY8ZFu4iLcxESEEhMeSlSYi/BQF+5QIdLtIi7CTVykm5hw635kmIuEKDdp8ZFEhpmWRW/jN9OAInIp8HvABcxV1cd9rFLAIyLERriJjXCTnhDJqLR4vnt2Wps8DY1NfFlSxZaiY2wpKifvYAXbD1ZQWllHdX0N+4/UeF1+YnQYA+IiSIoNJykmjNjwUFwhIbhdQlhoCBFuF+GhIS1GJCHKTVJMOKlx4cSEh5qxC+Cdz4sATvrcuqJPGQA74MjTwDeB/cCnIrL4xA1JDKeeUFcIIwbEMmJALFedezz9WI2Ho9X1lNc0cKzGQ2VdA1V1DVTVN1Df0ERdQxO1nkYqahsor/FQXttAjaeBqrpGDlfVc+BYDYer6q1prAPO9YpwhxDpduF2heB2hRDuDiEi1EWU3cJIiLJaJ+GhIYSFhhAe6iLCbRmVqLDmlkkocRFu4qPcdl7/a5EsWJMP+LkBAMYCu1R1N4CIvIoVd9AYgD5K88ChtzQ1KSWVdRwqr6W00hq0rK5vpLFJ8TQq9Q1N1DY0thiRo9WWwSmprKO4vI4aTyO1np6NTh8WGkJ0mItou7sSZ3dvosNdRLpdRLhdhIYIrhBBRGjdABEgxE4TEQRrXYdwPF/r9kpPNF7OGdyP+TeP9Uq2rxmAdKCg1fV+rA1JDAFKSIiQGhdBalz7i1g6Q1Wprm+krqEJT2OT3eKwDEJVXQNHazwcqaqnvNZDfUNTmxZJraeJqvoGKmqtlkt5rYfyGg9Hq4/nPVLt6YUa9zyzvjaEiUOTvJLtawagW4jILGAWwODBg7vIbQhURITo8FCie3CRpKpS12AZkKq6RirqPJTXNFBe66G6voGa+iZqPI00NjXRpLQ7MNrUpDQpKNYrqi3BM1uPuWsPhdQck5XI2xv2AzDlnEGOZPuaASgEMlpdD7LT2qCqzwHPgTULcGpUMwQDIkKE3czv73yjHZ/R7Afg7wbgU2CYiJyG9cOfBlznW5UMhr7Pgpne9ZT7lAFQ1QYRuR34J9Y04IuqusXHahkMfR63lytC+5QBAFDVvwN/97UeBoM/8cY6a+z8mjEZXeRsi1lIbjAEAG+u38+b6/c7lvN7V2ARKQHyu8iWBDjfNsX/CJZ6QvDUtafqmamqyScm+r0B6A4isq49P+hAI1jqCcFT196up+kCGAxBjDEABkMQEywG4DlfK3CKCJZ6QvDUtVfrGRRjAAaDoX2CpQVgMBjaIeANgIhcKiJ5IrJLRO7xtT49hYhkiMgKEdkqIltE5E47PVFElonITvu1n6917QlExCUiG0TkXfv6NBFZa3+ur4mI32+vLCIJIvKmiGwXkW0ickFvf54BbQBaBRi5DBgJTBeRkb7VqsdoAP5dVUcC44Ef23W7B1iuqsOA5fZ1IHAnsK3V9RPA71R1KHAEuMUnWvUsvweWqGoOcDZWfXv381TVgD2AC4B/trq+F7jX13r1Ul0XYUVSygMG2mkDgTxf69YDdRtkf/m/AbyLFVOjFAht73P2xwOIB/Zgj8u1Su/VzzOgWwC0H2Ak3Ue69BoikgWcA6wFUlW1ObjWQcDZXlF9k6eAXwDNoX/6A0dVtXnzg0D4XE8DSoB5dldnrohE08ufZ6AbgIBHRGKAvwI/VdXy1vfU+tvw62keEbkcKFbV9b7WpZcJBc4FnlHVc4AqTmju98bnGegGoFsBRvwVEXFj/fj/V1XfspMPichA+/5AoNhX+vUQE4ErRGQv8CpWN+D3QIKINK9mDYTPdT+wX1XX2tdvYhmEXv08A90AtAQYsUeJpwGLfaxTjyBWLOwXgG2q+t+tbi0GbrTPb8QaG/BbVPVeVR2kqllYn98Hqno9sAK42s4WCPU8CBSIyAg76SKsYLi9+nkGvCOQiHwbqw/ZHGDkV77VqGcQkQuBj4DNHO8b34c1DvA6MBhrleRUVT3sEyV7GBHJBf5DVS8XkSFYLYJEYAMwQ1W73t6oDyMio4G5QBiwG7gZ60+61z7PgDcABoOhYwK9C2AwGDrBGACDIYgxBsBgCGKMATAYghhjAAyGIMYYAINfIyK5IjLB13r4K8YAGPydXMAYAC8xBiAAEJEse/3483ZsgKUiEtlB3qEi8r6IfC4in4lItlj8RkS+EJHNInKtnTdXRD4UkUUisltEHheR60XkEztftp1vvog8KyLrRGSH7b+PiESIyDw77wYR+bqdfpOIvCUiS+x17r9upd8lIrLa1u0Ne60DIrJXRB6y0zeLSI69CGo2cJeIbBSRSSJyjV2Pz0XkX736xgcCvl4GaY6vfgBZWPEBRtvXr2N5xrWXdy0wxT6PAKKA7wPLsLwlU4F9WEtPc4Gj9nk4lr/9Q7bsncBT9vl8YAnWH8owLL/2CODfsbwvAXLs50YAN2F5usXb1/lYazaSgH8B0bbM3cAc+3wvcId9/m/AXPv8QSzvwOb6bQbS7fMEX382ff0wLYDAYY+qbrTP12MZhTaISCzWj+NtAFWtVdVq4EJgoao2quoh4EPgfFvsU1U9oJab7ZfAUjt98wllvK6qTaq6E+vHnWM/d4Fd1nasH/pwO/9yVT2mqrVYPu+ZWIFNRgL/JyIbsXzfM1uV0bzgqd362fwfMF9EbsUyaIZO6HN7Axq8prUffCPQbhfgKz63qdV1E22/Pyf6lHflY36ivqFYgT6Wqer0LmSa85+Eqs4WkXHAd4D1InKeqpZ1oUvQYloAQYSqVgD7ReR7ACISLiJRWIuKrrXj7iUDXwM+cfj4a0QkxB4XGIIVyeYj4Hq7rOFYC1ryOnnGGmCiiAy1ZaJtuc6oAGKbL0QkW1XXquocrAAbznbLDDKMAQg+bgB+IiKbgFXAAOBtYBPwOfAB8Au1lqc6YR+W0fgHMNtu2v8PECIim4HXgJu0kxV7qlqCNT6w0NZvNVZXojPeAaY0DwICv7EHCb+w6/e5w3oEFWY1oOErIyLzgXdV9U1f62JwhmkBGAxBjGkBBCgi8jRWOK3W/F5V5/lCH0PfxBgAgyGIMV0AgyGIMQbAYAhijAEwGIIYYwAMhiDGGACDIYgxBsBgCGL+H+Frsufz/2CmAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 288x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# % matplotlib inline\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn import linear_model, decomposition, datasets\n",
    "from sklearn.manifold import Isomap\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "logistic = linear_model.LogisticRegression()\n",
    "\n",
    "pca = decomposition.PCA()\n",
    "pipe = Pipeline(steps=[('pca', pca), ('logistic', logistic)])\n",
    "\n",
    "digits = datasets.load_digits()\n",
    "X_digits = digits.data\n",
    "y_digits = digits.target\n",
    "\n",
    "# Plot the PCA spectrum\n",
    "pca.fit(X_digits)\n",
    "\n",
    "plt.figure(1, figsize=(4, 3))\n",
    "plt.clf()\n",
    "plt.axes([.2, .2, .7, .7])\n",
    "plt.plot(pca.explained_variance_, linewidth=2)\n",
    "plt.axis('tight')\n",
    "plt.xlabel('n_components')\n",
    "plt.ylabel('explained_variance_')\n",
    "\n",
    "# Prediction\n",
    "n_components = [20, 40, 64]\n",
    "Cs = np.logspace(-4, 4, 3)\n",
    "\n",
    "# Parameters of pipelines can be set using ‘__’ separated parameter names:\n",
    "estimator = GridSearchCV(pipe,\n",
    "                         dict(pca__n_components=n_components,\n",
    "                              logistic__C=Cs))\n",
    "estimator.fit(X_digits, y_digits)\n",
    "\n",
    "plt.axvline(estimator.best_estimator_.named_steps['pca'].n_components,\n",
    "            linestyle=':', label='n_components chosen')\n",
    "plt.legend(prop=dict(size=12))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1797, 64)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/utils/deprecation.py:143: FutureWarning: The sklearn.linear_model.logistic module is  deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.linear_model. Anything that cannot be imported from sklearn.linear_model is now part of the private API.\n",
      "  warnings.warn(message, FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPoAAAECCAYAAADXWsr9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAAAL1UlEQVR4nO3df6hX9R3H8ddrptVS0laL0MiMIUSw/IEsitg0w1a4f5YoFCw29I8tkg3K9s/ov/6K9scIxGpBZqQljNhaSkYMtprXbJnaKDFSKgsNsz+U7L0/vsdhznXPvZ3P537v9/18wBe/997vPe/3vdfX95zz/Z5z3o4IARhs3xrrBgCUR9CBBAg6kABBBxIg6EACBB1IoC+CbnuJ7bdtv2N7TeFaj9k+ZHtXyTqn1bvc9jbbu22/ZfuewvXOs/2a7Teaeg+UrNfUnGD7ddvPl67V1Ntv+03bO21vL1xrqu1Ntvfa3mP7uoK1Zjc/06nbUdurO1l4RIzpTdIESe9KmiVpkqQ3JF1dsN6NkuZK2lXp57tM0tzm/hRJ/y7881nS5Ob+REmvSvpB4Z/x15KekvR8pd/pfkkXV6r1hKRfNPcnSZpaqe4ESR9KuqKL5fXDGn2BpHciYl9EnJD0tKSflCoWEa9IOlxq+Wep90FE7GjufyZpj6TpBetFRBxrPpzY3IodFWV7hqRbJa0rVWOs2L5QvRXDo5IUESci4tNK5RdJejci3utiYf0Q9OmS3j/t4wMqGISxZHumpDnqrWVL1plge6ekQ5K2RETJeg9LulfSlwVrnCkkvWh7yPbKgnWulPSxpMebXZN1ti8oWO90yyVt6Gph/RD0FGxPlvSspNURcbRkrYg4GRHXSpohaYHta0rUsX2bpEMRMVRi+V/jhoiYK+kWSb+0fWOhOueot5v3SETMkfS5pKKvIUmS7UmSlkra2NUy+yHoByVdftrHM5rPDQzbE9UL+fqIeK5W3WYzc5ukJYVKXC9pqe396u1yLbT9ZKFa/xURB5t/D0narN7uXwkHJB04bYtok3rBL+0WSTsi4qOuFtgPQf+npO/ZvrJ5Jlsu6U9j3FNnbFu9fbw9EfFQhXqX2J7a3D9f0mJJe0vUioj7I2JGRMxU7+/2UkTcUaLWKbYvsD3l1H1JN0sq8g5KRHwo6X3bs5tPLZK0u0StM6xQh5vtUm/TZExFxBe2fyXpr+q90vhYRLxVqp7tDZJ+KOli2wck/S4iHi1VT7213p2S3mz2myXptxHx50L1LpP0hO0J6j2RPxMRVd72quRSSZt7z586R9JTEfFCwXp3S1rfrIT2SbqrYK1TT16LJa3qdLnNS/kABlg/bLoDKIygAwkQdCABgg4kQNCBBPoq6IUPZxyzWtSj3ljX66ugS6r5y6z6h6Me9cayXr8FHUABRQ6YsT3QR+FMmzZtxN9z/PhxnXvuuaOqN336yE/mO3z4sC666KJR1Tt6dOTn3Bw7dkyTJ08eVb2DB0d+akNEqDk6bsROnjw5qu8bLyLif34xY34I7Hh00003Va334IMPVq23devWqvXWrCl+QthXHDlypGq9fsCmO5AAQQcSIOhAAgQdSICgAwkQdCABgg4kQNCBBFoFvebIJADdGzbozUUG/6DeJWivlrTC9tWlGwPQnTZr9KojkwB0r03Q04xMAgZVZye1NCfK1z5nF0ALbYLeamRSRKyVtFYa/NNUgfGmzab7QI9MAjIYdo1ee2QSgO612kdv5oSVmhUGoDCOjAMSIOhAAgQdSICgAwkQdCABgg4kQNCBBAg6kACTWkah9uSUWbNmVa03mpFT38Thw4er1lu2bFnVehs3bqxa72xYowMJEHQgAYIOJEDQgQQIOpAAQQcSIOhAAgQdSICgAwkQdCCBNiOZHrN9yPauGg0B6F6bNfofJS0p3AeAgoYNekS8IqnuWQcAOsU+OpAAs9eABDoLOrPXgP7FpjuQQJu31zZI+ruk2bYP2P55+bYAdKnNkMUVNRoBUA6b7kACBB1IgKADCRB0IAGCDiRA0IEECDqQAEEHEhiI2Wvz5s2rWq/2LLSrrrqqar19+/ZVrbdly5aq9Wr/f2H2GoAqCDqQAEEHEiDoQAIEHUiAoAMJEHQgAYIOJEDQgQQIOpBAm4tDXm57m+3dtt+yfU+NxgB0p82x7l9I+k1E7LA9RdKQ7S0RsbtwbwA60mb22gcRsaO5/5mkPZKml24MQHdGtI9ue6akOZJeLdINgCJan6Zqe7KkZyWtjoijZ/k6s9eAPtUq6LYnqhfy9RHx3Nkew+w1oH+1edXdkh6VtCciHirfEoCutdlHv17SnZIW2t7Z3H5cuC8AHWoze+1vklyhFwCFcGQckABBBxIg6EACBB1IgKADCRB0IAGCDiRA0IEEBmL22rRp06rWGxoaqlqv9iy02mr/PjNijQ4kQNCBBAg6kABBBxIg6EACBB1IgKADCRB0IAGCDiRA0IEE2lwF9jzbr9l+o5m99kCNxgB0p82x7sclLYyIY8313f9m+y8R8Y/CvQHoSJurwIakY82HE5sbAxqAcaTVPrrtCbZ3SjokaUtEMHsNGEdaBT0iTkbEtZJmSFpg+5ozH2N7pe3ttrd33COAb2hEr7pHxKeStklacpavrY2I+RExv6PeAHSkzavul9ie2tw/X9JiSXsL9wWgQ21edb9M0hO2J6j3xPBMRDxfti0AXWrzqvu/JM2p0AuAQjgyDkiAoAMJEHQgAYIOJEDQgQQIOpAAQQcSIOhAAsxeG4WtW7dWrTfoav/9jhw5UrVeP2CNDiRA0IEECDqQAEEHEiDoQAIEHUiAoAMJEHQgAYIOJEDQgQRaB70Z4vC6bS4MCYwzI1mj3yNpT6lGAJTTdiTTDEm3SlpXth0AJbRdoz8s6V5JX5ZrBUApbSa13CbpUEQMDfM4Zq8BfarNGv16SUtt75f0tKSFtp8880HMXgP617BBj4j7I2JGRMyUtFzSSxFxR/HOAHSG99GBBEZ0KamIeFnSy0U6AVAMa3QgAYIOJEDQgQQIOpAAQQcSIOhAAgQdSICgAwkMxOy12rO05s2bV7VebbVnodX+fW7cuLFqvX7AGh1IgKADCRB0IAGCDiRA0IEECDqQAEEHEiDoQAIEHUiAoAMJtDoEtrnU82eSTkr6gks6A+PLSI51/1FEfFKsEwDFsOkOJNA26CHpRdtDtleWbAhA99puut8QEQdtf1fSFtt7I+KV0x/QPAHwJAD0oVZr9Ig42Px7SNJmSQvO8hhmrwF9qs001QtsTzl1X9LNknaVbgxAd9psul8qabPtU49/KiJeKNoVgE4NG/SI2Cfp+xV6AVAIb68BCRB0IAGCDiRA0IEECDqQAEEHEiDoQAIEHUjAEdH9Qu3uF/o1Zs2aVbOctm/fXrXeqlWrqta7/fbbq9ar/febP3+wT8eICJ/5OdboQAIEHUiAoAMJEHQgAYIOJEDQgQQIOpAAQQcSIOhAAgQdSKBV0G1Ptb3J9l7be2xfV7oxAN1pO8Dh95JeiIif2p4k6dsFewLQsWGDbvtCSTdK+pkkRcQJSSfKtgWgS2023a+U9LGkx22/bntdM8jhK2yvtL3ddt1TuwAMq03Qz5E0V9IjETFH0ueS1pz5IEYyAf2rTdAPSDoQEa82H29SL/gAxolhgx4RH0p63/bs5lOLJO0u2hWATrV91f1uSeubV9z3SbqrXEsAutYq6BGxUxL73sA4xZFxQAIEHUiAoAMJEHQgAYIOJEDQgQQIOpAAQQcSGIjZa7WtXLmyar377ruvar2hoaGq9ZYtW1a13qBj9hqQFEEHEiDoQAIEHUiAoAMJEHQgAYIOJEDQgQQIOpDAsEG3Pdv2ztNuR22vrtAbgI4Me824iHhb0rWSZHuCpIOSNpdtC0CXRrrpvkjSuxHxXolmAJQx0qAvl7ShRCMAymkd9Oaa7kslbfw/X2f2GtCn2g5wkKRbJO2IiI/O9sWIWCtprTT4p6kC481INt1XiM12YFxqFfRmTPJiSc+VbQdACW1HMn0u6TuFewFQCEfGAQkQdCABgg4kQNCBBAg6kABBBxIg6EACBB1IgKADCZSavfaxpNGcs36xpE86bqcfalGPerXqXRERl5z5ySJBHy3b2yNi/qDVoh71xroem+5AAgQdSKDfgr52QGtRj3pjWq+v9tEBlNFva3QABRB0IAGCDiRA0IEECDqQwH8An6mM7XzL9vMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 288x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Compare the performance\n",
    "from sklearn.datasets import load_digits\n",
    "from sklearn.linear_model.logistic import LogisticRegression\n",
    "from sklearn import decomposition\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import accuracy_score\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "# load digital data\n",
    "digits, dig_label = load_digits(return_X_y=True)\n",
    "print(digits.shape)\n",
    "\n",
    "# draw one digital\n",
    "plt.gray() \n",
    "plt.matshow(digits[0].reshape([8, 8])) \n",
    "plt.show() \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy train = 1.000000, accuracy_test = 0.905556\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# calculate train/test data number\n",
    "N = len(digits)\n",
    "N_train = int(N*0.8)\n",
    "N_test = N - N_train\n",
    "\n",
    "# split train/test data\n",
    "x_train = digits[:N_train, :]\n",
    "y_train = dig_label[:N_train]\n",
    "x_test  = digits[N_train:, :]\n",
    "y_test  = dig_label[N_train:]\n",
    "\n",
    "# do logistic regression\n",
    "lr=LogisticRegression()\n",
    "lr.fit(x_train,y_train)\n",
    "\n",
    "pred_train = lr.predict(x_train)\n",
    "pred_test  = lr.predict(x_test)\n",
    "\n",
    "# calculate train/test accuracy\n",
    "acc_train = accuracy_score(y_train, pred_train)\n",
    "acc_test = accuracy_score(y_test, pred_test)\n",
    "print(\"accuracy train = %f, accuracy_test = %f\" % (acc_train, acc_test))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy train = 1.000000, accuracy_test = 0.897222\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/bushuhui/virtualenv/dl/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n"
     ]
    }
   ],
   "source": [
    "# do PCA with 'n_components=40'\n",
    "pca = decomposition.PCA(n_components=40)\n",
    "pca.fit(x_train)\n",
    "\n",
    "x_train_pca = pca.transform(x_train)\n",
    "x_test_pca = pca.transform(x_test)\n",
    "\n",
    "# do logistic regression\n",
    "lr=LogisticRegression()\n",
    "lr.fit(x_train_pca,y_train)\n",
    "\n",
    "pred_train = lr.predict(x_train_pca)\n",
    "pred_test  = lr.predict(x_test_pca)\n",
    "\n",
    "# calculate train/test accuracy\n",
    "acc_train = accuracy_score(y_train, pred_train)\n",
    "acc_test = accuracy_score(y_test, pred_test)\n",
    "print(\"accuracy train = %f, accuracy_test = %f\" % (acc_train, acc_test))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy train = 0.148921, accuracy_test = 0.102778\n"
     ]
    }
   ],
   "source": [
    "# do kernel PCA\n",
    "#   Ref: http://scikit-learn.org/stable/auto_examples/decomposition/plot_kernel_pca.html\n",
    "\n",
    "from sklearn.decomposition import PCA, KernelPCA\n",
    "\n",
    "kpca = KernelPCA(n_components=45, kernel=\"rbf\", fit_inverse_transform=True, gamma=10)\n",
    "kpca.fit(x_train)\n",
    "\n",
    "x_train_pca = kpca.transform(x_train)\n",
    "x_test_pca = kpca.transform(x_test)\n",
    "\n",
    "# do logistic regression\n",
    "lr=LogisticRegression()\n",
    "lr.fit(x_train_pca,y_train)\n",
    "\n",
    "pred_train = lr.predict(x_train_pca)\n",
    "pred_test  = lr.predict(x_test_pca)\n",
    "\n",
    "# calculate train/test accuracy\n",
    "acc_train = accuracy_score(y_train, pred_train)\n",
    "acc_test = accuracy_score(y_test, pred_test)\n",
    "print(\"accuracy train = %f, accuracy_test = %f\" % (acc_train, acc_test))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "* [Pipelining: chaining a PCA and a logistic regression](http://scikit-learn.org/stable/auto_examples/plot_digits_pipe.html)\n",
    "* [PCA进行无监督降维](https://ljalphabeta.gitbooks.io/python-/content/pca.html)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}