SMS_Spam_Detection_NLP_.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "SMS Spam Detection : NLP .ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyNJUawe/r6XNkbjIbffEqyp",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/MainakRepositor/Data-Analysis/blob/master/SMS_Spam_Detection_NLP_.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EsyQyOF1ccef"
      },
      "source": [
        "# SMS Spam Detection using Natural Language Processing\n",
        "\n",
        "<hr>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vRRCtuLXcqPm"
      },
      "source": [
        "## Approach :\n",
        "### i.  Representing text as numerical data\n",
        "### ii. Reading a text-based dataset into pandas\n",
        "### iii.Vectorizing our dataset\n",
        "### iv. Building and evaluating a model\n",
        "### v.  Comparing models\n",
        "### vi. Examining a model for further insight\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QH0PTBSFdBQe"
      },
      "source": [
        "## Importing necessary libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wSJGpEu1cWmI",
        "outputId": "07098125-2340-48cf-a0cd-67440e9f24db"
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "import nltk\n",
        "nltk.download('stopwords')\n",
        "%matplotlib inline\n",
        "sns.set_style(\"darkgrid\")\n",
        "plt.style.use(\"fivethirtyeight\")\n",
        "print(\"Necessary packages included successfully!\")"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n",
            "Necessary packages included successfully!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6gz7JglLjH8U"
      },
      "source": [
        "## 1. Representing text as numerical data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HEh2dqT6i7-i"
      },
      "source": [
        "# example text for model training (SMS messages)\n",
        "simple_train = ['call you tonight', 'Call me a cab', 'Please call me... PLEASE!']"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QhGclG-ZjRui"
      },
      "source": [
        "# import and instantiate CountVectorizer (with the default parameters)\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "vect = CountVectorizer()"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2OZQeUtxj2a3",
        "outputId": "8e0bd715-c0c1-4587-a4a0-1865dcaadd56"
      },
      "source": [
        "# learn the 'vocabulary' of the training data (occurs in-place)\n",
        "vect.fit(simple_train)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
              "                dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
              "                lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
              "                ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
              "                strip_accents=None, token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
              "                tokenizer=None, vocabulary=None)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3mwQZpkFj5Hw",
        "outputId": "981e5255-4d28-44dd-e1d0-4fb5d809b713"
      },
      "source": [
        "# examine the fitted vocabulary\n",
        "vect.get_feature_names()"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['cab', 'call', 'me', 'please', 'tonight', 'you']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DYXT6QnCj8Y-",
        "outputId": "e0df455a-6810-4298-8396-030a1aa58cfb"
      },
      "source": [
        "# transform training data into a 'document-term matrix'\n",
        "simple_train_dtm = vect.transform(simple_train)\n",
        "simple_train_dtm"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<3x6 sparse matrix of type '<class 'numpy.int64'>'\n",
              "\twith 9 stored elements in Compressed Sparse Row format>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Vx0TufUWj_f7",
        "outputId": "d7851980-c53a-41a7-9ebb-aad6b7b9bf7d"
      },
      "source": [
        "# convert sparse matrix to a dense matrix\n",
        "simple_train_dtm.toarray()"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0, 1, 0, 0, 1, 1],\n",
              "       [1, 1, 1, 0, 0, 0],\n",
              "       [0, 1, 1, 2, 0, 0]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 137
        },
        "id": "L72IJthbkDH-",
        "outputId": "5e34feb8-9219-4c14-a5b9-16d4fd626c25"
      },
      "source": [
        "# examine the vocabulary and document-term matrix together\n",
        "pd.DataFrame(simple_train_dtm.toarray(), columns=vect.get_feature_names())"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>cab</th>\n",
              "      <th>call</th>\n",
              "      <th>me</th>\n",
              "      <th>please</th>\n",
              "      <th>tonight</th>\n",
              "      <th>you</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   cab  call  me  please  tonight  you\n",
              "0    0     1   0       0        1    1\n",
              "1    1     1   1       0        0    0\n",
              "2    0     1   1       2        0    0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "__82s5G9kGSb",
        "outputId": "d7457d64-a598-43dd-a1ed-e88915271b98"
      },
      "source": [
        "# check the type of the document-term matrix\n",
        "type(simple_train_dtm)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "scipy.sparse.csr.csr_matrix"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HGBIqef6kLPj",
        "outputId": "2078557e-bd13-4485-a2fe-590797e10422"
      },
      "source": [
        "# examine the sparse matrix contents\n",
        "print(simple_train_dtm)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "  (0, 1)\t1\n",
            "  (0, 4)\t1\n",
            "  (0, 5)\t1\n",
            "  (1, 0)\t1\n",
            "  (1, 1)\t1\n",
            "  (1, 2)\t1\n",
            "  (2, 1)\t1\n",
            "  (2, 2)\t1\n",
            "  (2, 3)\t2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kMv7gIQ4kNx8"
      },
      "source": [
        "# example text for model testing\n",
        "simple_test = [\"please don't call me\"]"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "v76GaavFkSDd",
        "outputId": "f4c4e694-1100-48ee-c888-2fade4a7b330"
      },
      "source": [
        "# transform testing data into a document-term matrix (using existing vocabulary)\n",
        "simple_test_dtm = vect.transform(simple_test)\n",
        "simple_test_dtm.toarray()"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0, 1, 1, 1, 0, 0]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 77
        },
        "id": "jOZOL10OkUh6",
        "outputId": "62dffe55-9741-4b14-ba86-aa4faab32681"
      },
      "source": [
        "# examine the vocabulary and document-term matrix together\n",
        "pd.DataFrame(simple_test_dtm.toarray(), columns=vect.get_feature_names())"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>cab</th>\n",
              "      <th>call</th>\n",
              "      <th>me</th>\n",
              "      <th>please</th>\n",
              "      <th>tonight</th>\n",
              "      <th>you</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   cab  call  me  please  tonight  you\n",
              "0    0     1   1       1        0    0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ghaGexa1kwEQ"
      },
      "source": [
        "## 2. Reading a text-based dataset into notebook"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 196
        },
        "id": "Tie7p7IfkXpQ",
        "outputId": "5d01f03c-0429-4e0e-a049-1c4a4b7bc7a9"
      },
      "source": [
        "url = 'https://raw.githubusercontent.com/MainakRepositor/Datasets-/master/spam.csv'\n",
        "sms = pd.read_csv(url,error_bad_lines=False,encoding='latin-1')\n",
        "sms.dropna(how=\"any\", inplace=True, axis=1)\n",
        "sms.columns = ['label', 'message']\n",
        "sms.head()"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>label</th>\n",
              "      <th>message</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ham</td>\n",
              "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ham</td>\n",
              "      <td>Ok lar... Joking wif u oni...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>spam</td>\n",
              "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ham</td>\n",
              "      <td>U dun say so early hor... U c already then say...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ham</td>\n",
              "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  label                                            message\n",
              "0   ham  Go until jurong point, crazy.. Available only ...\n",
              "1   ham                      Ok lar... Joking wif u oni...\n",
              "2  spam  Free entry in 2 a wkly comp to win FA Cup fina...\n",
              "3   ham  U dun say so early hor... U c already then say...\n",
              "4   ham  Nah I don't think he goes to usf, he lives aro..."
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ftTUkK67k0Wd"
      },
      "source": [
        "## 3. Exploratory Data Analysis (EDA)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 166
        },
        "id": "tgkEiFG9kqrl",
        "outputId": "4971ab2d-026a-4a91-c5c2-d68e91de6586"
      },
      "source": [
        "sms.describe()"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>label</th>\n",
              "      <th>message</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>5572</td>\n",
              "      <td>5572</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>unique</th>\n",
              "      <td>2</td>\n",
              "      <td>5169</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>top</th>\n",
              "      <td>ham</td>\n",
              "      <td>Sorry, I'll call later</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>freq</th>\n",
              "      <td>4825</td>\n",
              "      <td>30</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       label                 message\n",
              "count   5572                    5572\n",
              "unique     2                    5169\n",
              "top      ham  Sorry, I'll call later\n",
              "freq    4825                      30"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 166
        },
        "id": "8tOf8rdrk5pu",
        "outputId": "8831c62e-7b70-4be0-a33f-5ae27015acbe"
      },
      "source": [
        "sms.groupby('label').describe()"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead tr th {\n",
              "        text-align: left;\n",
              "    }\n",
              "\n",
              "    .dataframe thead tr:last-of-type th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th></th>\n",
              "      <th colspan=\"4\" halign=\"left\">message</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th></th>\n",
              "      <th>count</th>\n",
              "      <th>unique</th>\n",
              "      <th>top</th>\n",
              "      <th>freq</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>label</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>ham</th>\n",
              "      <td>4825</td>\n",
              "      <td>4516</td>\n",
              "      <td>Sorry, I'll call later</td>\n",
              "      <td>30</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>spam</th>\n",
              "      <td>747</td>\n",
              "      <td>653</td>\n",
              "      <td>Please call our customer service representativ...</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "      message                                                               \n",
              "        count unique                                                top freq\n",
              "label                                                                       \n",
              "ham      4825   4516                             Sorry, I'll call later   30\n",
              "spam      747    653  Please call our customer service representativ...    4"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 196
        },
        "id": "fvkzXgIxk8qd",
        "outputId": "a39c1b97-5d1b-4196-ae9f-222389039515"
      },
      "source": [
        "# convert label to a numerical variable\n",
        "sms['label_num'] = sms.label.map({'ham':0, 'spam':1})\n",
        "sms.head()"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>label</th>\n",
              "      <th>message</th>\n",
              "      <th>label_num</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ham</td>\n",
              "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ham</td>\n",
              "      <td>Ok lar... Joking wif u oni...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>spam</td>\n",
              "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ham</td>\n",
              "      <td>U dun say so early hor... U c already then say...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ham</td>\n",
              "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  label                                            message  label_num\n",
              "0   ham  Go until jurong point, crazy.. Available only ...          0\n",
              "1   ham                      Ok lar... Joking wif u oni...          0\n",
              "2  spam  Free entry in 2 a wkly comp to win FA Cup fina...          1\n",
              "3   ham  U dun say so early hor... U c already then say...          0\n",
              "4   ham  Nah I don't think he goes to usf, he lives aro...          0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 196
        },
        "id": "0tH1qcZ4lAJh",
        "outputId": "19fe514d-c68f-4a6e-c396-d6b7ec43e95c"
      },
      "source": [
        "sms['message_len'] = sms.message.apply(len)\n",
        "sms.head()"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>label</th>\n",
              "      <th>message</th>\n",
              "      <th>label_num</th>\n",
              "      <th>message_len</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ham</td>\n",
              "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
              "      <td>0</td>\n",
              "      <td>111</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ham</td>\n",
              "      <td>Ok lar... Joking wif u oni...</td>\n",
              "      <td>0</td>\n",
              "      <td>29</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>spam</td>\n",
              "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
              "      <td>1</td>\n",
              "      <td>155</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ham</td>\n",
              "      <td>U dun say so early hor... U c already then say...</td>\n",
              "      <td>0</td>\n",
              "      <td>49</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ham</td>\n",
              "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
              "      <td>0</td>\n",
              "      <td>61</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  label  ... message_len\n",
              "0   ham  ...         111\n",
              "1   ham  ...          29\n",
              "2  spam  ...         155\n",
              "3   ham  ...          49\n",
              "4   ham  ...          61\n",
              "\n",
              "[5 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 553
        },
        "id": "bWwHe6YWlDRa",
        "outputId": "f50b177e-854b-43f3-9a66-fcdd8752fccd"
      },
      "source": [
        "plt.figure(figsize=(17, 8))\n",
        "\n",
        "sms[sms.label=='ham'].message_len.plot(bins=35, kind='hist', color='blue', \n",
        "                                       label='Ham messages', alpha=0.6)\n",
        "sms[sms.label=='spam'].message_len.plot(kind='hist', color='red', \n",
        "                                       label='Spam messages', alpha=0.6)\n",
        "plt.legend()\n",
        "plt.xlabel(\"Message Length\")"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Text(0.5, 0, 'Message Length')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAABHQAAAIICAYAAAAPPLk2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeXyNZ/7/8fdJIjkkspNI2liThgS1JfZpaEtFpR3fUWqrDlqdYqpqLUGVVtWuNNFWbTVjKIofNWNaM2o3NC0qmVJEYkkkollkOb8/jDPNxHY4We56PR+PPurc13Xf1+c+cnk88n5c93WbMjIyLAIAAAAAAIBhOJR3AQAAAAAAALANgQ4AAAAAAIDBEOgAAAAAAAAYDIEOAAAAAACAwRDoAAAAAAAAGAyBDgAAAAAAgMEQ6AAAAAAAABgMgQ4AAAAAAIDBEOj8SiQmJpZ3CcCvDvMKsC/mFGBfzCnAvphTMBoCHQAAAAAAAIMh0AEAAAAAADAYAh0AAAAAAACDIdABAAAAAAAwGKfyLgAAAAAAgLL0888/q6CgoNgxs9mszMzMcqoIDypXV1c5Od1bNEOgAwAAAAB4YOTl5UmSPDw8ih13cXGR2Wwuj5LwgLJYLMrIyFDVqlXvKdThkSsAAAAAwAMjNzdXVapUKe8yAJlMJnl6eurnn3++p/MJdAAAAAAADxSTyVTeJQCS7u9nkUAHAAAAAADAYAh0AAAAAAAADIZABwAAAAAAwGDK9S1Xhw4d0sqVK3X8+HFdvHhREydOVNeuXa3tERERNz3v//7v/zRq1ChJ0uTJk7V58+Zi7eHh4fr444+tn69du6a5c+fqyy+/VF5enlq0aKFRo0bJz8+vFO4KAAAAAGAk586ZdOaMs5ycymbNQ40aFgUEWO66/+TJk5WRkaHZs2cXO3706FG98MILWr9+vQICAuxdJiq4cg10cnJyVLduXXXp0kWTJk0q0b5ly5Zin48dO6bXX39djz/+eLHjERERxc6vVKlSsfZZs2Zp586dmjp1qjw8PDRnzhyNGDFCy5Ytk6Ojo93uBwAAAABgPCkpJq1Y4VJmvx/27XvNpkAHuJlyDXTatGmjNm3aSJKmTJlSot3X17fY5507dyooKEhNmzYtdrxSpUol+t5w9epVbdy4URMnTlRkZKSk6+lmt27dtG/fPrVq1coetwIAAAAAQLkqLCzUtGnTdODAAaWnp6t69eqKiYlRnz595OBwffXRjdU+jz76qD777DPl5eWpe/fueuWVV7RkyRKtXbtWJpNJvXr1Uv/+/W85VlxcnHbs2KE+ffooPj5ely9f1uOPP66xY8dqw4YNWrp0qXJzcxUdHa3hw4dbx8/Pz9fixYu1bds2ZWZmqk6dOnr55Zetv5sXFBRozpw52rFjhzIzM+Xl5aXOnTvr1VdflST9/e9/V3x8vM6cOSMXFxfVrVtX06ZNk4+Pj86ePas5c+bou+++U3Z2tmrWrKnBgwerXbt21rrT0tI0bdo07du3T15eXho0aJBWrlypDh06aPDgwZKu5wjz5s3T119/rby8PD3yyCMaPny4GjRoYG1/7733tGfPHv3888/y9fXVc889p169etn/L/U2yjXQsUV2dra2b9+ugQMHlmg7cuSIOnXqJDc3NzVt2lRDhgyRt7e3pOuregoKCqxhjiT5+fmpVq1aSkhIINABAAAAAPwqWCwWVa9eXdOnT5enp6e+//57TZ8+XR4eHoqJibH2O3z4sKpXr67Fixfrhx9+0MSJE3XixAk98sgjiouL04EDB/Tuu+8qIiJC9evXv+V4KSkp2rlzp2bNmqULFy5ozJgxunTpknx9fTV//nydOnVK48aNU+PGjdWhQwdJ1xdzJCcna8qUKfLz89OuXbv0+uuva+nSpQoJCdGf/vQnffXVV5o6daoCAgJ04cIF/fTTT5KkS5cuafz48frDH/6gqKgo5eTkKCEhwVpPdna2WrVqpZdfflkuLi7avn27Ro8erVWrVqlWrVqSrgdaly5d0sKFC2U2mzVnzhylpqYW+w5fe+01ubm5adasWXJ3d9fmzZv1hz/8QWvWrJGvr68WLVqkpKQkzZo1S97e3jp37pwuX75sz7/Ku2KYQGfbtm3Kz89XdHR0seOtWrVSVFSUAgIClJKSokWLFumVV17RsmXL5OzsrLS0NDk6OsrT07PYed7e3kpLS7vleImJiaVyH6XJiDUDFR3zCrAv5hRgX8wpwHZms1kuLi7FjhUUOEtyVGFhYZnUUFBQoNzca3fdv7CwUHv27NFvfvObYseLiookSXl5ecrNzZWkYqtq2rVrp++//15bt25Vp06drNeqUqWKhg0bJkdHR/n7+yskJEQXL17UjBkzJEnR0dH69NNPtWfPHtWuXfuW91BYWKhRo0bJzc1NgYGBioiI0OHDh7Vu3TpVqlRJNWrUUMOGDbV37161bt1aycnJ+vLLL/XnP//Zuqdtt27dtGfPHv3lL3/RiBEjdPbsWT300EOqX7++TCaTPD09FRISotzcXCUnJ6ugoEBt2rSxLuIIDAyUJOXm5iooKEhBQUHWGp9//nnt3LlT27ZtU//+/XX69Gnt2bNHixYtUkhIiCRp9OjReu655/7zd5KrgwcP6sSJE9q4caP15+SFF17Qzp07tXHjRj3//PNKTk5WcHCw6tatK0ny8vKy1nAvrly5ogsXLpQ4HhwcfNvzDBPorF+/Xu3bt7d+UTc8+eST1j/Xq1dPoaGh6tatm3bt2qWoqKh7Hu9OX1xFk5iYaLiagYqOeQXYF3MKsC/mFHBvMjMzZTabix27sRlyWe2h4+TkJLP57jdgdnR01KOPPqpx48YVO/7vf/9bo0aNkouLi/We1q5dqw0bNig1NVV5eXkqKCiQv7+/td3R0VF16tSRq6ur9Tq+vr5yc3Mr9r34+PgoKyurxHf1y3vw9/cvtv1JtWrVVLNmTVWtWrXYta9cuSKz2ayTJ0/KYrGoX79+xa517do1NW/eXGazWTExMXr11VfVp08fRUZGqnXr1mrdurUcHBwUHh6uiIgIvfDCC4qMjFSLFi3UsWNHa06Qk5Oj+Ph47dq1S5cuXVJBQYGuXbumkJAQmc1mpaSkyMHBQY0bN5aT0/U4pGbNmvL19f3P34lZP/74o3Jzc9WtW7cSNZ4/f15ms1k9evTQmDFjdOLECUVGRqpdu3Yltoaxhbu7ux5++GGbzzNEoHPixAkdO3ZMr7zyyh37VqtWTdWrV9fp06clXf8hLCwsVEZGRrEwKD09XU2aNCm1mgEAAAAAsBez2Vzil/6srKxin7dv367Zs2dr2LBhatSokVxdXbVmzRp9/fXXxfrdCDNud8xkMsliuf3GzTc752bHbqwkslgsMplMWrp0aYl+N1bDhIaGasOGDdqzZ4/279+vyZMnKzg4WAsWLJCjo6Pmz5+vhIQE7d27Vxs3btQHH3ygxYsXKyQkRHPnztXu3bs1fPhwPfzwwzKbzZo0aZLy8/Nvex+/VFRUJG9vb8XFxZVouxGCtW7dWhs3btQ333yj/fv367XXXlPHjh01ceLEux7HHgwR6Hz++ecKCAi45WvMfykjI0MXL160poT169eXk5OT9u7dq86dO0uSzp8/r1OnTqlhw4alWjcAAAAAAGXl8OHDCgsLU48ePazHkpOTy7Gi4kJCQmSxWJSWlqbmzZvfsp+rq6s6duyojh07qmvXrnrxxRd15swZ1axZUyaTSY0aNVKjRo00cOBA9ezZU3/9618VEhKiI0eOKDo62rpfT15enpKTk62PYdWqVUtFRUU6fvy4wsPDJV3PBy5dumQdOzQ0VOnp6XJwcLA+znUznp6e6tKli7p06aLWrVvrzTff1JgxY+Ts7GyPr+qulGugk52drbNnz0q6noKlpqbqxIkTcnd3l7+/v6Trz6Bt3bpVffv2lclkKnF+fHy8oqKi5Ovrq5SUFC1cuFDe3t567LHHJElubm7q1q2bFixYIG9vb+try+vVq3dXARFu79w5k1JSTHfuaCc1alh4vR8AAAAA3ERQUJA2b96sb775Rg899JC2b9+uQ4cOyd3dvbxLk3T98abOnTtrypQpGj58uEJDQ5WZmalDhw4pMDBQUVFRWrlypXx9fRUSEiInJydt27ZNrq6u8vPzU0JCgvVt1d7e3vrhhx90/vx56z4/QUFB+uqrr9S+fXs5OTlpyZIlysvLKzZ+y5Yt9c4772j06NFycXHRvHnzZDabrXlDRESEGjVqpJEjR2ro0KGqWbOm0tLStGfPHrVo0UJNmjTRhx9+qEceeUR16tRRYWGh/v73vyswMLBMwxypnAOdY8eOaciQIdbPcXFxiouLU3R0tGJjYyVdXzKWm5urp59+usT5Dg4OSkpK0pYtW5SVlSVfX181a9ZM06dPL/Y84IgRI+To6Khx48YpLy9PLVq00KRJk8rs+chfs5QUk5YvL7sf2r59rxHoAAAAALCrGjUs6tMn76aPIpXWeKXht7/9rU6cOKEJEybIYrEoKipKvXv31hdffFEq492LiRMn6uOPP9b8+fN14cIFubu7KywsTM2aNZN0fXXOihUrdObMGZlMJuujVGazWW5ubvr222+1Zs0aZWVlyc/PTy+++KKeeuopSdIf//hHTZ06VYMHD5a7u7t69uxZLNC5Mf60adM0ZMgQeXl56aWXXlJycrI1jDGZTJozZ44WL16sadOmKT09Xd7e3mrcuLG6dOkiSapUqZIWLVqkc+fOycXFReHh4Xr//ffL8Fu8zpSRkcFvx78C5bUp3sGDDmUe6DRrVlRm4+HBxmaTgH0xpwD7Yk4B9yYzM1MeHh4ljufm5t5yA2D8emVkZKhLly6aOnWq9VGtsnarn8k7McQeOgAAAAAAAPdr//79ys7OVr169ZSenq5FixbJ09NTrVq1Ku/SbEagAwAAAAAAHggFBQVavHixkpOTZTabFR4erg8//FCVK1cu79JsRqADAAAAAAAeCK1atTLkapybcSjvAgAAAAAAAGAbAh0AAAAAAACDIdABAAAAAAAwGAIdAAAAAAAAgyHQAQAAAAAAMBgCHQAAAAAAAIMh0AEAAAAAADAYp/IuAAAAAACA8mQ6d07OZ87IwalsfkW21KghS0DAXfe/fPmy4uLi9M033+jSpUuqWrWq6tSpo/79+ysyMrIUK0VFRqADAAAAAHigmVJS5LJihRwdHctkvGt9+9oU6IwePVq5ubkaP368Hn74YV2+fFmHDh1SZmZmKVaJio5ABwAAAACACiorK0uHDx/WggULFBERIUmqUaOGGjRoUKxfTEyMoqOjdfbsWX399deqXLmy+vTpoz59+lj7rFy5Ups3b9bZs2dVtWpVtWrVSsOHD1fVqlUlSZs2bdJ7772n6dOna/bs2UpNTVVERIQmT56svXv36oMPPlB6errat2+vsWPHymw237TmgwcPasiQIZozZ44WL16skydPKjQ0VFOnTlVycrJmzpyp5ORkNW3aVLGxsfL09LSe+8UXX2jFihVKTk6Wn5+funfvrp49e8rB4fqOMevWrdPKlSt1/vx5Va5cWaGhoZo9e7acnJyUlJSkWbNm6dixYyoqKlJgYKBGjBih5s2bq7CwUNOmTdOBAweUnp6u6tWrKyYmRn369LFeu6CgQPPmzdPmzZslSdHR0bp27ZpOnTqlxYsXS5IsFouWL1+uzz//XJcuXdJDDz2kfv366amnnrLew5IlS7Rx40alpaWpatWqioyM1OTJk+/r5+BmCHQAAAAAAKigKleurCpVqmjnzp1q3LixXFxcbtn3s88+U9++ffX73/9eBw8e1MyZMxUYGKioqChJkoODg1577TUFBgYqNTVVM2fO1MyZM4uFDdeuXdPKlSs1ZcoU5efna8yYMRo9erRcXFz0zjvvKDMzU6NHj9batWvVu3fv29YeFxen1157TW5ubpowYYLGjx8vZ2dnjRs3Tg4ODho7dqzi4+P1xhtvSJLWr1+vDz/8UCNHjlRoaKh+/PFHvf3223JyclKPHj109OhRvffee4qNjVXjxo2VlZWlAwcOWMebMGGCgoOD9cknn8jR0VFJSUlydnaWdD2IqV69uqZPny5PT099//33mj59ujw8PBQTEyPpv4HX+PHjVbduXf3lL3/Rtm3b9Mgjj1jHWLRokXbs2KFRo0YpKChICQkJmjZtmqpWraq2bdtqx44dWrlypd566y3Vq1dP6enp+u6772z8W787BDoAAAAAAFRQTk5OmjhxoqZNm6b169crJCREjRs3VseOHRUeHl6sb1hYmF588UVJUs2aNXX06FGtWrXKGuj06tXL2jcgIEBDhw7VyJEjFRsba12lUlhYqFGjRqlmzZqSpE6dOumzzz7T1q1brStp2rdvrwMHDtwx0HnppZfUpEkTSdJvf/tbzZw5U8uWLVNoaKik6ytgduzYYe3/0UcfaejQoerYsaMkKTAwUP3799fatWvVo0cPnT9/XmazWe3atZOrq6tq1KihkJAQ6/mpqanq3bu3atWqJUl6+OGHi32PL730UrH7/+GHH/Tll19aA53Vq1erX79+6tChgyRpxIgR2r17t/WcnJwcffbZZ5o3b571vgIDA3X06FH95S9/Udu2bZWSkiIfHx+1bNlSTk5O8vf3L7Gayl4IdAAAAAAAqMA6dOigNm3a6PDhw0pISNDu3bu1cuVKDRkyRAMGDLD2a9iwYbHzGjZsqK+++sr6ef/+/fr000916tQpXb16VYWFhcrPz1daWpqqVasmSXJ2draGOZLk7e0tHx+fYo9FeXt76+TJk3esOzg4uNg5klSvXr1ixy5fvizp+sbP58+f1/Tp0/Xuu+9a+xQWFspisUiSIiIiVKNGDT3zzDNq2bKlIiMjFRUVJVdXV0nXA6u3335bmzdvVosWLdShQwdruCNJa9eu1YYNG5Samqq8vDwVFBTI399fknT16lWlpaUVC19MJpPCwsJ0/vx5SdLJkyeVl5en4cOHy2QyWfsVFBSoRo0akqTHH39cf/rTn/TMM88oMjJSrVq1Uvv27a0rheyJQAcAAAAAgArOxcVFkZGRioyM1MCBAzV16lTFx8erT58+qlSp0h3PT0lJ0YgRIxQTE6PBgwfLw8NDP/zwg958803l5+db+/3vxtAmk0lO//P2L5PJpKKiojuO+cvzbgQg/3vsxnVu/H/MmDFq1KjRTa/n6uqqZcuW6V//+pf27dunTz/9VIsWLdLSpUtVrVo1DR48WJ07d9bu3bu1Z88eLVmyRGPGjFG3bt20fft2zZ49W8OGDVOjRo3k6uqqNWvW6Ouvv77jfdxwo8b333/fGgT97736+flpzZo12r9/v/bt26e5c+dqyZIl+uSTT1S5cuW7HutuEOgAAAAAAGAwtWvXVmFhofLy8qyBzv/u1fLdd99ZV6gcO3ZM+fn5eu2116yhzT//+c8yrfl2fHx8VK1aNSUnJys6OvqW/ZycnNSiRQu1aNFCgwcPVqdOnfTPf/5Tzz77rCQpKChIQUFBeu655/TOO+9ow4YN6tatmw4fPqywsDD16NHDeq3k5GTrn93c3OTj46OjR4+qRYsWkq7vu3P06FH5+PhIuv6dOzs7KzU11drnZlxcXNS2bVu1bdtW/fv311NPPaUjR46oZcuW9/Udlfgu7Ho1AAAAAABgNxkZGRo7dqy6deumevXqqUqVKjp27JiWL1+uFi1ayM3Nzdr3u+++09KlS9WhQwcdOnRIW7Zs0ZQpUyRd30+mqKhIq1ev1mOPPabvvvtOq1evLq/buqlBgwbp/fffl5ubm9q0aaOCggIdP35cFy9e1AsvvKB//OMfSk5OVpMmTeTu7q6DBw8qOztbtWrVUm5urubNm6eOHTuqRo0aSk9P15EjRxQWFibpetCzefNmffPNN3rooYe0fft2HTp0SO7u7tbxe/bsqeXLlysoKEh16tTRunXrdOnSJfn6+kq6vkKod+/emjt3riwWi5o0aaKcnBwlJCTIwcFBzz77rDZt2qSCggKFh4ercuXK+utf/yonJ6di+/nYC4EOAAAAAOCBZqlRQ3l9+pR4tKg0x7tbVapUUcOGDbV69WqdPXtW165dU/Xq1dWpUyfrBsg39OrVS0lJSdbHewYPHmzdYDg4OFivv/66li1bpsWLF6thw4YaNmyYxo8fb9d7ux/PPPOMKleurBUrVuiDDz6Qi4uL6tSpo9/97neSpKpVq+rrr7/WkiVLlJeXp8DAQI0fP15NmjRRfn6+rly5oilTpujSpUvy8PBQ27ZtNWzYMEnXN2U+ceKEJkyYIIvFoqioKPXu3VtffPGFdfzevXsrLS1Nb731lkwmk7p27arHHntM6enp1j4vv/yyvL29tXLlSr377rtydXVVSEiI+vbtK+n6Sp9ly5Zp3rx5KigoUO3atfXuu+8qMDDQ7t+XKSMjw2L3q6LMJSYmFttwqqwcPOig5cvtv7nTrfTte03Nmt35WU3AHsprXgG/VswpwL6YU8C9yczMlIeHR4njubm5MpvN5VCRfcTExOh3v/ud+vTpU96l/Kr06dNHjRs3tr5avTTc6mfyTlihAwAAAAAAHngpKSnas2ePmjZtqoKCAq1fv15JSUkaN25ceZd2UwQ6AAAAAADggefg4KAtW7Zo3rx5slgsql27tubMmVPsVeYVCYEOAAAAAAAGt2HDhvIuwfD8/PwUHx9f3mXcNYfyLgAAAAAAAAC2IdABAAAAAAAwGAIdAAAAAMADxWLhZc+oGO7nZ5FABwAAAADwwDCbzcrOzi7vMgBZLBZlZGTI1dX1ns5nU2QAAAAAwAPDxcVFBQUFyszMLHb8ypUrcnd3L6eq8KCqWrWqnJzuLZoh0AEAAAAAPFButiLiwoULevjhh8uhGuDe8MgVAAAAAACAwRDoAAAAAAAAGAyBDgAAAAAAgMEQ6AAAAAAAABgMgQ4AAAAAAIDBEOgAAAAAAAAYDIEOAAAAAACAwRDoAAAAAAAAGAyBDgAAAAAAgMEQ6AAAAAAAABgMgQ4AAAAAAIDBEOgAAAAAAAAYDIEOAAAAAACAwRDoAAAAAAAAGAyBDgAAAAAAgMEQ6AAAAAAAABgMgQ4AAAAAAIDBEOgAAAAAAAAYDIEOAAAAAACAwRDoAAAAAAAAGAyBDgAAAAAAgMEQ6AAAAAAAABgMgQ4AAAAAAIDBOJXn4IcOHdLKlSt1/PhxXbx4URMnTlTXrl2t7ZMnT9bmzZuLnRMeHq6PP/7Y+vnatWuaO3euvvzyS+Xl5alFixYaNWqU/Pz8rH1SU1M1Y8YMHThwQC4uLurUqZOGDx+uSpUqlf5NAgAAAAAA2Fm5Bjo5OTmqW7euunTpokmTJt20T0RERLG2/w1hZs2apZ07d2rq1Kny8PDQnDlzNGLECC1btkyOjo4qLCzUa6+9Jg8PD8XFxSkzM1OTJ0+WxWLRG2+8UYp3BwAAAAAAUDrK9ZGrNm3a6JVXXlHHjh3l4HDzUipVqiRfX1/rfx4eHta2q1evauPGjRo2bJgiIyMVGhqqyZMnKykpSfv27ZMk7d27Vz/++KMmT56s0NBQRUZGaujQodqwYYOuXr1aJvcJAAAAAABgTxV+D50jR46oU6dO6t69u95++22lp6db244dO6aCggJFRkZaj/n5+alWrVpKSEiQJCUkJKhWrVrFHsFq2bKlrl27puPHj5fdjQAAAAAAANhJuT5ydSetWrVSVFSUAgIClJKSokWLFumVV17RsmXL5OzsrLS0NDk6OsrT07PYed7e3kpLS5MkpaWlydvbu1i7p6enHB0drX1uJjEx0f43VMrKo+a0tGrKzKxShuNlKzHxYpmNBxjx3wKgImNOAfbFnALsizmFiiQ4OPi27RU60HnyySetf65Xr55CQ0PVrVs37dq1S1FRUaU69p2+uIomMTGxXGq+csVBHh7OZTaej49ZwcGed+4I2EF5zSvg14o5BdgXcwqwL+YUjKbCP3L1S9WqVVP16tV1+vRpSZKPj48KCwuVkZFRrF96erp8fHysfX75mJYkZWRkqLCw0NoHAAAAAADASAwV6GRkZOjixYvy9fWVJNWvX19OTk7au3evtc/58+d16tQpNWzYUJLUsGFDnTp1SufPn7f22bt3r5ydnRUaGlq2NwAAAAAAAGAH5frIVXZ2ts6ePStJKioqUmpqqk6cOCF3d3e5u7srPj5eUVFR8vX1VUpKihYuXChvb2899thjkiQ3Nzd169ZNCxYskLe3t/W15fXq1VNERIQkKTIyUnXq1NGkSZP0xz/+UZmZmZo/f75iYmLk5uZWXrcOAAAAAABwz8o10Dl27JiGDBli/RwXF6e4uDhFR0dr9OjRSkpK0pYtW5SVlSVfX181a9ZM06dPl6urq/WcESNGyNHRUePGjVNeXp5atGihSZMmydHRUZLk6Oio2bNn691339XAgQPl4uKizp07a9iwYWV+vwAAAAAAAPZgysjIsJR3Ebh/5bWB18GDDlq+vOw2Re7b95qaNSsqs/HwYGNjPMC+mFOAfTGnAPtiTsFoDLWHDgAAAAAAAAh0AAAAAAAADIdABwAAAAAAwGAIdAAAAAAAAAyGQAcAAAAAAMBgCHQAAAAAAAAMhkAHAAAAAADAYAh0AAAAAAAADIZABwAAAAAAwGAIdAAAAAAAAAyGQAcAAAAAAMBgCHQAAAAAAAAMhkAHAAAAAADAYAh0AAAAAAAADIZABwAAAAAAwGAIdAAAAAAAAAyGQAcAAAAAAMBgCHQAAAAAAAAMhkAHAAAAAADAYAh0AAAAAAAADIZABwAAAAAAwGAIdAAAAAAAAAyGQAcAAAAAAMBgCHQAAAAAAAAMhkAHAAAAAADAYAh0AAAAAAAADIZABwAAAAAAwGAIdAAAAAAAAAyGQAcAAAAAAMBgCHQAAAAAAAAMhkAHAAAAAADAYAh0AAAAAAAADIZABwAAAAAAwGAIdAAAAAAAAAyGQAcAAAAAAMBgCHQAAAAAAAAMhkAHAAAAAADAYAh0AAAAAAAADIZABwAAAAAAwGAIdAAAAAAAAAyGQAcAAAAAAMBgCHQAAAAAAAAMhkAHAAAAAADAYAh0AAAAAAAADM5vJF8AACAASURBVIZABwAAAAAAwGAIdAAAAAAAAAyGQAcAAAAAAMBgCHQAAAAAAAAMhkAHAAAAAADAYAh0AAAAAAAADIZABwAAAAAAwGAIdAAAAAAAAAyGQAcAAAAAAMBgnMpz8EOHDmnlypU6fvy4Ll68qIkTJ6pr166SpIKCAi1atEi7d+/W2bNn5erqqmbNmunVV1+Vv7+/9Rovv/yyDh06VOy6TzzxhN5++23r5ytXruj999/Xzp07JUnt27fXyJEjVbVq1TK4SwAAAAAAAPsq10AnJydHdevWVZcuXTRp0qRibbm5ufrhhx80YMAAhYSE6OrVq5ozZ46GDx+ulStXysnpv6U//fTTGjJkiPWz2Wwudq0JEyYoNTVVc+fOlSS9/fbbio2N1axZs0rv5gAAAAAAAEpJuQY6bdq0UZs2bSRJU6ZMKdbm5uamBQsWFDs2duxY9ezZU6dOnVK9evWsx81ms3x9fW86xsmTJ7V7927Fx8erUaNG1usMHjxYP/30k2rWrGnPWwIAAAAAACh1htpD5+eff5akEo9Kffnll3riiSf03HPPae7cudZ+kpSQkKAqVapYwxxJaty4sSpXrqxvv/22bAoHAAAAAACwo3JdoWOL/Px8zZ07V+3atZOfn5/1eKdOneTv769q1arpxx9/1MKFC5WUlKT58+dLktLS0uTp6SmTyWQ9x2QyycvLS2lpabccLzExsfRuppSUR81padWUmVmlDMfLVmLixTIbDzDivwVARcacAuyLOQXYF3MKFUlwcPBt2w0R6BQUFGjixInKysrSzJkzi7U9++yz1j/Xq1dPgYGBGjBggI4fP67Q0NB7HvNOX1xFk5iYWC41X7niIA8P5zIbz8fHrOBgzzIbDw+28ppXwK8VcwqwL+YUYF/MKRhNhX/kqqCgQG+++aaSkpK0cOFCeXre/pf5+vXry9HRUWfOnJEk+fj4KCMjQxaLxdrHYrHo8uXL8vHxKdXaAQAAAAAASkOFDnQKCgo0fvx4JSUladGiRbfc+PiXkpKSVFhYaA1rGjZsqOzsbCUkJFj7JCQkKCcnp9i+OgAAAAAAAEZRro9cZWdn6+zZs5KkoqIipaam6sSJE3J3d5evr6/GjBmjY8eO6f3335ckXbp0SdL1N2CZzWadPXtWW7duVevWreXp6amTJ09q7ty5euSRR9S4cWNJUu3atdWqVStNnz5dY8eOlSRNnz5dbdu25Q1XAAAAAADAkMo10Dl27JiGDBli/RwXF6e4uDhFR0dr0KBB2rlzpySpX79+xc6bOHGiunbtqkqVKmn//v1avXq1cnJy5OfnpzZt2mjgwIFydHS09n/rrbc0c+ZMDR8+XJLUrl07vfHGG2VwhwAAAAAAAPZXroFOs2bNtG/fvlu2365Nkvz8/PThhx/ecRx3d3dNmTLF5voAAAAAAAAqogq9hw4AAAAAAABKItABAAAAAAAwGAIdAAAAAAAAgyHQAQAAAAAAMBgCHQAAAAAAAIMh0AEAAAAAADAYAh0AAAAAAACDIdABAAAAAAAwGAIdAAAAAAAAgyHQAQAAAAAAMBgCHQAAAAAAAIMh0AEAAAAAADAYAh0AAAAAAACDIdABAAAAAAAwGAIdAAAAAAAAgyHQAQAAAAAAMBgCHQAAAAAAAIMh0AEAAAAAADAYAh0AAAAAAACDIdABAAAAAAAwGAIdAAAAAAAAgyHQAQAAAAAAMBgCHQAAAAAAAIMh0AEAAAAAADAYAh0AAAAAAACDIdABAAAAAAAwGAIdAAAAAAAAgyHQAQAAAAAAMBgCHQAAAAAAAIMh0AEAAAAAADAYAh0AAAAAAACDIdABAAAAAAAwGAIdAAAAAAAAgyHQAQAAAAAAMBgCHQAAAAAAAIMh0AEAAAAAADAYAh0AAAAAAACDIdABAAAAAAAwGAIdAAAAAAAAgyHQAQAAAAAAMBibA53du3fLYrGURi0AAAAAAAC4C062nvDHP/5Rvr6+6tSpkzp37qyQkJDSqAsAAAAAAAC3YPMKnffee0+NGjXSmjVr1K9fPz3//PNasWKFLl68WBr1AQAAAAAA4H/YvEKnffv2at++vX7++Wf97W9/09atW7Vw4UItXLhQzZs3V5cuXRQVFSWz2Vwa9QIAAAAAADzwbA50bnB1dVW3bt3UrVs3XbhwQdu2bdPWrVs1efJkvfvuu/rNb36j6OhoRURE2LNeAAAAAACAB55d3nJVVFSkgoIC5efny2KxyMXFRfv379fQoUPVp08f/fvf/7bHMAAAAAAAANB9rNC5evWq/vrXv+r//b//p2+//VaOjo5q06aN/vCHP6hNmzZycHDQzp07NXv2bE2ZMkWffvqpPesGAAAAAAB4YNkc6Hz11VfaunWrdu3apWvXrqlBgwYaMWKEnnzySXl4eBTr+9hjjykjI0MzZsywW8EAAAAAAAAPOpsDndGjR6tatWrq2bOnoqOjVatWrdv2Dw4OVufOne+1PgAAAAAAAPwPmwOd+fPnq0WLFjKZTHfVPywsTGFhYTYXBgAAAAAAgJuzOdDhrVUAAAAAAADly+a3XM2aNUvdu3e/ZXv37t01b968+yoKAAAAAAAAt2ZzoLNr1y49/vjjt2x/4okn9I9//OO+igIAAAAAAMCt2RzoXLhwQTVq1Lhlu7+/v86fP39X1zp06JBef/11RUdHKyIiQps2bSrWbrFYFBcXpy5duqhdu3Z6+eWX9e9//7tYnytXrig2NlZRUVGKiopSbGyssrKyivVJSkrSSy+9pHbt2ik6OlpLliyRxWK5yzsGAAAAAACoWGwOdFxdXXXu3Llbtp87d04uLi53da2cnBzVrVtXI0aMuOk5y5Yt06pVqzRy5EgtXbpUXl5eGjp0qH7++WdrnwkTJuj48eOaO3eu5s6dq+PHjys2NtbafvXqVb366qvy9vbW0qVLNWLECK1YsUKrVq2y4a4BAAAAAAAqDpsDnebNm+vzzz9XampqibZz585p3bp1at68+V1dq02bNnrllVfUsWNHOTgUL8VisWj16tXq16+fOnTooLp16yo2NlbZ2dnatm2bJOnkyZPavXu3xo0bp0aNGqlRo0YaO3as/vnPf+qnn36SJG3dulV5eXmKjY1V3bp11bFjR/Xt21erVq1ilQ4AAAAAADAkmwOdwYMHq7CwUL169dL777+vzz//XJ9//rlmzpyp3r17y2Kx6OWXX77vws6dO6e0tDS1bNnSesxsNqtJkyb69ttvJUkJCQmqUqWKGjVqZO3TuHFjVa5cuVifRx99VGaz2dqnVatWunjx4m1XGgEAAAAAAFRUNr+2PCgoSPHx8Xrvvff05z//uVhb06ZNNXLkSNWsWfO+C0tLS5MkeXt7Fzvu7e2tCxcuWPt4enrKZDJZ200mk7y8vKznp6enq3r16iWuceP8wMDAm46fmJh43/dQ1sqj5rS0asrMrFKG42UrMfFimY0HGPHfAqAiY04B9sWcAuyLOYWKJDg4+LbtNgc6klS3bl0tXrxYGRkZSk5OliQFBgbK09PzXi5XId3pi6toEhMTy6XmK1cc5OHhXGbj+fiYFRz86/k5Q8VWXvMK+LViTgH2xZwC7Is5BaO5p0DnBk9Pz1ILcXx8fCRdX2Hj7+9vPZ6enm5t8/HxUUZGhiwWi3WVjsVi0eXLl619vL29lZ6eXuzaNz7f6AMAAAAAAGAk9xToFBYWas+ePUpOTlZWVlaJzYVNJpN+//vf31dhAQEB8vHx0d69e9WgQQNJUl5eng4fPqyhQ4dKkho2bKjs7GwlJCRY99FJSEhQTk6O9XPDhg21cOFC5eXlWd+ktXfvXlWrVk0BAQH3VSMAAAAAAEB5sDnQOXr0qMaMGaMLFy7c8i1RdxvoZGdn6+zZs5KkoqIipaam6sSJE3J3d5e/v7969uyppUuXqlatWgoKCtLHH3+sypUrq1OnTpKk2rVrq1WrVpo+fbrGjh0rSZo+fbratm1r3cenc+fOWrJkiaZMmaIXX3xRp0+f1rJlyzRw4MBie+8AAAAAAAAYhc2BzowZM5SXl6f33ntPjz76qKpWrXrPgx87dkxDhgyxfo6Li1NcXJyio6MVGxurfv36KS8vTzNmzFBWVpbCwsI0f/58ubq6Ws956623NHPmTA0fPlyS1K5dO73xxhvWdjc3Ny1YsEAzZsxQ//79VbVqVT3//PN6/vnn77luAAAAAACA8mRzoJOUlKQhQ4aoXbt29z14s2bNtG/fvlu2m0wmDR48WIMHD75lH3d3d02ZMuW249SrV09xcXH3XCcAAAAAAEBF4mDrCdWrV7/lo1YAAAAAAAAofTYHOv369dP69et19erV0qgHAAAAAAAAd2DzI1dXrlxR5cqV1b17d3Xo0EF+fn5ycCieC5lMJvXt29duRQIAAAAAAOC/bA50Fi5caP3zunXrbtqHQAcAAAAAAKD02BzorF+/vjTqAAAAAAAAwF2yOdCpUaNGadQBAAAAAACAu2RzoHPD6dOndejQIaWnp6tz584KCAhQfn6+0tLS5OPjo0qVKtmzTgAAAAAAAPyHzYFOUVGR3nnnHW3cuFEWi0Umk0kNGza0Bjq9evXSwIED1bt379KoFwAAAAAA4IFn82vLP/nkE33xxRd66aWX9NFHH8lisVjbqlSpog4dOujvf/+7XYsEAAAAAADAf9kc6GzatElPP/20BgwYoIceeqhEe926dXXmzBm7FAcAAAAAAICSbA50Lly4oAYNGtyy3cXFRdnZ2fdVFAAAAAAAAG7N5kDHx8dHqampt2w/fvy4/P3976soAAAAAAAA3JrNgU5UVJTWrl2r06dPW4+ZTCZJ0jfffKMtW7bo8ccft1+FAAAAAAAAKMbmt1wNGjRIhw4dUt++fdW4cWOZTCYtXbpUH3zwgY4eParQ0FD179+/NGoFAAAAAACA7mGFjpubm5YsWaIXXnhB6enpcnZ21pEjR5Sdna1Bgwbpww8/lNlsLo1aAQAAAAAAoHtYoSNd3/h4wIABGjBggL3rAQAAAAAAwB3YvEIHAAAAAAAA5cvmFTpvvfXWXfWbMGGCzcUAAAAAAADgzmwOdA4cOFDiWFFRkS5duqSioiJ5eXmxhw4AAAAAAEApsjnQ2bBhw02PFxQUaN26dVq9erXmz59/34UBAAAAAADg5uy2h46Tk5N69OihyMhIzZw5016XBQAAAAAAwP+w+6bIwcHB+te//mXvywIAAAAAAOA/7B7o7Nu3jz10AAAAAAAASpHNe+gsWbLkpsezsrL0r3/9Sz/88IP69+9/34UBAAAAAADg5mwOdOLj4296vGrVqnrooYc0ZswYPfPMM/ddGAAAAAAAAG7O5kBn7969pVEHAAAAAAAA7pLd99ABAAAAAABA6bJ5hU5qauo9DeTv739P5wEAAAAAAKA4mwOdmJgYmUwmmwfas2ePzecAAAAAAACgJJsDnfHjx+tPf/qTzp8/r06dOikoKEiSdPr0aW3btk3+/v567rnn7in0AQAAAAAAwJ3ZHOhcunRJ+fn5Wrdundzd3Yu1DRo0SIMGDVJaWppeeOEFe9UIAAAAAACAX7A50Fm3bp2ef/75EmGOJHl6euqZZ57R6tWrCXRQKvLypIMHy24v7xo1LAoIsJTZeAAAAAAA3A2bA53MzEzl5OTcsj0nJ0eZmZn3VRRwK2lpJv3tb5XKbLy+fa8R6AAAAAAAKhyblzqEh4dr9erVOnbsWIm2o0ePavXq1QoLC7NLcQAAAAAAACjJ5hU6b7zxhoYMGaIBAwaoQYMGevjhhyVJZ86c0dGjR+Xu7q6RI0favVAAAAAAAABcZ3OgU6dOHa1atUqffvqpdu/erR07dkiS9e1Wffv2la+vr90LBQAAAAAAwHU2BzqS5OPjoxEjRti7FgAAAAAAANyF+3pd0OnTp3XkyBFdvXrVXvUAAAAAAADgDu4p0Nm6dauefvpp9ejRQy+99JJ1g+SMjAx1795d27dvt2uRAAAAAAAA+C+bA50dO3YoNjZWtWrV0tChQ2Wx/PeVzp6enqpdu7a2bNli1yIBAAAAAADwXzYHOp988okiIiI0f/58RUdHl2gPCwtTYmKiXYoDAAAAAABASTYHOqdOndJvfvObW7Z7eXkpIyPjvooCAAAAAADArdkc6JjNZuXk5NyyPTk5WZ6envdVFAAAAAAAAG7N5kCnefPm2rRpk/Lz80u0Xbx4UevXr1dkZKRdigMAAAAAAEBJNgc6Q4YMUVpamvr376+1a9fKZDLpm2++0YIFC9SrVy85ODho0KBBpVErAAAAAAAAdA+BTlBQkOLj4+Xj46P4+HhZLBatWrVKy5cvV0hIiOLi4uTv718atQIAAAAAAECSky2dCwsLdfHiRXl7e2v+/Pm6cuWKzp49q6KiIgUGBsrLy6u06gQAAAAAAMB/2LRCx2Kx6Nlnn9WmTZskSe7u7mrQoIHCw8MJcwAAAAAAAMqITYGOk5OTqlWrJpPJVFr1AAAAAAAA4A5s3kOna9eu2rRpk/Ly8kqjHgAAAAAAANyBTXvoSNc3RS4qKlKPHj0UHR2tgIAAubi4lOj3xBNP2KVAAAAAAAAAFGdzoBMbG2v980cffXTTPiaTiUAHAAAAAACglNxVoDNz5kxFR0erfv36WrRokSQpOztbLi4ucnR0LNUCAQAAAAAAUNxdBTpr1qxReHi46tevr6ZNmyojI0OdO3fW/Pnz1bRp01IrLiYmRikpKSWOt2nTRrNnz1ZcXJyWLFlSrM3b21tbt261frZYLIqPj9f69euVlZWlsLAwvfHGG6pbt26p1Q0AAAAAAFCabH7k6gaLxWLPOm5q6dKlKiwstH5OS0tTv3791LFjR+uxmjVrWlcNSSqxYmjZsmVatWqVJk6cqJo1a2rJkiUaOnSo1qxZI1dX11K/BwAAAAAAAHuz+S1XZcnLy0u+vr7W/3bt2iVXV1c9/vjj1j6Ojo7F+nh5eVnbLBaLVq9erX79+qlDhw6qW7euYmNjlZ2drW3btpXHLQEAAAAAANy3Ch3o/JLFYtHGjRv11FNPyWw2W48nJyerS5cuiomJ0fjx45WcnGxtO3funNLS0tSyZUvrMbPZrCZNmujbb78t0/oBAAAAAADs5a4fuUpOTtb3338vSbp69aok6aefflKVKlVu2j8sLMwO5f3X3r17de7cOcXExFiPhYeHa+LEiapVq5YuX76sjz/+WL///e+1evVqeXp6Ki0tTdL1fXV+ydvbWxcuXLjteImJiXatvyyUR81padWUmXnzn4HSkJNjUmZmTpmNl5aWrcTEi2U2HioeI/5bAFRkzCnAvphTgH0xp1CRBAcH37b9rgOd+Ph4xcfHFzs2c+bMEv0sFotMJpP27Nlzt5e+K+vXr1eDBg0UEhJiPda6detifcLDw/Xss89q8+bN6t27932Nd6cvrqJJTEy01nzunEkpKaYyGdfZ2SQPj0plMpYkVa6cX6bj+fiYFRzsWWbjoWL55bwCcP+YU4B9MacA+2JOwWjuKtCZMGFCaddxW+np6dq5c6dGjRp1235VqlRRnTp1dObMGUmSj4+P9Xx/f/9i17vR9muUkmLS8uXOZTJWx475ZTIOAAAAAAD4r7sKdLp27VraddzWpk2b5OzsrCeffPK2/fLy8nTq1Ck1a9ZMkhQQECAfHx/t3btXDRo0sPY5fPiwhg4dWup1AwAAAAAAlIZ7fm15WbFYLNqwYYOeeOKJEvv1zJ07V+3atZOfn58uX76sjz76SLm5uYqOjpYkmUwm9ezZU0uXLlWtWrUUFBSkjz/+WJUrV1anTp3K43YAAAAAAADuW4UPdA4ePKgzZ85oypQpJdouXLigN998UxkZGfLy8lJ4eLg++ugj1ahRw9qnX79+ysvL04wZM5SVlaWwsDDNnz9frq6uZXkbAAAAAAAAdlPhA53mzZtr3759N217++2373i+yWTS4MGDNXjwYHuXBgAAAAAAUC4cyrsAAAAAAAAA2IZABwAAAAAAwGAIdAAAAAAAAAyGQAcAAAAAAMBgCHQAAAAAAAAMhkAHAAAAAADAYAh0AAAAAAAADIZABwAAAAAAwGAIdAAAAAAAAAyGQAcAAAAAAMBgCHQAAAAAAAAMhkAHAAAAAADAYAh0AAAAAAAADIZABwAAAAAAwGAIdAAAAAAAAAyGQAcAAAAAAMBgCHQAAAAAAAAMhkAHAAAAAADAYAh0AAAAAAAADIZABwAAAAAAwGAIdAAAAAAAAAyGQAcAAAAAAMBgCHQAAAAAAAAMhkAHAAAAAADAYAh0AAAAAAAADIZA5/+3d+9RUdf5H8df48g9BGdQBDRRtFJQI5O0xAteqDQvlXZT2zY1j+ZmbWS2Jlprtq5oaXkJNVDznlJbKplZrptJWgnVdtLNBAEzB0dQLhrM7w9/ze7kJVFg+MrzcQ7nMN/vZz6f93eOH8GXn+/nCwAAAAAAYDAEOgAAAAAAAAZDoAMAAAAAAGAwBDoAAAAAAAAGQ6ADAAAAAABgMAQ6AAAAAAAABkOgAwAAAAAAYDAEOgAAAAAAAAZDoAMAAAAAAGAwBDoAAAAAAAAGQ6ADAAAAAABgMAQ6AAAAAAAABkOgAwAAAAAAYDAEOgAAAAAAAAZDoAMAAAAAAGAwBDoAAAAAAAAGQ6ADAAAAAABgMAQ6AAAAAAAABkOgAwAAAAAAYDAEOgAAAAAAAAZDoAMAAAAAAGAwBDoAAAAAAAAGQ6ADAAAAAABgMAQ6AAAAAAAABkOgAwAAAAAAYDAEOgAAAAAAAAZDoAMAAAAAAGAwBDoAAAAAAAAGU9/dBVzMG2+8ocWLF7scs1gs2rJliyTJ4XAoOTlZaWlpKioqUmRkpBISEhQREeFsX1hYqKSkJO3YsUOS1K1bNz399NPy9/evuQsBAAAAAACoQrU60JGk5s2ba8GCBc7XZrPZ+f2yZcu0cuVKTZkyRc2bN9fixYs1fvx4rVu3Tn5+fpKk559/XkeOHNGrr74qSZo+fboSExM1e/bsmr0QAAAAAACAKlLrb7kym80KCgpyfjVs2FDS2dU5q1ev1ogRIxQXF6eIiAglJiaquLhY6enpkqSDBw9q165deu6559S+fXu1b99ekyZN0s6dO3Xo0CF3XhYAAAAAAMBlq/WBTm5uru68804NHDhQf/nLX5SbmytJysvLk81mU+fOnZ1tvb29FR0drczMTElSVlaWfH191b59e2ebDh06yMfHx9kGAAAAAADAaGr1LVdRUVGaMmWKwsPDdfz4cS1dulSPPvqoVq9eLZvNJunsnjr/y2Kx6OjRo5Ikm82mwMBAmUwm53mTyaSGDRs6338h+/fvr+KrqX6/1myzNdKJE741MmZJiUknTpTUyFjuGM9mK9b+/T/X2HiofYz4dwFQmzGngKrFnAKqFnMKtUnr1q0ver5WBzq33nqry+uoqCgNHjxY77//vtq1a1etY//eB1fb7N+/31lzYWE9BQR41si4Pj5nFBDgUSNjuWM8q9VbrVsH1th4qF3+d14BuHLMKaBqMaeAqsWcgtHU6kDnt3x9fdWyZUvl5OSoR48ekqSCggI1adLE2aagoEBWq1WSZLVaZbfb5XA4nKt0HA6Hjh8/7mwDXExZmbR3b83cmRgS4lBoqKNGxgIAAAAAGJuhAp2ysjL9+OOP6tixo0JDQ2W1WrV79261bdvWef6rr77S+PHjJUnt2rVTcXGxsrKynPvoZGVlqaSkxGVfHeBCbDaTtm2rmRVBw4efJtABAAAAAFySWh3ovPrqq4qNjVVwcLCOHz+uJUuWqLS0VP369ZPJZNL999+vlJQUhYeH69prr9XSpUvl4+Oj+Ph4SVKLFi3UpUsXzZgxQ5MmTZIkzZgxQ127dlXz5s3deWkAAAAAAACXrVYHOkePHtXkyZNlt9vVsGFDRUVFacmSJQoJCZEkjRgxQmVlZZo5c6aKiooUGRmpefPmyc/Pz9nHiy++qFmzZumJJ56QJMXGxiohIcEt1wMAAAAAAFAVanWgM3369IueN5lMGj16tEaPHn3BNg0aNNALL7xQ1aUBAAAAAAC4Tc3s9goAAAAAAIAqQ6ADAAAAAABgMAQ6AAAAAAAABkOgAwAAAAAAYDAEOgAAAAAAAAZDoAMAAAAAAGAwBDoAAAAAAAAGQ6ADAAAAAABgMAQ6AAAAAAAABkOgAwAAAAAAYDAEOgAAAAAAAAZDoAMAAAAAAGAwBDoAAAAAAAAGQ6ADAAAAAABgMAQ6AAAAAAAABlPf3QUAqJ1MeXky5edXS9+OkBA5QkOrpW8AAAAAqAsIdACclyk/X57Ll1dL36eHDyfQAQAAAIArwC1XAAAAAAAABkOgAwAAAAAAYDAEOgAAAAAAAAZDoAMAAAAAAGAwBDoAAAAAAAAGQ6ADAAAAAABgMAQ6AAAAAAAABkOgAwAAAAAAYDAEOgAAAAAAAAZDoAMAAAAAAGAw9d1dAAD3yMszKT/fdMHzTfNM8vv5wucry9fXIV+/KusOAAAAAOo0Ah2gjsrPN2n5cs8Lnu8fbJZ1f9Ut4mvdukK+fo4q6w8AAAAA6jJuuQIAAAAAADAYAh0AAAAAAACDIdABAAAAAAAwGAIdAAAAAAAAg2FTZKCWKCuT9u6tuYy1qKjqnmAFAAAAAKhZBDpALWGzmbRtm0eNjder15kaGwsAAAAAULW45QoAAAAAAMBgCHQAAAAAAAAMhkAHAAAAAADAYAh0AAAAAAAADIZABwAAAAAAwGAIdAAAAAAAAAyGQAcAAAAAAMBgCHQAAAAAAAAMhkAHAAAAAADAYAh0AAAAAAAADIZABwAAAAAAwGAIdAAAAAAAAAyGQAcAAAAAAMBgCHQAAAAAAAAMhkAHAAAAAADAYAh0AAAAAAAADIZABwAAAAAAwGAIdAAAAAAAAAymvrsLuJiUlBRt375d2dnZ8vDwUFRUlMaNG6eIaEn5jAAAG9VJREFUiAhnm2nTpun99993eV9UVJSWLl3qfH369Gm9+uqr+uCDD1RWVqZOnTrpmWeeUXBwcI1dCwAAAAAAQFWp1YHO3r17de+996pt27ZyOBxatGiRxo0bpzVr1iggIMDZLiYmRlOnTnW+9vDwcOln9uzZ2rFjh/76178qICBAr7zyip566iktW7ZMZrO5pi4HAAAAAACgStTqQGfevHkur6dNm6a4uDhlZmYqNjbWedzDw0NBQUHn7ePkyZN69913NWXKFN1yyy3OfgYMGKCMjAx16dKl+i4AAAAAAACgGhhqD53i4mJVVFTI39/f5fi+ffsUHx+ve+65R9OnT1dBQYHz3L///W/98ssvzjBHkoKDgxUeHq6srKwaqx0AAAAAAKCq1OoVOr+VlJSk6667Tu3atXMe69Kli3r27KnQ0FDl5+drwYIFGjt2rJYtWyZPT0/ZbDaZzWYFBga69GWxWGSz2S441v79+6vtOqrLrzXbbI104oRvjYxZUmLSiRMlNTLW1T5ebbu2soAynT59usrGKy37RYUnSiVJxTabfjbIHDPi3wVAbcacAqoWcwqoWswp1CatW7e+6HnDBDpz5szRvn37lJyc7LLvTd++fZ3ft2rVSjfccIMGDBigf/3rX+rZs+dlj/d7H1xts3//fmfNhYX1FBDgWSPj+vicUUCAx+83ZLxaNdaljOfl7SVPz6r7c+TtVV8NArzOfm+1KtAAc+x/5xWAK8ecAqoWcwqoWswpGI0hbrmaPXu2PvjgA82fP19hYWEXbduoUSM1btxY2dnZkiSr1ary8nLZ7XaXdgUFBbJardVWMwAAAAAAQHWp9YFOUlKSM8wJDw//3fZ2u10///yzc5PkNm3aqH79+tq9e7ezzU8//aQff/zR5dYtAAAAAAAAo6jVt1zNnDlTmzdv1syZM+Xv769jx45Jknx9feXr66vi4mIlJyerZ8+eCgoKUn5+vl5//XVZLBb16NFDknTNNddowIABeu2112SxWJyPLW/VqpViYmLceHUAAAAAAACXp1YHOuvXr5ckjRs3zuX4yJEjNXr0aNWrV08HDhzQpk2bVFRUpKCgIHXs2FEzZsyQn5+fs/1TTz0ls9ms5557TmVlZerUqZOmTp3qshcPAAAAAACAUdTqQCcjI+Oi5729vTVv3rzf7cfT01MJCQlKSEioqtIAAAAAAADcptbvoQMAAAAAAABXBDoAAAAAAAAGQ6ADAAAAAABgMAQ6AAAAAAAABkOgAwAAAAAAYDAEOgAAAAAAAAZDoAMAAAAAAGAwBDoAAAAAAAAGQ6ADAAAAAABgMAQ6AAAAAAAABkOgAwAAAAAAYDAEOgAAAAAAAAZDoAMAAAAAAGAwBDoAAAAAAAAGQ6ADAAAAAABgMAQ6AAAAAAAABkOgAwAAAAAAYDAEOgAAAAAAAAZT390FAKgbyiukYz+bJEmn8kw6XI15ckiIQ6GhjmrrHwAAAADcjUAHQI0oK5Vyc8+GOLavzXpvm2e1jTV8+GkCHQAAAABXNW65AgAAAAAAMBgCHQAAAAAAAIMh0AEAAAAAADAYAh0AAAAAAACDIdABAAAAAAAwGJ5yBRhcl2Y5sp7Oq/T7Wpwsl0+w+YLnmzY4oZIrKQwAAAAAUG0IdACDs57Ok3Xjskq/zz+sQtbcCy/S8xkWR6ADAAAAALUUt1wBAAAAAAAYDIEOAAAAAACAwRDoAAAAAAAAGAyBDgAAAAAAgMEQ6AAAAAAAABgMgQ4AAAAAAIDBEOgAAAAAAAAYDIEOAAAAAACAwRDoAAAAAAAAGAyBDgAAAAAAgMEQ6AAAAAAAABhMfXcXAKDuCWtUqv7aXeX92jxDtSunWZX3CwAAAAC1DYEOgBrnW3xM1o0fVX3Hg0dIItABAAAAcPXjlisAAAAAAACDIdABAAAAAAAwGAIdAAAAAAAAg2EPHaAGdGmWI+vpvIu2aXGyXD7B5kr33bTBCZVcbmEAAAAAAEMi0AFqgPV0nqwbl120jX9Yhay5lV805zMsjkAHAAAAAOoYbrkCAAAAAAAwGAIdAAAAAAAAgyHQAQAAAAAAMBj20AFw1Skrk/buvfK82mZrpMLC3+8nJMSh0FDHFY8HAAAAAJeKQAfAVcdmM2nbNo8r7ufECV8FBHj+brvhw08T6AAAAACoUdxyBQAAAAAAYDCs0IHhdGmWI+vpvGrpO7CJt+xHSp2vW5wsl0+w+Yr7bdrgBI8WBwAAAABUGQIdGI71dJ6sG5dVS9+WYXEyb/zI+do/rELW3CtfyOYzLI5AB4aUl2dSfr6pxsZjPyIAAADg0tSpQGf9+vVavny5bDabWrZsqSeffFLR0dHuLuuqVR0raVqcLNeZBicJR1CrVNUmzJfKx0cqqaFJUFRkUlrale9HdKnYjwgAAAC4NHUm0Nm6dauSkpI0ceJEdejQQevXr9eECRO0Zs0aNWnSxN3lXZWqYyWNf1iFSnr2JtBBrVJVmzBfql69ztTYeL16namRcX5V0+EYK4IAAABgVHUm0Fm5cqX69++vQYMGSZISEhL02Wef6e2339a4cePcXB0AQKr5cGzo0NM1dksZ4REAAACqkslut1/1v12eOXNG3bp104svvqjevXs7j8+cOVP/+c9/tGjRIjdWBwAAAAAAUDl14rHldrtd5eXlslgsLsctFotsNpubqgIAAAAAALg8dSLQAQAAAAAAuJrUiUAnMDBQZrNZBQUFLscLCgpktVrdVBUAAAAAAMDlqROBjoeHh2644QZlZGS4HN+9e7fat2/vpqoAAAAAAAAuT515ytWDDz6oxMREtW3bVh06dNCGDRt07Ngx3X333e4uDQAAAAAAoFLqTKDTp08fnThxQm+++aaOHTumiIgIzZkzRyEhIe4u7YqtX79ey5cvl81mU8uWLfXkk08qOjra3WUBtU5KSoq2b9+u7OxseXh4KCoqSuPGjVNERISzjcPhUHJystLS0lRUVKTIyEglJCS4tCksLFRSUpJ27NghSerWrZuefvpp+fv71/g1AbVFSkqK5s+fryFDhighIUES8wm4HMeOHdNrr72mTz/9VMXFxQoLC9PEiRN10003SWJeAZVRXl6u5ORkbd68WTabTVarVbfffrtGjRql+vXP/lOYOQUjqxOPLb+abd26VVOmTNHEiRPVoUMHrV+/Xu+9957WrFmjJk2auLs8oFYZP368+vbtq7Zt28rhcGjRokXKysrSmjVrFBAQIElKTU3Vm2++qSlTpqh58+ZavHix9u3bp3Xr1snPz0+S9MQTT+jIkSP6y1/+IkmaPn26wsLCNHv2bLddG+BOWVlZmjx5svz8/BQdHe0MdJhPQOUUFRVp+PDh6tChg4YOHarAwEDl5uaqUaNGatGihSTmFVAZb775pt566y0lJiYqIiJCBw4c0LRp0/Tggw/q0UcflcScgrHViT10rmYrV65U//79NWjQILVo0UIJCQkKCgrS22+/7e7SgFpn3rx5uuuuuxQREaFWrVpp2rRpstvtyszMlHT2f2hWr16tESNGKC4uThEREUpMTFRxcbHS09MlSQcPHtSuXbv03HPPqX379mrfvr0mTZqknTt36tChQ+68PMAtTp48qSlTpuj5559XgwYNnMeZT0DlLV++XEFBQZo2bZoiIyMVFhammJgYZ5jDvAIqJzMzU127dlVsbKxCQ0PVrVs3xcbG6uuvv5bEnILxEegY2JkzZ/Tdd9/plltucTl+yy23OP+BCuDCiouLVVFR4Vwum5eXJ5vNps6dOzvbeHt7Kzo62jmnsrKy5Ovr67KheocOHeTj48O8Q5300ksvKS4uTjfffLPLceYTUHmffPKJIiMj9dxzzyk+Pl4PPfSQ1q5dK4fj7IJ65hVQOTfeeKP27t2rH3/8UZL0ww8/aM+ePbrtttskMadgfHVmD52rkd1uV3l5uSwWi8txi8VyzhO9AJwrKSlJ1113ndq1aydJstlsknTeOXX06FFnm8DAQJlMJud5k8mkhg0bOt8P1BVpaWk6fPiwXnjhhXPOMZ+AysvNzdXbb7+tBx54QA8//LC+//57zZo1S5I0dOhQ5hVQSSNGjNCpU6d03333qV69eiovL9cjjzyie++9VxI/q2B8BDoA6qQ5c+Zo3759Sk5Oltlsdnc5gOEcOnRI8+fPV3JysnNjSQBXpqKiQm3atNG4ceMkSddff71ycnK0fv16DR061M3VAcazdetWbdq0SS+++KJatmyp77//XrNnz1ZoaKgGDhzo7vKAK8ZvYAYWGBgos9msgoICl+MFBQWyWq1uqgqo/WbPnq2tW7dqwYIFCgsLcx7/dd4UFBS4bCr+v3PKarXKbrfL4XA4/6fG4XDo+PHjzDvUKVlZWbLb7br//vudx8rLy/Xll19qw4YNWr16tSTmE1AZQUFBzv1yfhUeHq4jR45I4ucUUFlz587VsGHD1LdvX0lSq1atlJ+fr9TUVA0cOJA5BcNjDx0D8/Dw0A033HDO7VW7d+92uccTwH8lJSXpgw8+0Pz58xUeHu5yLjQ0VFarVbt373YeKysr01dffeWcU+3atVNxcbGysrKcbbKyslRSUsK8Q53SvXt3rVq1SitWrHB+tWnTRn369NGKFSt07bXXMp+ASmrfvv05m6xmZ2crJCREEj+ngMoqLS1VvXqu/+Q1m82qqKiQxJyC8ZmfffbZqe4uApfPz89Pb7zxhqxWq7y8vLR06VJ9+eWXev75550bvQI4a+bMmdq0aZNmzJih4OBgFRcXq7i4WNLZgNRkMqm8vFypqalq3ry5Kioq9Morr+jYsWOaNGmSPD091bBhQ3399ddKT0/Xddddp59++kkzZsxQZGSk7rvvPjdfIVBzvLy8ZLFYXL7S09MVEhKi/v37M5+Ay9CkSRMtXrxY9erVU1BQkD7//HMtWLBADz/8sCIjI5lXQCUdPHhQmzdv1rXXXqv69etrz549WrBggfr06aPOnTszp2B4Jrvd7nB3Ebgy69ev1/Lly3Xs2DFFRERowoQJuummm9xdFlDrxMTEnPf4yJEjNXr0aElnl9AmJydr48aNKioqUmRkpJ555hlFREQ42xcWFmrWrFn65z//KUmKjY1VQkICISrqvDFjxigiIkIJCQmSmE/A5di5c6fmz5+v7OxsBQcHa+jQoRo6dKjLrR7MK+DSnDp1SosWLdLHH3/svEWqb9++evTRR+Xl5SWJOQVjI9ABAAAAAAAwGPbQAQAAAAAAMBgCHQAAAAAAAIMh0AEAAAAAADAYAh0AAAAAAACDIdABAAAAAAAwGAIdAAAAAAAAgyHQAQAAwGXLy8tTTEyMUlNT3V0KAAB1CoEOAACoMu+9955iYmIUExOjL7/88rxt7r77bsXExGjMmDE1XF3tNnDgQP3pT39ydxkXtGXLFq1atcrdZQAAgP9HoAMAAKqcl5eX0tPTzzmelZWlw4cPy8vLyw1V4Uqkp6dr9erV7i4DAAD8PwIdAABQ5W699VZt27ZNv/zyi8vx9PR0hYeHKywszE2VAQAAXB3qu7sAAABw9enbt68+/vhj7dq1S7GxsZKk8vJyffjhhxoyZIi2bt16znscDofWrl2rtLQ05eTkyM/PT127dtX48eMVGBjobPfdd99p4cKF+vbbb1VcXCyLxaLo6GhNmjRJ3t7ekqStW7dqxYoVys7OlsPhUOPGjRUfH69HH31UknTixAmlpqbqs88+U15enhwOh9q0aaPHHntM0dHRLnXZ7XbNmTNHO3bskMlkUrdu3fTggw9q2LBhmjJlivr37+9se+jQIS1cuFCff/65SktLFR4erkceeUS9evWqss/211uffvjhB3l6eqpTp04aP368S0g2ZswY2Ww2/e1vf9Pf//53ff3112rQoIHuu+8+jRgxwqW//Px8zZo1S59//rl8fHwUHx+vLl266IknntCCBQvUsWNHjRkzRl988YUkKSYmxvnejIwMl77S0tKUmpqqo0ePqlWrVpo4caLatm1bZdcOAAD+i0AHAABUucaNG+vGG29Uenq6M9DJyMhQQUGB4uPjzxvovPzyy3r33XfVr18/DRkyRD/99JPWrl2rb7/9VikpKfLy8tLx48f1+OOPKzAwUMOHD5e/v7+OHj2qHTt2qLS0VN7e3srIyNDkyZN18803a+zYsapXr54OHTqkr776yjlWbm6utm3bpt69eyssLExFRUV699139fjjjyslJUWtW7eWJFVUVOjPf/6zvvnmG91zzz0KDw/Xjh07NHXq1HPqP3jwoEaOHCmr1arhw4fL19dXH3/8sSZNmqRp06bpjjvuuOLPNTU1VfPnz1dcXJzuuusuFRYWat26dRo1apTeeustNWzY0Nn25MmTmjBhgnr06KFevXrpo48+0muvvaZWrVrp1ltvlSSVlJRo7NixOnbsmO677z41atRIW7Zs0Z49e1zGfeSRR3Ty5EkdPXpUTz755Hlr27p1q0pKSjR48GCZTCYtX75czzzzjNLS0lS/Pr9yAgBQ1fjpCgAAqkV8fLxeeeUVlZSUyMfHR1u2bFFUVJSaNm16TtvMzExt3LhRiYmJ6tevn/N4ly5dNHr0aG3atEmDBw9WZmamCgsLNXfuXJeVH6NHj3Z+v3PnTvn6+mru3Lkym83nra1Vq1bauHGj6tX7793ngwcP1tChQ7VmzRpNnjxZkvTJJ58oKytLTzzxhB566CFJ0j333KPHH3/8nD5nz56toKAgpaamOlcKDRkyROPHj9frr7+u22+/XSaTqTIfoYsjR45o4cKFGjVqlEaOHOk83rdvX91///1atWqVxo4d6zxus9lcPs+BAwdqwIABevfdd52BzoYNG5Sbm6uXX35ZcXFxkqRBgwZp+PDhLmPfcsstWr16tYqKii4YTP300096++231aBBA0lS8+bN9fTTT7us0gIAAFWHPXQAAEC16NWrl3755Rd98sknKi0t1Y4dOxQfH3/eth9++KF8fX3VpUsX2e1251d4eLgsFov27t0rSbrmmmsknQ1tfrs/z6+uueYalZaWavfu3ReszdPT0xnmlJWVyW63q6KiQm3atNF3333nbLdr1y6ZzWYNHjzYeaxevXoaMmSIS38nTpxQRkaGevfurdLSUpdr6Ny5s44ePars7OxL+NQubPv27SovL1efPn1c+r/mmmsUERFxzqoaLy8vl/DFw8NDkZGRys3NdR777LPPZLVa1bNnT5f3DRw4sNL1xcXFOcMcSbrxxhslnX2sOQAAqHqs0AEAANUiICBAnTt31pYtW2Q2m1VaWqo+ffqct212draKi4t1++23n/d8QUGBJOmmm25SXFycFi9erFWrVik6OlrdunVTfHy8fHx8JEn33nuvtm3bpgkTJqhRo0bq1KmT4uLiFBsb61whU1FRoeXLl2vjxo3nBA6hoaHO7/Pz82WxWOTr6+vS5rerjA4fPiyHw6Hk5GQlJydf8BqaN29+oY/rd/0aCP02TPrVbzeabty4scsKJEny9/fX/v37na/z8/MVFhZ2zsqhZs2aVbq+4OBgl9e/hjuFhYWV7gsAAPw+Ah0AAFBt4uPjNXXqVJ06dUqdOnWSxWI5b7uKigoFBARo+vTp5z3v7+8vSTKZTHr55Zf1zTffaOfOndq9e7deeuklpaSkaOnSpbJYLLJYLFqxYoUyMjL06aef6rPPPtOmTZvUtWtXJSUlyWQyKSUlRQsXLlS/fv00ZswYBQQEyGw2KyUlxWUFy6WqqKiQJD3wwAO67bbbztsmIiKi0v2eb4xXX331vLeS/fZR8L8Nc6rbhW5vczgcNVoHAAB1BYEOAACoNt27d5enp6f27dunxMTEC7Zr2rSpMjIyFBUVdc5qmPOJjIxUZGSkHnvsMX366aeaMGGC0tLS9Mc//lHS2duLbrvtNt12221yOBx6/fXXtWzZMmVmZqpDhw7atm2bOnbseE5Nb7zxhsvrkJAQ7dmzR8XFxS51HT582KXdr6tjzGazy1OgqtKvq4KCg4PVsmXLKukzJCREBw4ckMPhcFmlk5OTc07bK9n/BwAAVD320AEAANXG29tbEydO1MiRI9WjR48Ltuvdu7cqKiq0ePHic86Vl5c7b9spLCw8Z8XH9ddfL+nsU52ks48Z/18mk8nZpqioSNLZ4OW3/WRmZiorK8vlWOfOnVVeXq6NGzc6j1VUVGjdunUu7SwWizp27Kh33nlHR48ePecajh8/foErv3RxcXEym81asmTJeVe9/Pa6L0Xnzp1ls9m0fft257GysjK9884757T18fFRUVERK24AAKglWKEDAACq1Z133vm7bW666SYNGTJEK1as0IEDB9S5c2d5enoqJydHH330kR577DH1799f77//vtavX6/u3buradOmKisr0z/+8Q+ZzWbnU5qmT58uu92uTp06KTg4WD///LPWrVunoKAgRUdHS5JiY2OVnJysxMRERUdHKzs7W2lpaWrRooVKSkqcdXXv3l2RkZGaN2+e8vLynI8tP9++MM8++6xGjhypBx98UIMGDVLTpk1VUFCgb775RgcPHtSGDRt+93PIzc3VkiVLzjnesmVL9ezZU+PGjdPcuXOVn5+v7t27y9/fX3l5edqxY4d69+7t8rSvSzF48GCtXbtWiYmJ+vbbb52PLff09JTkuiqnTZs22rp1q2bPnq3IyEjVq1dPffv2rdR4AACg6hDoAACAWiEhIUHXX3+9NmzYoAULFqh+/foKDg5W7969dfPNN0s6G/x8++232rZtm2w2m/z8/HTdddfp6aefVlRUlCTpjjvu0DvvvKMNGzaoqKhIFotFt956q0aOHCk/Pz9J0h/+8AeVlpYqPT1dH330kVq2bKnp06frgw8+0BdffOGsyWw2a86cOUpKStKmTZtkMpnUo0cPjRw5UqNGjXLZt6Z58+ZKTU3V4sWLtWnTJtntdjVs2FCtW7fWY489dkmfQU5OjhYtWnTO8T59+qhnz54aNmyYmjVrppUrV+rNN99URUWFGjdurJtvvlm9evWq9Gfu6+ur+fPna9asWVqzZo18fX115513ql27dnr22WedwY50drPpAwcOaPPmzVq7dq0cDgeBDgAAbmSy2+2smwUAAKiEjz/+WM8884ySk5PVoUMHd5dT5VatWqU5c+bovffeU+PGjd1dDgAAOA/20AEAALiI0tJSl9fl5eVau3at/Pz8nHvzGNlvr6+srEwbN25Us2bNCHMAAKjFuOUKAADgImbNmqWysjK1a9dOZ86c0fbt25WZmamxY8fK29vb3eVdsYkTJ6pJkyZq3bq1Tp06pc2bN+vHH3/UCy+84O7SAADARXDLFQAAwEVs2bJFK1euVE5Ojk6fPq2mTZvqnnvu0dChQ91dWpVYtWqV3nnnHeXn56uiokItWrTQ8OHD1adPH3eXBgAALoJABwAAAAAAwGDYQwcAAAAAAMBgCHQAAAAAAAAMhkAHAAAAAADAYAh0AAAAAAAADIZABwAAAAAAwGAIdAAAAAAAAAzm/wAiXzUEqN9+dQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 1224x576 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 286
        },
        "id": "pEax9b3WlHCT",
        "outputId": "f058c836-36a2-4dd2-9b2a-001ce8540432"
      },
      "source": [
        "sms[sms.label=='ham'].describe()"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>label_num</th>\n",
              "      <th>message_len</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>4825.0</td>\n",
              "      <td>4825.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>0.0</td>\n",
              "      <td>71.023627</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>0.0</td>\n",
              "      <td>58.016023</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>0.0</td>\n",
              "      <td>2.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>0.0</td>\n",
              "      <td>33.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>0.0</td>\n",
              "      <td>52.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>0.0</td>\n",
              "      <td>92.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>0.0</td>\n",
              "      <td>910.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       label_num  message_len\n",
              "count     4825.0  4825.000000\n",
              "mean         0.0    71.023627\n",
              "std          0.0    58.016023\n",
              "min          0.0     2.000000\n",
              "25%          0.0    33.000000\n",
              "50%          0.0    52.000000\n",
              "75%          0.0    92.000000\n",
              "max          0.0   910.000000"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 286
        },
        "id": "N8qmqZyQlKcP",
        "outputId": "e11b5776-6550-414f-b815-e1b220b68408"
      },
      "source": [
        "sms[sms.label=='spam'].describe()"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>label_num</th>\n",
              "      <th>message_len</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>747.0</td>\n",
              "      <td>747.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>1.0</td>\n",
              "      <td>138.866131</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>0.0</td>\n",
              "      <td>29.183082</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>1.0</td>\n",
              "      <td>13.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>1.0</td>\n",
              "      <td>132.500000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>1.0</td>\n",
              "      <td>149.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>1.0</td>\n",
              "      <td>157.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>1.0</td>\n",
              "      <td>224.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       label_num  message_len\n",
              "count      747.0   747.000000\n",
              "mean         1.0   138.866131\n",
              "std          0.0    29.183082\n",
              "min          1.0    13.000000\n",
              "25%          1.0   132.500000\n",
              "50%          1.0   149.000000\n",
              "75%          1.0   157.000000\n",
              "max          1.0   224.000000"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 118
        },
        "id": "kUcvNzFqlNEo",
        "outputId": "79a639f0-b7a2-4282-f31f-504f8122b851"
      },
      "source": [
        "sms[sms.message_len == 910].message.iloc[0]"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "\"For me the love should start with attraction.i should feel that I need her every time around me.she should be the first thing which comes in my thoughts.I would start the day and end it with her.she should be there every time I dream.love will be then when my every breath has her name.my life should happen around her.my life will be named to her.I would cry for her.will give all my happiness and take all her sorrows.I will be ready to fight with anyone for her.I will be in love when I will be doing the craziest things for her.love will be when I don't have to proove anyone that my girl is the most beautiful lady on the whole planet.I will always be singing praises for her.love will be when I start up making chicken curry and end up makiing sambar.life will be the most beautiful then.will get every morning and thank god for the day because she is with me.I would like to say a lot..will tell later..\""
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7a_J_HjxlV8W"
      },
      "source": [
        "## 4. Text Pre-processing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 196
        },
        "id": "ABcHa1mslPuH",
        "outputId": "d61dd583-dedb-4fd5-e7a8-23280ad46bb7"
      },
      "source": [
        "import string\n",
        "from nltk.corpus import stopwords\n",
        "\n",
        "def text_process(mess):\n",
        "    \"\"\"\n",
        "    Takes in a string of text, then performs the following:\n",
        "    1. Remove all punctuation\n",
        "    2. Remove all stopwords\n",
        "    3. Returns a list of the cleaned text\n",
        "    \"\"\"\n",
        "    STOPWORDS = stopwords.words('english') + ['u', 'ü', 'ur', '4', '2', 'im', 'dont', 'doin', 'ure']\n",
        "    # Check characters to see if they are in punctuation\n",
        "    nopunc = [char for char in mess if char not in string.punctuation]\n",
        "\n",
        "    # Join the characters again to form the string.\n",
        "    nopunc = ''.join(nopunc)\n",
        "    \n",
        "    # Now just remove any stopwords\n",
        "    return ' '.join([word for word in nopunc.split() if word.lower() not in STOPWORDS])\n",
        "    \n",
        "sms.head()"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>label</th>\n",
              "      <th>message</th>\n",
              "      <th>label_num</th>\n",
              "      <th>message_len</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ham</td>\n",
              "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
              "      <td>0</td>\n",
              "      <td>111</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ham</td>\n",
              "      <td>Ok lar... Joking wif u oni...</td>\n",
              "      <td>0</td>\n",
              "      <td>29</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>spam</td>\n",
              "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
              "      <td>1</td>\n",
              "      <td>155</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ham</td>\n",
              "      <td>U dun say so early hor... U c already then say...</td>\n",
              "      <td>0</td>\n",
              "      <td>49</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ham</td>\n",
              "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
              "      <td>0</td>\n",
              "      <td>61</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  label  ... message_len\n",
              "0   ham  ...         111\n",
              "1   ham  ...          29\n",
              "2  spam  ...         155\n",
              "3   ham  ...          49\n",
              "4   ham  ...          61\n",
              "\n",
              "[5 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 196
        },
        "id": "bujuGVqSlcFn",
        "outputId": "3763a794-c1b9-4262-c97a-5dd7656a7330"
      },
      "source": [
        "sms['clean_msg'] = sms.message.apply(text_process)\n",
        "sms.head()"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>label</th>\n",
              "      <th>message</th>\n",
              "      <th>label_num</th>\n",
              "      <th>message_len</th>\n",
              "      <th>clean_msg</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ham</td>\n",
              "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
              "      <td>0</td>\n",
              "      <td>111</td>\n",
              "      <td>Go jurong point crazy Available bugis n great ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ham</td>\n",
              "      <td>Ok lar... Joking wif u oni...</td>\n",
              "      <td>0</td>\n",
              "      <td>29</td>\n",
              "      <td>Ok lar Joking wif oni</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>spam</td>\n",
              "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
              "      <td>1</td>\n",
              "      <td>155</td>\n",
              "      <td>Free entry wkly comp win FA Cup final tkts 21s...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ham</td>\n",
              "      <td>U dun say so early hor... U c already then say...</td>\n",
              "      <td>0</td>\n",
              "      <td>49</td>\n",
              "      <td>dun say early hor c already say</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ham</td>\n",
              "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
              "      <td>0</td>\n",
              "      <td>61</td>\n",
              "      <td>Nah think goes usf lives around though</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  label  ...                                          clean_msg\n",
              "0   ham  ...  Go jurong point crazy Available bugis n great ...\n",
              "1   ham  ...                              Ok lar Joking wif oni\n",
              "2  spam  ...  Free entry wkly comp win FA Cup final tkts 21s...\n",
              "3   ham  ...                    dun say early hor c already say\n",
              "4   ham  ...             Nah think goes usf lives around though\n",
              "\n",
              "[5 rows x 5 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "r_n-DKwelfan",
        "outputId": "3feee23f-e322-4cb1-82f5-273229cedaa1"
      },
      "source": [
        "type(stopwords.words('english'))"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "list"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yNrGdgljlsAx",
        "outputId": "38f42f2e-17ae-4ee4-e4c2-43ac5a10d4cb"
      },
      "source": [
        "from collections import Counter\n",
        "\n",
        "words = sms[sms.label=='ham'].clean_msg.apply(lambda x: [word.lower() for word in x.split()])\n",
        "ham_words = Counter()\n",
        "\n",
        "for msg in words:\n",
        "    ham_words.update(msg)\n",
        "    \n",
        "print(ham_words.most_common(50))"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[('get', 303), ('ltgt', 276), ('ok', 272), ('go', 247), ('ill', 236), ('know', 232), ('got', 231), ('like', 229), ('call', 229), ('come', 224), ('good', 222), ('time', 189), ('day', 187), ('love', 185), ('going', 167), ('want', 163), ('one', 162), ('home', 160), ('lor', 160), ('need', 156), ('sorry', 153), ('still', 146), ('see', 137), ('n', 134), ('later', 134), ('da', 131), ('r', 131), ('back', 129), ('think', 128), ('well', 126), ('today', 125), ('send', 123), ('tell', 121), ('cant', 118), ('ì', 117), ('hi', 117), ('take', 112), ('much', 112), ('oh', 111), ('night', 107), ('hey', 106), ('happy', 105), ('great', 100), ('way', 100), ('hope', 99), ('pls', 98), ('work', 96), ('wat', 95), ('thats', 94), ('dear', 94)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3KVBmlrIlujl",
        "outputId": "fb138d9a-51fc-458a-99b5-74d6d81629d6"
      },
      "source": [
        "words = sms[sms.label=='spam'].clean_msg.apply(lambda x: [word.lower() for word in x.split()])\n",
        "spam_words = Counter()\n",
        "\n",
        "for msg in words:\n",
        "    spam_words.update(msg)\n",
        "    \n",
        "print(spam_words.most_common(50))"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[('call', 347), ('free', 216), ('txt', 150), ('mobile', 123), ('text', 120), ('claim', 113), ('stop', 113), ('reply', 101), ('prize', 92), ('get', 83), ('new', 69), ('send', 67), ('nokia', 65), ('urgent', 63), ('cash', 62), ('win', 60), ('contact', 56), ('service', 55), ('please', 52), ('guaranteed', 50), ('customer', 49), ('16', 49), ('week', 49), ('tone', 48), ('per', 46), ('phone', 45), ('18', 43), ('chat', 42), ('awarded', 38), ('draw', 38), ('latest', 36), ('å£1000', 35), ('line', 35), ('150ppm', 34), ('mins', 34), ('receive', 33), ('camera', 33), ('1', 33), ('every', 33), ('message', 32), ('holiday', 32), ('landline', 32), ('shows', 31), ('å£2000', 31), ('go', 31), ('box', 30), ('number', 30), ('apply', 29), ('code', 29), ('live', 29)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IeKB4sQtlwxj",
        "outputId": "553294bd-30e7-4e8d-cacc-185a42665a8d"
      },
      "source": [
        "# how to define X and y (from the SMS data) for use with COUNTVECTORIZER\n",
        "X = sms.clean_msg\n",
        "y = sms.label_num\n",
        "print(X.shape)\n",
        "print(y.shape)"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(5572,)\n",
            "(5572,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qwXh2kDel0a3",
        "outputId": "c6395419-8fac-47cd-c02b-a97b964d63e1"
      },
      "source": [
        "# split X and y into training and testing sets \n",
        "from sklearn.model_selection import train_test_split\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)\n",
        "print(X_train.shape)\n",
        "print(X_test.shape)\n",
        "print(y_train.shape)\n",
        "print(y_test.shape)"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(4179,)\n",
            "(1393,)\n",
            "(4179,)\n",
            "(1393,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9CdsKNV9l6Ij",
        "outputId": "554084da-ec1b-4311-caee-e1ef4faea2dd"
      },
      "source": [
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "\n",
        "# instantiate the vectorizer\n",
        "vect = CountVectorizer()\n",
        "vect.fit(X_train)"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
              "                dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
              "                lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
              "                ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
              "                strip_accents=None, token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
              "                tokenizer=None, vocabulary=None)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RfkRg3e-l8Wy"
      },
      "source": [
        "# learn training data vocabulary, then use it to create a document-term matrix\n",
        "X_train_dtm = vect.transform(X_train)"
      ],
      "execution_count": 35,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1AoyWLyhl_aR"
      },
      "source": [
        "# equivalently: combine fit and transform into a single step\n",
        "X_train_dtm = vect.fit_transform(X_train)"
      ],
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dnx4f9FrmCyI",
        "outputId": "fe20bf55-55ca-4278-c068-0f9d3ea64927"
      },
      "source": [
        "# examine the document-term matrix\n",
        "X_train_dtm"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<4179x7996 sparse matrix of type '<class 'numpy.int64'>'\n",
              "\twith 34796 stored elements in Compressed Sparse Row format>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "v5Tc7njsmE-7",
        "outputId": "e2fa087f-06f9-494f-d4ad-3f9662fdc624"
      },
      "source": [
        "# transform testing data (using fitted vocabulary) into a document-term matrix\n",
        "X_test_dtm = vect.transform(X_test)\n",
        "X_test_dtm"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<1393x7996 sparse matrix of type '<class 'numpy.int64'>'\n",
              "\twith 9971 stored elements in Compressed Sparse Row format>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KLXHpl3cmHSF",
        "outputId": "bd78b3f7-d3c8-45f4-e056-525148d962bb"
      },
      "source": [
        "from sklearn.feature_extraction.text import TfidfTransformer\n",
        "\n",
        "tfidf_transformer = TfidfTransformer()\n",
        "tfidf_transformer.fit(X_train_dtm)\n",
        "tfidf_transformer.transform(X_train_dtm)"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<4179x7996 sparse matrix of type '<class 'numpy.float64'>'\n",
              "\twith 34796 stored elements in Compressed Sparse Row format>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5jrMB2sGmNtH"
      },
      "source": [
        "## 5. Building and evaluating a model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_UA_grwfmKAx",
        "outputId": "68f9c903-b1bf-4109-c056-d7e7e70e44cd"
      },
      "source": [
        "# import and instantiate a Multinomial Naive Bayes model\n",
        "from sklearn.naive_bayes import MultinomialNB\n",
        "nb = MultinomialNB()\n",
        "# train the model using X_train_dtm (timing it with an IPython \"magic command\")\n",
        "%time nb.fit(X_train_dtm, y_train)"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "CPU times: user 6.55 ms, sys: 0 ns, total: 6.55 ms\n",
            "Wall time: 9.38 ms\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sX3Xtcn8mQ6F",
        "outputId": "07e0995c-2596-41d3-b467-a1039e6300cc"
      },
      "source": [
        "# make class predictions for X_test_dtm\n",
        "y_pred_class = nb.predict(X_test_dtm)\n",
        "# calculate accuracy of class predictions\n",
        "from sklearn import metrics\n",
        "metrics.accuracy_score(y_test, y_pred_class)"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9827709978463748"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Zs88HASKmUOy",
        "outputId": "500deb05-b52e-4c8c-b171-d01de5050da4"
      },
      "source": [
        "# print the confusion matrix\n",
        "metrics.confusion_matrix(y_test, y_pred_class)"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[1205,    8],\n",
              "       [  16,  164]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "o_UjIBNEmW1Q",
        "outputId": "899110aa-c49c-463c-df46-cd03830198f7"
      },
      "source": [
        "X_test.shape"
      ],
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(1393,)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 43
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d28YU2iUmZXm",
        "outputId": "47bb2ae0-2e40-44e5-95ed-8aba3975f965"
      },
      "source": [
        "X_test[y_pred_class > y_test]"
      ],
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2418    Madamregret disturbancemight receive reference...\n",
              "4598                                laid airtel line rest\n",
              "386                                   Customer place call\n",
              "1289    HeyGreat dealFarm tour 9am 5pm 95pax 50 deposi...\n",
              "5094    Hi ShanilRakhesh herethanksi exchanged uncut d...\n",
              "494                                      free nowcan call\n",
              "759     Call youcarlos isare phones vibrate acting mig...\n",
              "3140                                  Customer place call\n",
              "Name: clean_msg, dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QCdvZlsNmfY6",
        "outputId": "3adc09a8-6bd6-476f-da06-253ac0d5dfaa"
      },
      "source": [
        "# print message text for false negatives (spam incorrectly classifier)\n",
        "X_test[y_pred_class < y_test]"
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "4674    Hi babe Chloe r smashed saturday night great w...\n",
              "3528    Xmas New Years Eve tickets sale club day 10am ...\n",
              "3417    LIFE never much fun great came made truly spec...\n",
              "2773    come takes little time child afraid dark becom...\n",
              "1960    Guess Somebody know secretly fancies Wanna fin...\n",
              "5       FreeMsg Hey darling 3 weeks word back Id like ...\n",
              "2078                         85233 FREERingtoneReply REAL\n",
              "1457    CLAIRE havin borin time alone wanna cum 2nite ...\n",
              "190     unique enough Find 30th August wwwareyouunique...\n",
              "2429    Guess IThis first time created web page WWWASJ...\n",
              "3057    unsubscribed services Get tons sexy babes hunk...\n",
              "1021    Guess Somebody know secretly fancies Wanna fin...\n",
              "4067    TBSPERSOLVO chasing us since Sept forå£38 defi...\n",
              "3358         Sorry missed call lets talk time 07090201529\n",
              "2821    ROMCAPspam Everyone around responding well pre...\n",
              "2247    Back work 2morro half term C 2nite sexy passio...\n",
              "Name: clean_msg, dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "ptFhEiLimiCZ",
        "outputId": "514a66db-36a0-4214-bb3f-a774a9b82e72"
      },
      "source": [
        "# example of false negative \n",
        "X_test[4949]"
      ],
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'Hi probably much fun get message thought id txt cos bored james farting night'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KCIj1ItimlhQ",
        "outputId": "f7a656ee-0d05-448f-8b7b-69c90a5be155"
      },
      "source": [
        "# calculate predicted probabilities for X_test_dtm (poorly calibrated)\n",
        "y_pred_prob = nb.predict_proba(X_test_dtm)[:, 1]\n",
        "y_pred_prob"
      ],
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([2.11903975e-02, 3.97831612e-04, 1.06470895e-03, ...,\n",
              "       1.31939653e-02, 9.99821127e-05, 6.04083365e-06])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wZjMVAZemomm",
        "outputId": "43ec9b84-3d96-4065-e066-d4bd8eb15b5f"
      },
      "source": [
        "# calculate AUC\n",
        "metrics.roc_auc_score(y_test, y_pred_prob)"
      ],
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9774342768159751"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sfeq_Xr7mrDm",
        "outputId": "d607a8ec-7303-45a2-eb7e-3fa67146a39e"
      },
      "source": [
        "from sklearn.feature_extraction.text import TfidfTransformer\n",
        "from sklearn.pipeline import Pipeline\n",
        "\n",
        "pipe = Pipeline([('bow', CountVectorizer()), \n",
        "                 ('tfid', TfidfTransformer()),  \n",
        "                 ('model', MultinomialNB())])\n",
        "pipe.fit(X_train, y_train)"
      ],
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Pipeline(memory=None,\n",
              "         steps=[('bow',\n",
              "                 CountVectorizer(analyzer='word', binary=False,\n",
              "                                 decode_error='strict',\n",
              "                                 dtype=<class 'numpy.int64'>, encoding='utf-8',\n",
              "                                 input='content', lowercase=True, max_df=1.0,\n",
              "                                 max_features=None, min_df=1,\n",
              "                                 ngram_range=(1, 1), preprocessor=None,\n",
              "                                 stop_words=None, strip_accents=None,\n",
              "                                 token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
              "                                 tokenizer=None, vocabulary=None)),\n",
              "                ('tfid',\n",
              "                 TfidfTransformer(norm='l2', smooth_idf=True,\n",
              "                                  sublinear_tf=False, use_idf=True)),\n",
              "                ('model',\n",
              "                 MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True))],\n",
              "         verbose=False)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VXV7G_RtmtuI",
        "outputId": "c0f9ab52-d7a3-4573-d06e-5d66d22af906"
      },
      "source": [
        "y_pred = pipe.predict(X_test)\n",
        "metrics.accuracy_score(y_test, y_pred)"
      ],
      "execution_count": 50,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9669777458722182"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 50
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gPRQSIP-mwym",
        "outputId": "5fd9f66a-3b74-43bb-e577-c1720abba3f3"
      },
      "source": [
        "metrics.confusion_matrix(y_test, y_pred)"
      ],
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[1213,    0],\n",
              "       [  46,  134]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 51
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yZW33zDRm3Lz"
      },
      "source": [
        "## 6. Comparing models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Bgl9ffKdmz8O",
        "outputId": "8bd03dd4-97fb-45d8-ff03-d8252b6ac139"
      },
      "source": [
        "# import an instantiate a logistic regression model\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "logreg = LogisticRegression(solver='liblinear')\n",
        "# train the model using X_train_dtm\n",
        "%time logreg.fit(X_train_dtm, y_train)"
      ],
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "CPU times: user 19.8 ms, sys: 3.81 ms, total: 23.6 ms\n",
            "Wall time: 29 ms\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
              "                   intercept_scaling=1, l1_ratio=None, max_iter=100,\n",
              "                   multi_class='auto', n_jobs=None, penalty='l2',\n",
              "                   random_state=None, solver='liblinear', tol=0.0001, verbose=0,\n",
              "                   warm_start=False)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "E-EYcA-sm6Uu",
        "outputId": "9b11a513-b2f4-490a-b336-048b4ce177f7"
      },
      "source": [
        "# make class predictions for X_test_dtm\n",
        "y_pred_class = logreg.predict(X_test_dtm)\n",
        "# calculate predicted probabilities for X_test_dtm (well calibrated)\n",
        "y_pred_prob = logreg.predict_proba(X_test_dtm)[:, 1]\n",
        "y_pred_prob"
      ],
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0.01694418, 0.0152182 , 0.08261755, ..., 0.02198942, 0.00531726,\n",
              "       0.00679188])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 53
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gm65nFGkm9fB",
        "outputId": "0f1b8e50-1aa6-4544-9787-533e26a097ea"
      },
      "source": [
        "# calculate accuracy\n",
        "metrics.accuracy_score(y_test, y_pred_class)"
      ],
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9842067480258435"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 54
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UXP8643anACj",
        "outputId": "45a3e687-769a-4bb0-f3c3-f01c4af370fa"
      },
      "source": [
        "metrics.confusion_matrix(y_test, y_pred_class)"
      ],
      "execution_count": 55,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[1213,    0],\n",
              "       [  22,  158]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 55
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zVoMyCBdnCMl",
        "outputId": "6e03c07f-a2b8-4ade-daa0-fab8d9511bad"
      },
      "source": [
        "# calculate AUC\n",
        "metrics.roc_auc_score(y_test, y_pred_prob)"
      ],
      "execution_count": 62,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9835714940001832"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 62
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g8EoAyq6npFn"
      },
      "source": [
        "## Conclusion:\n",
        "\n",
        "### Thus the model is 98% accurate and most spam messeages have been categorized accordingly."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fyp0b_8fngFu"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}