basic-models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "## Brief overview of approach\n",
    "- From the original dataset we keep only the 'star_rating' and 'review_body' columns to train/test multiple classifiers\n",
    "- We convert 'star_rating' to integer and 'review_body' to string for uniformity of datatypes across these columns\n",
    "- We sample 100,000 random reviews from each 'star_rating' class since it was discovered that the mininmum number of reviews for a class is around 100,000. This sampling was performed to fit the TFIDFVectorizer to a larger corpus so that it can understand the data and provide better features when 20K reviews are randomly sampled\n",
    "- Following data cleaning techniques have been performed on the dataset -\n",
    "    1. convert all reviews to lowercase\n",
    "    2. remove html tags and urls from reviews using BeautifulSoup\n",
    "    3. remove extra spaces in the reviews\n",
    "    4. remove punctuations\n",
    "    5. remove non-alphabetical characters\n",
    "    6. perform contractions on the reviews\n",
    "- Following data preprocessing techniques have been performed on the dataset -\n",
    "    1. remove stop words\n",
    "    2. perform lemmatization\n",
    "- Once we have a fitted TFIDFVectorizer - we further sample 20K reviews for each class and transform it using the fitted TFIDFVectorizer\n",
    "- Next, we create our training and testing split with an 80-20 ratio and pass it to classifiers like the Perceptron, SVM, Logistic Regression and Multinomial Naive Bayes\n",
    "- Finally, we evaluate our trained models on the testing splits and present the results\n",
    "\n",
    "**Notes**\n",
    "- It was observed that the removal of stop words from the reviews did not help the performance of the models but actually degraded it\n",
    "- However lemmatization helped with the performance of the models\n",
    "- Therefore we perform two types of preprocessing\n",
    "    1. Reviews with stop words removed and lemmatization performed\n",
    "    2. Reviews with no stop word removal but lemmatization is performed (This data is used for training and testing the model as it yielded better performances)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Imports"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "source": [
    "# ! pip install bs4\n",
    "# ! pip install lxml\n",
    "# ! pip install contractions"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import nltk\n",
    "import re\n",
    "from bs4 import BeautifulSoup\n",
    "nltk.download('wordnet', quiet=True)\n",
    "nltk.download('stopwords', quiet=True)\n",
    "nltk.download('punkt', quiet=True)\n",
    "nltk.download('omw-1.4', quiet=True)\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "import string\n",
    "import contractions\n",
    "from sklearn.metrics import classification_report\n",
    "import warnings"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "source": [
    "# Dataset: https://s3.amazonaws.com/amazon-reviews-pds/tsv/amazon_reviews_us_Jewelry_v1_00.tsv.gz"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Read Data"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "source": [
    "filepath = './amazon_reviews_us_Jewelry_v1_00.tsv'\n",
    "reviews_df = pd.read_csv(filepath, sep='\\t', on_bad_lines='skip', dtype='unicode')"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "source": [
    "reviews_df.info()"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1766992 entries, 0 to 1766991\n",
      "Data columns (total 15 columns):\n",
      " #   Column             Dtype \n",
      "---  ------             ----- \n",
      " 0   marketplace        object\n",
      " 1   customer_id        object\n",
      " 2   review_id          object\n",
      " 3   product_id         object\n",
      " 4   product_parent     object\n",
      " 5   product_title      object\n",
      " 6   product_category   object\n",
      " 7   star_rating        object\n",
      " 8   helpful_votes      object\n",
      " 9   total_votes        object\n",
      " 10  vine               object\n",
      " 11  verified_purchase  object\n",
      " 12  review_headline    object\n",
      " 13  review_body        object\n",
      " 14  review_date        object\n",
      "dtypes: object(15)\n",
      "memory usage: 202.2+ MB\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Keep Reviews and Ratings\n",
    "- Selecting only 'star_rating' and 'review_body'\n",
    "- We use 'review_body' to develop the input features\n",
    "- We use 'star_rating' as the target results which must be predicted"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "source": [
    "reviews_df = reviews_df[['star_rating','review_body']]"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "source": [
    "reviews_df.info()"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1766992 entries, 0 to 1766991\n",
      "Data columns (total 2 columns):\n",
      " #   Column       Dtype \n",
      "---  ------       ----- \n",
      " 0   star_rating  object\n",
      " 1   review_body  object\n",
      "dtypes: object(2)\n",
      "memory usage: 27.0+ MB\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Randomly selecting reviews from each star_rating_class"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "source": [
    "# Converting all 'star_rating' to integer representations\n",
    "# Select all rows which have 'star_rating' and 'review_body' as existing int/string values for optimal training results of the models\n",
    "\n",
    "reviews_df['star_rating'] = pd.to_numeric(reviews_df['star_rating'],errors='coerce')\n",
    "reviews_df = reviews_df[reviews_df['star_rating'].notna()]\n",
    "reviews_df = reviews_df[reviews_df['review_body'].notna()]\n",
    "\n",
    "#Convert all 'star_rating' to int\n",
    "reviews_df['star_rating'] = reviews_df['star_rating'].astype(int)\n",
    "\n",
    "#Convert all reviews to string\n",
    "reviews_df['review_body'] = reviews_df['review_body'].astype(str)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "source": [
    "reviews_df.info()"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 1766748 entries, 0 to 1766991\n",
      "Data columns (total 2 columns):\n",
      " #   Column       Dtype \n",
      "---  ------       ----- \n",
      " 0   star_rating  int64 \n",
      " 1   review_body  object\n",
      "dtypes: int64(1), object(1)\n",
      "memory usage: 40.4+ MB\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Sampling 100,000 samples per class\n",
    "- It was discovered that the minimum number of samples in a class (Rating - 2) was around 100K so I decided to select 100K samples per class to train the TFIDFVectorizer.\n",
    "- We select 100K per class so that there is no imbalance in representation of words/reviews/classes which may affect the TFIDFVectorizer since it takes into consideration term and document frequencies which could be influenced by major class imbalances\n",
    "- Once we have trained the TFIDFVectorizer on the 500K samples (100K samples for 5 rating classes)- we sample 20000 reviews randomly from each rating class to create the training and testing dataset."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "source": [
    "reviews_df['star_rating'].value_counts()"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "5    1080871\n",
       "4     270424\n",
       "3     159654\n",
       "1     155002\n",
       "2     100797\n",
       "Name: star_rating, dtype: int64"
      ]
     },
     "metadata": {},
     "execution_count": 9
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "source": [
    "rating_1 = reviews_df[reviews_df.star_rating.eq(1)].sample(100000, random_state=1)\n",
    "rating_2 = reviews_df[reviews_df.star_rating.eq(2)].sample(100000, random_state=1)\n",
    "rating_3 = reviews_df[reviews_df.star_rating.eq(3)].sample(100000, random_state=1)\n",
    "rating_4 = reviews_df[reviews_df.star_rating.eq(4)].sample(100000, random_state=1)\n",
    "rating_5 = reviews_df[reviews_df.star_rating.eq(5)].sample(100000, random_state=1)\n",
    "\n",
    "sampled_reviews_df_100000 = pd.concat([rating_1, rating_2, rating_3, rating_4, rating_5])"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "source": [
    "sampled_reviews_df_100000.info()"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 500000 entries, 344647 to 1044904\n",
      "Data columns (total 2 columns):\n",
      " #   Column       Non-Null Count   Dtype \n",
      "---  ------       --------------   ----- \n",
      " 0   star_rating  500000 non-null  int64 \n",
      " 1   review_body  500000 non-null  object\n",
      "dtypes: int64(1), object(1)\n",
      "memory usage: 11.4+ MB\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Utility functions"
   ],
   "metadata": {
    "tags": []
   }
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "source": [
    "def calculateAverageLength(df, columnName):\n",
    "    \"\"\"Calculates the average length of the given column in the dataframe provided as an argument\n",
    "\n",
    "    Args:\n",
    "        df (DataFrame): Dataframe in which we must locate the column and calculate it's average length\n",
    "        columnName (string): Name of the column for which we calculate the average length\n",
    "    \"\"\"\n",
    "    total_length = 0\n",
    "    for i in df[columnName].tolist():\n",
    "        total_length += len(i)\n",
    "    mean_length = total_length/len(df[columnName].tolist())\n",
    "    return mean_length"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "source": [
    "def URLRemoval(sentence):\n",
    "    \"\"\"Function to remove the HTML tags and URLs from reviews using BeautifulSoup\n",
    "\n",
    "    Args:\n",
    "        sentence (string): Sentence from which we remove the HTML tags and URLs\n",
    "\n",
    "    Returns:\n",
    "        string: Sentence which does not contain any HTML tags and URLs\n",
    "    \"\"\"\n",
    "    return BeautifulSoup(sentence, 'lxml').get_text() "
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "source": [
    "def nonAlphabeticRemoval(sentence):\n",
    "    \"\"\"Function to remove non-alphabetic characters from the sentence. \n",
    "    Note - We do not remove spaces from the sentence however extra spaces are removed in a different function\n",
    "\n",
    "    Args:\n",
    "        sentence (string): Sentence from which we remove the non-alphabetic characters\n",
    "\n",
    "    Returns:\n",
    "        string: Sentence from which non-alphabetic characters have been removed\n",
    "    \"\"\"\n",
    "    return re.sub(r\"[^a-zA-Z ]+\", \"\", sentence)  #This will also remove numbers."
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "source": [
    "def removeExtraSpaces(sentence):\n",
    "    \"\"\"Remove extra spaces from the sentence\n",
    "\n",
    "    Args:\n",
    "        sentence (string): Sentence from which we remove extra spaces\n",
    "\n",
    "    Returns:\n",
    "        string: Sentence from which extra spaces have been removed\n",
    "    \"\"\"\n",
    "    return ' '.join(sentence.split())"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "source": [
    "def stopWordRemoval(sentence):\n",
    "    \"\"\"Function to remove stop words from the sentence\n",
    "\n",
    "    Args:\n",
    "        sentence (string): Sentence from which stop words have to be removed\n",
    "\n",
    "    Returns:\n",
    "        string: Sentence from which stop words have been removed\n",
    "    \"\"\"\n",
    "    word_tokens = word_tokenize(sentence)\n",
    "    filtered_sentence = []\n",
    "    stop_words = {}\n",
    "    for stop_word in stopwords.words('english'):\n",
    "        if stop_words.get(stop_word) == None:\n",
    "            stop_words[stop_word] = 1\n",
    "    for w in word_tokens:\n",
    "        if stop_words.get(w) == None:\n",
    "            filtered_sentence.append(w)\n",
    "    return ' '.join(filtered_sentence)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "source": [
    "def lemmatizeSentence(sentence):\n",
    "    \"\"\"Function to lemmatize a sentence \n",
    "\n",
    "    Args:\n",
    "        sentence (string): Sentence which has to be lemmatized\n",
    "\n",
    "    Returns:\n",
    "        string: Sentence which has been lemmatized\n",
    "    \"\"\"\n",
    "    word_tokens = word_tokenize(sentence)\n",
    "    lemmatized_sentence = []\n",
    "    lemmatizer = WordNetLemmatizer()\n",
    "    for word in word_tokens:\n",
    "            lemmatized_sentence.append(lemmatizer.lemmatize(word, pos='a')) #We utilize POS tag 'a' - adjective\n",
    "    return ' '.join(lemmatized_sentence)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "source": [
    "def removePunctuation(sentence):\n",
    "    \"\"\"Function to remove punctuations from a sentence\n",
    "\n",
    "    Args:\n",
    "        sentence (string): Sentence from which punctuations have to be removed\n",
    "\n",
    "    Returns:\n",
    "        string: Sentence from which punctuations have been removed\n",
    "    \"\"\"\n",
    "    for value in string.punctuation:\n",
    "        if value in sentence:\n",
    "            sentence = sentence.replace(value, ' ')\n",
    "    return sentence.strip()"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "source": [
    "def displayReport(actualLabels, predictedLabels, classifierName):\n",
    "    \"\"\"Function to display precision/recall/f1-score metrics for a classifier\n",
    "\n",
    "    Args:\n",
    "        actualLabels (_type_): True labels of the data\n",
    "        predictedLabels (_type_): Labels predicted by the classifier\n",
    "        classifierName (_type_): Name of the classifier which predicted the labels\n",
    "    \"\"\"\n",
    "    targetNames = ['Rating 1', 'Rating 2', 'Rating 3', 'Rating 4', 'Rating 5']\n",
    "    # report = classification_report(actualLabels, predictedLabels, target_names=targetNames, output_dict=True)        \n",
    "    report = classification_report(actualLabels, predictedLabels, target_names=targetNames)\n",
    "    print(report)\n",
    "\n",
    "    # print(f'Precision, Recall, f1-score for Testing split for {classifierName}')\n",
    "    # print('============================================================')\n",
    "    # for targetClass in targetNames:\n",
    "    #     print(f'{targetClass}: {report[targetClass][\"precision\"]}, {report[targetClass][\"recall\"]}, {report[targetClass][\"f1-score\"]}')\n",
    "    # print(f'Macro Average: {report[\"macro avg\"][\"precision\"]}, {report[\"macro avg\"][\"recall\"]}, {report[\"macro avg\"][\"f1-score\"]}')\n",
    "    # print('============================================================')"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Data Cleaning\n",
    "\n"
   ],
   "metadata": {
    "tags": []
   }
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "source": [
    "lengthBeforeCleaning = calculateAverageLength(sampled_reviews_df_100000, 'review_body')"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Convert all reviews into the lower case"
   ],
   "metadata": {
    "tags": []
   }
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "source": [
    "sampled_reviews_df_100000['review_body'] = sampled_reviews_df_100000['review_body'].apply(lambda value:value.lower())"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Remove the HTML and URLs from the reviews"
   ],
   "metadata": {
    "tags": []
   }
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "source": [
    "sampled_reviews_df_100000['review_body'] = sampled_reviews_df_100000['review_body'].apply(URLRemoval)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stderr",
     "text": [
      "/home1/adiyer/.conda/envs/timesformer/lib/python3.7/site-packages/bs4/__init__.py:439: MarkupResemblesLocatorWarning: The input looks more like a filename than markup. You may want to open this file and pass the filehandle into Beautiful Soup.\n",
      "  MarkupResemblesLocatorWarning\n",
      "/home1/adiyer/.conda/envs/timesformer/lib/python3.7/site-packages/bs4/__init__.py:408: MarkupResemblesLocatorWarning: The input looks more like a URL than markup. You may want to use an HTTP client like requests to get the document behind the URL, and feed that document to Beautiful Soup.\n",
      "  MarkupResemblesLocatorWarning\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Remove extra spaces"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "source": [
    "sampled_reviews_df_100000['review_body'] = sampled_reviews_df_100000['review_body'].apply(removeExtraSpaces)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Remove punctuations"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "source": [
    "sampled_reviews_df_100000['review_body'] = sampled_reviews_df_100000['review_body'].apply(removePunctuation)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Remove non-alphabetical characters (excluding spaces)"
   ],
   "metadata": {
    "tags": []
   }
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "source": [
    "sampled_reviews_df_100000['review_body'] = sampled_reviews_df_100000['review_body'].apply(nonAlphabeticRemoval)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Perform contractions on the reviews"
   ],
   "metadata": {
    "tags": []
   }
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "source": [
    "sampled_reviews_df_100000['review_body_contracted'] = sampled_reviews_df_100000['review_body'].apply(lambda value:[contractions.fix(word) for word in value.split()])\n",
    "sampled_reviews_df_100000['review_body_contracted'] = sampled_reviews_df_100000['review_body_contracted'].apply(' '.join)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Average length of the reviews before & after data cleaning"
   ],
   "metadata": {
    "tags": []
   }
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "source": [
    "lengthAfterCleaning = calculateAverageLength(sampled_reviews_df_100000, 'review_body_contracted')"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "source": [
    "print(f'Average length of reviews before and after data cleaning: {lengthBeforeCleaning}, {lengthAfterCleaning}')"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Average length of reviews before and after data cleaning: 189.71903, 182.180316\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Pre-processing"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "source": [
    "lengthBeforePreprocessing = calculateAverageLength(sampled_reviews_df_100000, 'review_body_contracted')"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Remove the stop words "
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "source": [
    "sampled_reviews_df_100000['review_body_without_stop_words'] = sampled_reviews_df_100000['review_body_contracted'].apply(lambda value:stopWordRemoval(value))"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Perform lemmatization  "
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "source": [
    "# Perform lemmatization on sentences which had stop words removed from them\n",
    "sampled_reviews_df_100000['review_body_with_stop_after_lemma'] = sampled_reviews_df_100000['review_body_without_stop_words'].apply(lambda value:lemmatizeSentence(value))"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "source": [
    "# Perform lemmatization on sentences which did not have stop words removed from them\n",
    "sampled_reviews_df_100000['review_body_without_stop_after_lemma'] = sampled_reviews_df_100000['review_body_contracted'].apply(lambda value:lemmatizeSentence(value))"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Average length of the reviews after preprocessing"
   ],
   "metadata": {
    "tags": []
   }
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "source": [
    "lengthAfterPreprocessing = calculateAverageLength(sampled_reviews_df_100000, 'review_body_with_stop_after_lemma')"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "source": [
    "print(f'Average length of reviews before and after data preprocessing: {lengthBeforePreprocessing}, {lengthAfterPreprocessing}')"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Average length of reviews before and after data preprocessing: 182.180316, 108.783628\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# TF-IDF Feature Extraction"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Feature Extraction using TfidfVectorizer\n",
    "- Through the process of experimentation with multiple feature preprocessing and data cleaning techinques it was observed that the best results for all classifiers were obtained when a TFIDFVectorizer is fitted on the reviews data which did not have any stop word removal but lemmatization of the reviews had been performed.\n",
    "- Observed a precision/recall/f1-score boost of 1-2% when utilizing this data over reviews which had stop words removed and the content was lemmatized."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "tfidfVector = TfidfVectorizer(use_idf=True, min_df=1, ngram_range=(1,3)) \n",
    "fittedtfidfVector = tfidfVector.fit(sampled_reviews_df_100000['review_body_without_stop_after_lemma'].to_list())"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    " ## We select 20000 reviews randomly from each rating class and create the train-test split"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "source": [
    "rating_1 = sampled_reviews_df_100000[sampled_reviews_df_100000.star_rating.eq(1)].sample(20000, random_state=1)\n",
    "rating_2 = sampled_reviews_df_100000[sampled_reviews_df_100000.star_rating.eq(2)].sample(20000, random_state=1)\n",
    "rating_3 = sampled_reviews_df_100000[sampled_reviews_df_100000.star_rating.eq(3)].sample(20000, random_state=1)\n",
    "rating_4 = sampled_reviews_df_100000[sampled_reviews_df_100000.star_rating.eq(4)].sample(20000, random_state=1)\n",
    "rating_5 = sampled_reviews_df_100000[sampled_reviews_df_100000.star_rating.eq(5)].sample(20000, random_state=1)\n",
    "sampled_reviews_df_20000 = pd.concat([rating_1, rating_2, rating_3, rating_4, rating_5])"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### 80-20 train-test split is created in the next section with stratification being performed for equal representation of classes in the training and test sets."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "trainData, testData, trainLabels, testLabels = train_test_split(sampled_reviews_df_20000['review_body_without_stop_after_lemma'].to_list(), sampled_reviews_df_20000['star_rating'].to_list(), test_size=0.2, random_state=42, stratify=sampled_reviews_df_20000['star_rating'].to_list())"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "source": [
    "print(len(trainData), len(testData))"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "80000 20000\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Transform the training and testing data using the fitted TFIDFVectorizer"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "source": [
    "trainData = fittedtfidfVector.transform(trainData)\n",
    "testData = fittedtfidfVector.transform(testData)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Perceptron\n",
    "- After experimentation with different hyperparameter settings - best results are achieved with the configuration below."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "source": [
    "from sklearn.linear_model import Perceptron\n",
    "perceptronModel = Perceptron(eta0=0.1, tol=1e-5, n_jobs=-1, max_iter=5000)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "source": [
    "perceptronModel.fit(trainData, trainLabels)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "Perceptron(eta0=0.1, max_iter=5000, n_jobs=-1, tol=1e-05)"
      ]
     },
     "metadata": {},
     "execution_count": 42
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "source": [
    "#Printing for HW2\n",
    "predictedLabels = perceptronModel.predict(testData)\n",
    "displayReport(testLabels, predictedLabels, 'Perceptron')"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "    Rating 1       0.56      0.64      0.60      4000\n",
      "    Rating 2       0.39      0.37      0.38      4000\n",
      "    Rating 3       0.42      0.33      0.37      4000\n",
      "    Rating 4       0.45      0.45      0.45      4000\n",
      "    Rating 5       0.63      0.70      0.67      4000\n",
      "\n",
      "    accuracy                           0.50     20000\n",
      "   macro avg       0.49      0.50      0.49     20000\n",
      "weighted avg       0.49      0.50      0.49     20000\n",
      "\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "source": [
    "predictedLabels = perceptronModel.predict(testData)\n",
    "displayReport(testLabels, predictedLabels, 'Perceptron')"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Precision, Recall, f1-score for Testing split for Perceptron\n",
      "============================================================\n",
      "Rating 1: 0.5565935259613296, 0.6405, 0.595606183889341\n",
      "Rating 2: 0.38808290155440417, 0.3745, 0.3811704834605598\n",
      "Rating 3: 0.41912932952017795, 0.32975, 0.369105918567231\n",
      "Rating 4: 0.45373665480427045, 0.44625, 0.44996218805142424\n",
      "Rating 5: 0.6326301615798923, 0.70475, 0.6667455061494797\n",
      "Macro Average: 0.4900345146840149, 0.49915000000000004, 0.4925180560236071\n",
      "============================================================\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# SVM\n",
    "- After experimentation with different hyperparameter settings - best results are achieved with the configuration below."
   ],
   "metadata": {
    "tags": []
   }
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "source": [
    "from sklearn.svm import LinearSVC\n",
    "svmModel = LinearSVC(penalty='l2', max_iter=2000, dual=True, C=0.25, class_weight={1:1, 2:10, 3:10, 4:1, 5:3}).fit(trainData, trainLabels)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "source": [
    "#Printing for HW2 \n",
    "predictedLabels = svmModel.predict(testData)\n",
    "displayReport(testLabels, predictedLabels, 'SVM')"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "    Rating 1       0.76      0.40      0.52      4000\n",
      "    Rating 2       0.39      0.58      0.47      4000\n",
      "    Rating 3       0.40      0.59      0.48      4000\n",
      "    Rating 4       0.65      0.17      0.27      4000\n",
      "    Rating 5       0.65      0.83      0.73      4000\n",
      "\n",
      "    accuracy                           0.51     20000\n",
      "   macro avg       0.57      0.51      0.49     20000\n",
      "weighted avg       0.57      0.51      0.49     20000\n",
      "\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "source": [
    "# svmModel = LinearSVC(penalty='l2', max_iter=2000, dual=True, C=0.25, class_weight={1:1, 2:10, 3:10, 4:1, 5:3}).fit(trainData, trainLabels)\n",
    "predictedLabels = svmModel.predict(testData)\n",
    "displayReport(testLabels, predictedLabels, 'SVM')"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Precision, Recall, f1-score for Testing split for SVM\n",
      "============================================================\n",
      "Rating 1: 0.7631198844487241, 0.39625, 0.5216389665953596\n",
      "Rating 2: 0.3938931297709924, 0.5805, 0.46932794340576045\n",
      "Rating 3: 0.40298507462686567, 0.58725, 0.47797334418557325\n",
      "Rating 4: 0.649387370405278, 0.17225, 0.27227820588816437\n",
      "Rating 5: 0.6463604515375633, 0.83025, 0.7268548916611951\n",
      "Macro Average: 0.5711491821578847, 0.5133, 0.4936146703472105\n",
      "============================================================\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Logistic Regression\n",
    "- After experimentation with different hyperparameter settings - best results are achieved with the configuration below."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "logRegModel = LogisticRegression(penalty='elasticnet', solver='saga', l1_ratio=0.5, random_state=0, n_jobs=-1, class_weight={1:1, 2:2, 3:2, 4:1, 5:1}).fit(trainData, trainLabels)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "source": [
    "predictedLabelsLogRegn = logRegModel.predict(testData)\n",
    "displayReport(testLabels, predictedLabelsLogRegn, 'Logistic Regression')"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Precision, Recall, f1-score for Testing split for Logistic Regression\n",
      "============================================================\n",
      "Rating 1: 0.764235294117647, 0.406, 0.5302857142857144\n",
      "Rating 2: 0.39891250617894214, 0.60525, 0.480881914787963\n",
      "Rating 3: 0.4165202108963093, 0.5925, 0.48916408668730654\n",
      "Rating 4: 0.5695564516129032, 0.2825, 0.3776737967914438\n",
      "Rating 5: 0.6965150048402711, 0.7195, 0.7078209542547959\n",
      "Macro Average: 0.5691478935292146, 0.52115, 0.5171652933614447\n",
      "============================================================\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Naive Bayes\n",
    "- After experimentation with different hyperparameter settings - best results are achieved with the configuration below."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "mnbModel = MultinomialNB().fit(trainData, trainLabels)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "source": [
    "predictedLabelsMNB = mnbModel.predict(testData)\n",
    "displayReport(testLabels, predictedLabelsMNB, 'Naive Bayes')"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Precision, Recall, f1-score for Testing split for Naive Bayes\n",
      "============================================================\n",
      "Rating 1: 0.6522228474957794, 0.5795, 0.6137145882975907\n",
      "Rating 2: 0.4172443674176776, 0.4815, 0.447075208913649\n",
      "Rating 3: 0.4446354038792045, 0.45275, 0.44865601387340515\n",
      "Rating 4: 0.49503752118131206, 0.51125, 0.5030131595129751\n",
      "Rating 5: 0.7153888582460011, 0.6485, 0.6803042223970626\n",
      "Macro Average: 0.5449057996439949, 0.5347, 0.5385526385989364\n",
      "============================================================\n"
     ]
    }
   ],
   "metadata": {}
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "6012857e81e8cff14b1b6940147b062794364b8096ad9a2e1427430ee5b32590"
  },
  "kernelspec": {
   "display_name": "timesformer-pykernel",
   "language": "python",
   "name": "timesformer"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}