MSIS549_elainez1_HW3_RNN_Reuters.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "MSIS549_elainez1_HW3_RNN_Reuters.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r2fyLhUMQlyH"
      },
      "source": [
        "# MSIS 579 HW3 RNN to Classify Reuters Topics\n",
        "\n",
        "In this homework, we will train a recurrent neural network to Classify Reuters newswires into 46 Topics.\n",
        "\n",
        "Dataset of 11,228 newswires from Reuters, labeled over 46 topics. As with the IMDB dataset, each wire is encoded as a sequence of word indexes (same conventions)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r3Y8ZHFGR1uq",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "61814a05-a404-4cc1-c6fa-66df9663c103"
      },
      "source": [
        "%tensorflow_version 1.14\n",
        "%matplotlib inline\n",
        "import warnings\n",
        "warnings.filterwarnings('ignore')\n",
        "\n",
        "import tensorflow as tf\n",
        "tf.logging.set_verbosity(tf.logging.ERROR)\n",
        "\n",
        "from __future__ import print_function\n",
        "\n",
        "import numpy as np\n",
        "import keras\n",
        "keras.__version__\n",
        "\n",
        "!pip install numpy==1.16.1"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "`%tensorflow_version` only switches the major version: 1.x or 2.x.\n",
            "You set: `1.14`. This will be interpreted as: `1.x`.\n",
            "\n",
            "\n",
            "TensorFlow is already loaded. Please restart the runtime to change versions.\n",
            "Requirement already satisfied: numpy==1.16.1 in /usr/local/lib/python3.7/dist-packages (1.16.1)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Uvtq164jRPNz"
      },
      "source": [
        "## Load Reuters Dataset\n",
        "\n",
        "First let's load the Reuters dataset. Please refer to [this API page](https://keras.io/datasets/#reuters-newswire-topics-classification) for details on how to load the data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Mbt21X3mQjZw",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "05da6a49-677e-4359-8fb6-e5d594b65aa3"
      },
      "source": [
        "from keras.datasets import reuters\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense, Dropout, Activation\n",
        "from keras.preprocessing.text import Tokenizer\n",
        "\n",
        "max_words = 10000\n",
        "\n",
        "print('Loading data...')\n",
        "(x_train, y_train), (x_test, y_test) = reuters.load_data(num_words=max_words, test_split=0.2)\n",
        "word_index = reuters.get_word_index(path=\"reuters_word_index.json\")"
      ],
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loading data...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qd7wneSMSTpS"
      },
      "source": [
        "## Task 1: Fully Connected Neural Networks\n",
        "\n",
        "In this task, we will learn a word embedding layer as well as fully connected layers to classify Reuters newwires. Please refer to the lab code from lesson 4. Watch out the overfitting. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W7OPafLXRsnc"
      },
      "source": [
        "# TODO\n",
        "from keras.layers import Embedding\n",
        "embedding_layer = Embedding(10000, 64)"
      ],
      "execution_count": 45,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QOyMlTZagJ0O"
      },
      "source": [
        "from keras.datasets import reuters\n",
        "from keras import preprocessing\n",
        "from tensorflow.keras import utils as np_utils\n",
        "\n",
        "\n",
        "max_features = 10000\n",
        "\n",
        "maxlen = 20\n",
        "(x_train, y_train), (x_test, y_test) = reuters.load_data(num_words=max_features)\n",
        "\n",
        "x_train = preprocessing.sequence.pad_sequences(x_train, maxlen=maxlen)\n",
        "x_test = preprocessing.sequence.pad_sequences(x_test, maxlen=maxlen)\n",
        "y_train = np_utils.to_categorical(y_train)\n",
        "y_test = np_utils.to_categorical(y_test)"
      ],
      "execution_count": 56,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bq-Yhva1smdB",
        "outputId": "59185d8b-34b4-4dc7-c7a2-5b40768c5833"
      },
      "source": [
        "from keras.models import Sequential\n",
        "from keras.layers import Flatten, Dense, Embedding, Dropout\n",
        "from keras.layers.convolutional import Convolution1D, MaxPooling1D\n",
        "model = Sequential()\n",
        "model.add(Embedding(10000, 8, input_length= maxlen))\n",
        "model.add(Flatten())\n",
        "\n",
        "model.add(Dense(64, activation='relu'))\n",
        "\n",
        "model.add(Dense(46, activation='softmax'))\n",
        "\n",
        "model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])\n",
        "\n",
        "model.summary()\n",
        "history = model.fit(\n",
        "    x_train, \n",
        "    y_train,\n",
        "    epochs=10,\n",
        "    batch_size=32,\n",
        "    validation_split=0.2\n",
        ")\n"
      ],
      "execution_count": 58,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_24\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding_27 (Embedding)     (None, 20, 8)             80000     \n",
            "_________________________________________________________________\n",
            "flatten_18 (Flatten)         (None, 160)               0         \n",
            "_________________________________________________________________\n",
            "dense_44 (Dense)             (None, 64)                10304     \n",
            "_________________________________________________________________\n",
            "dense_45 (Dense)             (None, 46)                2990      \n",
            "=================================================================\n",
            "Total params: 93,294\n",
            "Trainable params: 93,294\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "Train on 7185 samples, validate on 1797 samples\n",
            "Epoch 1/10\n",
            "7185/7185 [==============================] - 1s 82us/step - loss: 2.4737 - accuracy: 0.3758 - val_loss: 2.0206 - val_accuracy: 0.4630\n",
            "Epoch 2/10\n",
            "7185/7185 [==============================] - 0s 62us/step - loss: 1.9048 - accuracy: 0.4831 - val_loss: 1.8803 - val_accuracy: 0.4908\n",
            "Epoch 3/10\n",
            "7185/7185 [==============================] - 0s 54us/step - loss: 1.7582 - accuracy: 0.5294 - val_loss: 1.8180 - val_accuracy: 0.5270\n",
            "Epoch 4/10\n",
            "7185/7185 [==============================] - 0s 61us/step - loss: 1.6458 - accuracy: 0.5736 - val_loss: 1.7700 - val_accuracy: 0.5409\n",
            "Epoch 5/10\n",
            "7185/7185 [==============================] - 0s 65us/step - loss: 1.5333 - accuracy: 0.6040 - val_loss: 1.7457 - val_accuracy: 0.5504\n",
            "Epoch 6/10\n",
            "7185/7185 [==============================] - 0s 63us/step - loss: 1.4292 - accuracy: 0.6337 - val_loss: 1.7214 - val_accuracy: 0.5632\n",
            "Epoch 7/10\n",
            "7185/7185 [==============================] - 0s 65us/step - loss: 1.3318 - accuracy: 0.6585 - val_loss: 1.7297 - val_accuracy: 0.5676\n",
            "Epoch 8/10\n",
            "7185/7185 [==============================] - 0s 66us/step - loss: 1.2419 - accuracy: 0.6821 - val_loss: 1.7299 - val_accuracy: 0.5704\n",
            "Epoch 9/10\n",
            "7185/7185 [==============================] - 0s 62us/step - loss: 1.1548 - accuracy: 0.6995 - val_loss: 1.7566 - val_accuracy: 0.5687\n",
            "Epoch 10/10\n",
            "7185/7185 [==============================] - 0s 63us/step - loss: 1.0738 - accuracy: 0.7235 - val_loss: 1.7756 - val_accuracy: 0.5737\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XWj68L3lw9AH",
        "outputId": "b5d624cf-a9e7-402e-8663-99e58684dfd7"
      },
      "source": [
        "results = model.evaluate(x_test, y_test)\n",
        "print(\"Test Accuracy:\",results[1])"
      ],
      "execution_count": 61,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2246/2246 [==============================] - 0s 20us/step\n",
            "Test Accuracy: 0.5494211912155151\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w9__yxAHUSrb"
      },
      "source": [
        "## Task 2: RNN/LSTM\n",
        "Now, we have a fully connected neural networks trained for prediction topics in Reuters data. In this task, we will swap out the fully connect layers and replace with a more powerful RNN layers (LSTM, GRU). Try experiment with different RNN layers and see if they can help improve the model performance."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QXGH40siUTT9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8434baed-519f-4c26-82d4-1a6a29f34d95"
      },
      "source": [
        "# TODO\n",
        "\n",
        "from keras.layers import LSTM\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Embedding, SimpleRNN\n",
        "\n",
        "model2 = Sequential()\n",
        "model2.add(Embedding(max_features, 20))\n",
        "model2.add(LSTM(32))\n",
        "model2.add(Dense(46, activation='softmax'))\n",
        "\n",
        "model2.summary()\n",
        "model2.compile(loss='categorical_crossentropy',\n",
        "              optimizer='rmsprop',\n",
        "              metrics=['acc'])\n",
        "history2 = model2.fit(\n",
        "    x_train, \n",
        "    y_train,\n",
        "    epochs=10,\n",
        "    batch_size=32,\n",
        "    validation_split=0.2\n",
        ")\n",
        "\n",
        "results = model2.evaluate(x_test, y_test)\n",
        "print(\"Test Accuracy for model2:\",results[1])"
      ],
      "execution_count": 67,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_30\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding_33 (Embedding)     (None, None, 20)          200000    \n",
            "_________________________________________________________________\n",
            "lstm_5 (LSTM)                (None, 32)                6784      \n",
            "_________________________________________________________________\n",
            "dense_48 (Dense)             (None, 46)                1518      \n",
            "=================================================================\n",
            "Total params: 208,302\n",
            "Trainable params: 208,302\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "Train on 7185 samples, validate on 1797 samples\n",
            "Epoch 1/10\n",
            "7185/7185 [==============================] - 3s 379us/step - loss: 2.2939 - acc: 0.3855 - val_loss: 2.0045 - val_acc: 0.4636\n",
            "Epoch 2/10\n",
            "7185/7185 [==============================] - 2s 322us/step - loss: 1.9058 - acc: 0.4681 - val_loss: 1.9282 - val_acc: 0.4758\n",
            "Epoch 3/10\n",
            "7185/7185 [==============================] - 2s 318us/step - loss: 1.8064 - acc: 0.5026 - val_loss: 1.9001 - val_acc: 0.4880\n",
            "Epoch 4/10\n",
            "7185/7185 [==============================] - 2s 325us/step - loss: 1.7309 - acc: 0.5173 - val_loss: 1.8610 - val_acc: 0.5025\n",
            "Epoch 5/10\n",
            "7185/7185 [==============================] - 2s 317us/step - loss: 1.6511 - acc: 0.5375 - val_loss: 1.8619 - val_acc: 0.4903\n",
            "Epoch 6/10\n",
            "7185/7185 [==============================] - 2s 322us/step - loss: 1.5724 - acc: 0.5646 - val_loss: 1.8822 - val_acc: 0.5337\n",
            "Epoch 7/10\n",
            "7185/7185 [==============================] - 2s 317us/step - loss: 1.4899 - acc: 0.5932 - val_loss: 1.7893 - val_acc: 0.5331\n",
            "Epoch 8/10\n",
            "7185/7185 [==============================] - 2s 322us/step - loss: 1.4017 - acc: 0.6206 - val_loss: 1.8545 - val_acc: 0.5042\n",
            "Epoch 9/10\n",
            "7185/7185 [==============================] - 2s 323us/step - loss: 1.3222 - acc: 0.6433 - val_loss: 1.8322 - val_acc: 0.5175\n",
            "Epoch 10/10\n",
            "7185/7185 [==============================] - 2s 321us/step - loss: 1.2383 - acc: 0.6671 - val_loss: 1.7613 - val_acc: 0.5615\n",
            "2246/2246 [==============================] - 0s 75us/step\n",
            "Test Accuracy for model2: 0.544968843460083\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NnGOStv7yyPH",
        "outputId": "935b3912-368b-4158-d23b-9aa6e7e23695"
      },
      "source": [
        "model3 = Sequential()\n",
        "model3.add(Embedding(max_features, 20))\n",
        "model3.add(SimpleRNN(32))\n",
        "model3.add(Dense(46, activation='softmax'))\n",
        "\n",
        "model3.summary()\n",
        "model3.compile(loss='categorical_crossentropy',\n",
        "              optimizer='rmsprop',\n",
        "              metrics=['acc'])\n",
        "history3 = model3.fit(\n",
        "    x_train, \n",
        "    y_train,\n",
        "    epochs=10,\n",
        "    batch_size=32,\n",
        "    validation_split=0.2\n",
        ")\n",
        "\n",
        "results = model3.evaluate(x_test, y_test)\n",
        "print(\"Test Accuracy for model3:\",results[1])"
      ],
      "execution_count": 68,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_31\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding_34 (Embedding)     (None, None, 20)          200000    \n",
            "_________________________________________________________________\n",
            "simple_rnn_3 (SimpleRNN)     (None, 32)                1696      \n",
            "_________________________________________________________________\n",
            "dense_49 (Dense)             (None, 46)                1518      \n",
            "=================================================================\n",
            "Total params: 203,214\n",
            "Trainable params: 203,214\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "Train on 7185 samples, validate on 1797 samples\n",
            "Epoch 1/10\n",
            "7185/7185 [==============================] - 2s 247us/step - loss: 2.5008 - acc: 0.3502 - val_loss: 2.3974 - val_acc: 0.3244\n",
            "Epoch 2/10\n",
            "7185/7185 [==============================] - 1s 185us/step - loss: 2.1613 - acc: 0.4653 - val_loss: 2.0883 - val_acc: 0.4736\n",
            "Epoch 3/10\n",
            "7185/7185 [==============================] - 1s 184us/step - loss: 2.0214 - acc: 0.4919 - val_loss: 2.0819 - val_acc: 0.4329\n",
            "Epoch 4/10\n",
            "7185/7185 [==============================] - 1s 193us/step - loss: 1.8660 - acc: 0.5166 - val_loss: 2.0310 - val_acc: 0.4636\n",
            "Epoch 5/10\n",
            "7185/7185 [==============================] - 1s 194us/step - loss: 1.7132 - acc: 0.5535 - val_loss: 1.9043 - val_acc: 0.5003\n",
            "Epoch 6/10\n",
            "7185/7185 [==============================] - 1s 192us/step - loss: 1.5817 - acc: 0.5904 - val_loss: 1.8841 - val_acc: 0.5114\n",
            "Epoch 7/10\n",
            "7185/7185 [==============================] - 1s 199us/step - loss: 1.4723 - acc: 0.6184 - val_loss: 1.9717 - val_acc: 0.4958\n",
            "Epoch 8/10\n",
            "7185/7185 [==============================] - 1s 189us/step - loss: 1.3636 - acc: 0.6500 - val_loss: 1.9438 - val_acc: 0.5131\n",
            "Epoch 9/10\n",
            "7185/7185 [==============================] - 1s 194us/step - loss: 1.2592 - acc: 0.6797 - val_loss: 1.9658 - val_acc: 0.5109\n",
            "Epoch 10/10\n",
            "7185/7185 [==============================] - 1s 194us/step - loss: 1.1687 - acc: 0.7005 - val_loss: 1.9738 - val_acc: 0.5120\n",
            "2246/2246 [==============================] - 0s 53us/step\n",
            "Test Accuracy for model3: 0.5133571028709412\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hIbhAoalzD0r",
        "outputId": "6b5eca0e-d304-4815-8f2b-d7f32ccdb81a"
      },
      "source": [
        "from keras.layers import GRU\n",
        "\n",
        "model4 = Sequential()\n",
        "model4.add(Embedding(max_features, 20))\n",
        "model4.add(GRU(32))\n",
        "model4.add(Dense(46, activation='softmax'))\n",
        "\n",
        "model4.summary()\n",
        "model4.compile(loss='categorical_crossentropy',\n",
        "              optimizer='rmsprop',\n",
        "              metrics=['acc'])\n",
        "history4 = model4.fit(\n",
        "    x_train, \n",
        "    y_train,\n",
        "    epochs=10,\n",
        "    batch_size=32,\n",
        "    validation_split=0.2\n",
        ")\n",
        "\n",
        "results = model4.evaluate(x_test, y_test)\n",
        "print(\"Test Accuracy for model4:\",results[1])"
      ],
      "execution_count": 70,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_33\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding_36 (Embedding)     (None, None, 20)          200000    \n",
            "_________________________________________________________________\n",
            "gru_2 (GRU)                  (None, 32)                5088      \n",
            "_________________________________________________________________\n",
            "dense_51 (Dense)             (None, 46)                1518      \n",
            "=================================================================\n",
            "Total params: 206,606\n",
            "Trainable params: 206,606\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "Train on 7185 samples, validate on 1797 samples\n",
            "Epoch 1/10\n",
            "7185/7185 [==============================] - 3s 423us/step - loss: 2.3680 - acc: 0.3556 - val_loss: 2.0599 - val_acc: 0.4007\n",
            "Epoch 2/10\n",
            "7185/7185 [==============================] - 3s 357us/step - loss: 1.9800 - acc: 0.4317 - val_loss: 1.9639 - val_acc: 0.4647\n",
            "Epoch 3/10\n",
            "7185/7185 [==============================] - 3s 367us/step - loss: 1.8655 - acc: 0.4708 - val_loss: 1.9521 - val_acc: 0.4680\n",
            "Epoch 4/10\n",
            "7185/7185 [==============================] - 3s 356us/step - loss: 1.8086 - acc: 0.4855 - val_loss: 1.9502 - val_acc: 0.4747\n",
            "Epoch 5/10\n",
            "7185/7185 [==============================] - 2s 347us/step - loss: 1.7588 - acc: 0.5022 - val_loss: 1.9097 - val_acc: 0.4864\n",
            "Epoch 6/10\n",
            "7185/7185 [==============================] - 3s 351us/step - loss: 1.7052 - acc: 0.5176 - val_loss: 2.0141 - val_acc: 0.4674\n",
            "Epoch 7/10\n",
            "7185/7185 [==============================] - 2s 346us/step - loss: 1.6527 - acc: 0.5388 - val_loss: 1.8946 - val_acc: 0.4981\n",
            "Epoch 8/10\n",
            "7185/7185 [==============================] - 3s 353us/step - loss: 1.5939 - acc: 0.5602 - val_loss: 1.9132 - val_acc: 0.4914\n",
            "Epoch 9/10\n",
            "7185/7185 [==============================] - 3s 350us/step - loss: 1.5501 - acc: 0.5734 - val_loss: 1.8974 - val_acc: 0.5103\n",
            "Epoch 10/10\n",
            "7185/7185 [==============================] - 3s 354us/step - loss: 1.5116 - acc: 0.5827 - val_loss: 2.0082 - val_acc: 0.5070\n",
            "2246/2246 [==============================] - 0s 66us/step\n",
            "Test Accuracy for model4: 0.5089046955108643\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "asCa299TUV7N"
      },
      "source": [
        "## Task 3: Use a Pre-trained Word Embedding\n",
        "\n",
        "In this task, instead of learning the word embedding layer from scratch, we apply a pre-trained word embedding layer and only use the classification base for reuters data. Please refer to the [API](https://keras.io/examples/pretrained_word_embeddings/) for different pre-trained word embedding.\n",
        "\n",
        "Does the pre-trained word embedding help improve the model prediction?\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MkQprFAyU-0g"
      },
      "source": [
        "# TODO\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UfUJWeOu2EAS"
      },
      "source": [
        "def loadGloveModel(gloveFile):\n",
        "    \"\"\"\n",
        "    Loads GloVe Model\n",
        "    \n",
        "    Arguments:\n",
        "    gloveFile -- path to the glove file\n",
        "\n",
        "    Returns:\n",
        "    model -- a word_to_vec_map, where keys are words, and values are vectors (represented by arrays)\n",
        "    \"\"\"\n",
        "    \n",
        "    print(\"Loading Glove Model\")\n",
        "    f = open(gloveFile,'r')\n",
        "    model = {}\n",
        "    for line in f:\n",
        "        splitLine = line.split()\n",
        "        word = splitLine[0]\n",
        "        embedding = np.array([float(val) for val in splitLine[1:]])\n",
        "        model[word] = embedding\n",
        "    print(\"Done.\",len(model),\" words loaded!\")\n",
        "    return model\n",
        "\n",
        "def pretrained_embedding_layer(word_to_vec_map, word_to_wordidx):\n",
        "    \"\"\"\n",
        "    Creates a Keras Embedding() layer and loads in pre-trained GloVe 50-dimensional vectors.\n",
        "    \n",
        "    Arguments:\n",
        "    word_to_vec_map -- dictionary mapping words to their GloVe vector representation.\n",
        "    word_to_wordidx -- dictionary mapping from words to their indices in the vocabulary (400,001 words)\n",
        "\n",
        "    Returns:\n",
        "    embedding_layer -- pretrained layer Keras instance\n",
        "    \"\"\"\n",
        "    \n",
        "    vocab_len = len(word_to_wordidx) + 1                  # adding 1 to fit Keras embedding (requirement)\n",
        "    emb_dim = word_to_vec_map[\"cucumber\"].shape[0]      # define dimensionality of your GloVe word vectors (= 50)\n",
        "    \n",
        "    ### START CODE HERE ###\n",
        "    # Initialize the embedding matrix as a numpy array of zeros of shape (vocab_len, dimensions of word vectors = emb_dim)\n",
        "    emb_matrix = np.zeros(((vocab_len, emb_dim)))\n",
        "    \n",
        "    # Set each row \"index\" of the embedding matrix to be the word vector representation of the \"index\"th word of the vocabulary\n",
        "    for word, index in word_to_wordidx.items():\n",
        "        if word in word_to_vec_map:\n",
        "            emb_matrix[index, :] = word_to_vec_map[word]\n",
        "        else:\n",
        "            emb_matrix[index, :] = word_to_vec_map[\"random\"]  #just to set something when work is not in word_to_vec_map\n",
        "\n",
        "    # Define Keras embedding layer with the correct output/input sizes, make it trainable. Use Embedding(...). Make sure to set trainable=False. \n",
        "    embedding_layer = Embedding(input_dim=vocab_len, output_dim=emb_dim, trainable=False)\n",
        "    ### END CODE HERE ###\n",
        "\n",
        "    # Build the embedding layer, it is required before setting the weights of the embedding layer. Do not modify the \"None\".\n",
        "    embedding_layer.build((None,))\n",
        "    \n",
        "    # Set the weights of the embedding layer to the embedding matrix. Your layer is now pretrained.\n",
        "    embedding_layer.set_weights([emb_matrix])\n",
        "    \n",
        "    return embedding_layer"
      ],
      "execution_count": 93,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_Q52sZTIF0Yh",
        "outputId": "558f8525-b3cb-44dd-f686-a3451ccd04e5"
      },
      "source": [
        "!wget --no-check-certificate \\\n",
        "    http://nlp.stanford.edu/data/glove.6B.zip \\\n",
        "    -O /tmp/glove.6B.zip"
      ],
      "execution_count": 96,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-05-27 04:31:18--  http://nlp.stanford.edu/data/glove.6B.zip\n",
            "Resolving nlp.stanford.edu (nlp.stanford.edu)... 171.64.67.140\n",
            "Connecting to nlp.stanford.edu (nlp.stanford.edu)|171.64.67.140|:80... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://nlp.stanford.edu/data/glove.6B.zip [following]\n",
            "--2021-05-27 04:31:18--  https://nlp.stanford.edu/data/glove.6B.zip\n",
            "Connecting to nlp.stanford.edu (nlp.stanford.edu)|171.64.67.140|:443... connected.\n",
            "HTTP request sent, awaiting response... 301 Moved Permanently\n",
            "Location: http://downloads.cs.stanford.edu/nlp/data/glove.6B.zip [following]\n",
            "--2021-05-27 04:31:19--  http://downloads.cs.stanford.edu/nlp/data/glove.6B.zip\n",
            "Resolving downloads.cs.stanford.edu (downloads.cs.stanford.edu)... 171.64.64.22\n",
            "Connecting to downloads.cs.stanford.edu (downloads.cs.stanford.edu)|171.64.64.22|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 862182613 (822M) [application/zip]\n",
            "Saving to: ‘/tmp/glove.6B.zip’\n",
            "\n",
            "/tmp/glove.6B.zip   100%[===================>] 822.24M  5.22MB/s    in 2m 42s  \n",
            "\n",
            "2021-05-27 04:34:01 (5.09 MB/s) - ‘/tmp/glove.6B.zip’ saved [862182613/862182613]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DnNeMqU8Fcsl",
        "outputId": "84c73c6c-6db2-4afe-af42-f49734f06ebc"
      },
      "source": [
        "word_to_vec_map = loadGloveModel('glove.6B.100d.txt')\n"
      ],
      "execution_count": 100,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loading Glove Model\n",
            "Done. 400000  words loaded!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vNUxm4IUFiMA",
        "outputId": "2a5266f9-e465-4fa9-b852-d7d7352abb91"
      },
      "source": [
        "from keras.models import Sequential, Model\n",
        "\n",
        "sentence_indices = Input(shape=(maxlen,), dtype='int32')\n",
        "embedding_layer = pretrained_embedding_layer(word_to_vec_map, word_index)\n",
        "embeddings = embedding_layer(sentence_indices)\n",
        "\n",
        "X = Flatten()(embeddings)\n",
        "\n",
        "X = Dense(46, activation=\"softmax\")(X)\n",
        "model = Model(inputs = sentence_indices, outputs = X)\n",
        "model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])\n",
        "model.summary()"
      ],
      "execution_count": 112,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model_5\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_11 (InputLayer)        (None, 20)                0         \n",
            "_________________________________________________________________\n",
            "embedding_46 (Embedding)     (None, 20, 100)           3098000   \n",
            "_________________________________________________________________\n",
            "flatten_20 (Flatten)         (None, 2000)              0         \n",
            "_________________________________________________________________\n",
            "dense_61 (Dense)             (None, 46)                92046     \n",
            "=================================================================\n",
            "Total params: 3,190,046\n",
            "Trainable params: 92,046\n",
            "Non-trainable params: 3,098,000\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9tFcNS3R3TjX",
        "outputId": "f7ec4aa1-b8fb-474f-d480-835a821ca5af"
      },
      "source": [
        "\n",
        "history = model.fit(\n",
        "    x_train, \n",
        "    y_train,\n",
        "    epochs=10,\n",
        "    batch_size=32,\n",
        "    validation_split=0.2\n",
        ")\n",
        "\n",
        "results = model.evaluate(x_test, y_test)\n",
        "print(\"Test Accuracy for model:\",results[1])"
      ],
      "execution_count": 113,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 7185 samples, validate on 1797 samples\n",
            "Epoch 1/10\n",
            "7185/7185 [==============================] - 1s 195us/step - loss: 2.3307 - accuracy: 0.4028 - val_loss: 2.2206 - val_accuracy: 0.4285\n",
            "Epoch 2/10\n",
            "7185/7185 [==============================] - 1s 95us/step - loss: 1.8209 - accuracy: 0.4848 - val_loss: 2.1930 - val_accuracy: 0.4457\n",
            "Epoch 3/10\n",
            "7185/7185 [==============================] - 1s 94us/step - loss: 1.5293 - accuracy: 0.5461 - val_loss: 2.1868 - val_accuracy: 0.4524\n",
            "Epoch 4/10\n",
            "7185/7185 [==============================] - 1s 107us/step - loss: 1.3178 - accuracy: 0.5932 - val_loss: 2.1919 - val_accuracy: 0.4519\n",
            "Epoch 5/10\n",
            "7185/7185 [==============================] - 1s 98us/step - loss: 1.1536 - accuracy: 0.6454 - val_loss: 2.3314 - val_accuracy: 0.4252\n",
            "Epoch 6/10\n",
            "7185/7185 [==============================] - 1s 109us/step - loss: 1.0326 - accuracy: 0.6811 - val_loss: 2.3038 - val_accuracy: 0.4407\n",
            "Epoch 7/10\n",
            "7185/7185 [==============================] - 1s 101us/step - loss: 0.9312 - accuracy: 0.7144 - val_loss: 2.2893 - val_accuracy: 0.4624\n",
            "Epoch 8/10\n",
            "7185/7185 [==============================] - 1s 109us/step - loss: 0.8510 - accuracy: 0.7400 - val_loss: 2.3335 - val_accuracy: 0.4613\n",
            "Epoch 9/10\n",
            "7185/7185 [==============================] - 1s 110us/step - loss: 0.7843 - accuracy: 0.7610 - val_loss: 2.3765 - val_accuracy: 0.4636\n",
            "Epoch 10/10\n",
            "7185/7185 [==============================] - 1s 109us/step - loss: 0.7271 - accuracy: 0.7769 - val_loss: 2.4648 - val_accuracy: 0.4702\n",
            "2246/2246 [==============================] - 0s 52us/step\n",
            "Test Accuracy for model: 0.47818344831466675\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n4xgR7CpIS8s"
      },
      "source": [
        "pre-trained didn't help with the accuracy, but part of the reason could be that the model only use basic catogorization, instead of RNN of CNN. "
      ]
    }
  ]
}