Jumps_modeling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1KIZAiZCryCi"
   },
   "source": [
    "# Financial ML Final Project-Part 2: Jumps Modeling\n",
    "\n",
    "\n",
    "Work by: Skander Chouchene, Mohamed Amine Mairech, Oussema Labidi\n",
    "3rd year EGES - EPT - 2020/2021"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Zf_Ql4VIe5zM"
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "import pickle\n",
    "import datetime\n",
    "import pandas_datareader.data as web\n",
    "import os\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn import metrics\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "#import tensorflow.keras as keras\n",
    "#from tensorflow.keras import layers\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kdsqpXnusHZ_"
   },
   "source": [
    "## Loading and preparing raw data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9tGSkLkRsIWV"
   },
   "source": [
    "### Raw data loading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sUn5wcRnsB--"
   },
   "source": [
    "We start by setting up the directory/path to the raw data. Notice that I clean up the names of the data folders and csv files. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fLzd6voDr6Zd"
   },
   "outputs": [],
   "source": [
    "raw_data_dir = '/content/drive/MyDrive/data'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ppEKmmTosEA1"
   },
   "source": [
    "The following loads the raw historical daily prices. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0s840SUSr-iy"
   },
   "outputs": [],
   "source": [
    "px_db = pd.read_csv(raw_data_dir+'/SHARADAR_SEP.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vekuyh-LXqVb"
   },
   "outputs": [],
   "source": [
    "#px_db=px_db.iloc[int(3*px_db.shape[0]/4):]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "rBr0GoFBLjrh"
   },
   "outputs": [],
   "source": [
    "px_db.date=pd.to_datetime(px_db.date, format='%Y-%m-%d', errors='ignore')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3YfcvbTbsPdC"
   },
   "source": [
    "The following loads the meta-data of the stocks/tickers (exhange, sector, market cap etc.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "QYucPo3Wr-lK",
    "outputId": "83880b96-e754-4fdb-ac36-63ddb1e56c9c"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py:2718: DtypeWarning: Columns (15,16) have mixed types.Specify dtype option on import or set low_memory=False.\n",
      "  interactivity=interactivity, compiler=compiler, result=result)\n"
     ]
    }
   ],
   "source": [
    "ticker_meta= pd.read_csv(raw_data_dir+'/SHARADAR_TICKERS.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "86DlkkhpsTkn"
   },
   "source": [
    "The following loads the constituents of the spy500 index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "HtFr-YkXr-o6"
   },
   "outputs": [],
   "source": [
    "sp500 = pd.read_csv(raw_data_dir+'/SHARADAR_SP500.csv', encoding='latin-1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LzqdOAxLsVcj"
   },
   "source": [
    "The following loads the last ~10 years of the relevant ETF prices from yahoo finance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JkmdRp2KsXe2"
   },
   "outputs": [],
   "source": [
    "sector_etf_list = ['XLK', 'XLI', 'XLV', 'XLE', 'XLP', 'XLY', 'XTL', 'XLRE', 'XLB', 'XLF', 'XLU']\n",
    "etf_list = ['SPY', 'XLK', 'XLI', 'XLV', 'XLE', 'XLP', 'XLY', 'XTL', 'XLRE', 'XLB', 'XLF', 'XLU']\n",
    "etf_dict = {'SPY': 'Market', \n",
    "            'XLK': 'Technology', \n",
    "            'XLI': 'Industrials', \n",
    "            'XLV': 'Healthcare', \n",
    "            'XLE': 'Energy', \n",
    "            'XLP': 'Consumer Defensive', \n",
    "            'XLY': 'Consumer Cyclical', \n",
    "            'XTL': 'Communication Services', \n",
    "            'XLRE': 'Real Estate', \n",
    "            'XLB': 'Basic Materials', \n",
    "            'XLF': 'Financial Services', \n",
    "            'XLU': 'Utilities'}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oHVghYsxsYwd"
   },
   "outputs": [],
   "source": [
    "start = datetime.datetime(2010, 1, 1)\n",
    "end = datetime.date.today() \n",
    "etf_data = []\n",
    "for etf in etf_list:\n",
    "    df = web.DataReader(etf,'yahoo',  start, end)\n",
    "    df['sector'] = etf_dict[etf]\n",
    "    df['etf']    = etf\n",
    "    df['Date']   = df.index.tolist()\n",
    "    etf_data.append(df)\n",
    "    \n",
    "etf_db = pd.concat(etf_data)\n",
    "etf_db.reset_index(drop = True, inplace= True)\n",
    "# etf_df = web.DataReader(etf_list,'yahoo',  start, end)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lRCVm5YFsco5"
   },
   "source": [
    "### Panel data preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OJTA0p5BseUO"
   },
   "source": [
    "Join meta-data data to historical prices and filter relevant information:\n",
    "    1. tickers that belong to the S&P500 index\n",
    "    2. Firms listed in NYSE and NASDAQ\n",
    "    3. 2015 to 2019 data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3hCvRqjvsc3Z"
   },
   "outputs": [],
   "source": [
    "# limiting the meta-data info to the table of historical prices\n",
    "ticker_sep = ticker_meta.loc[ticker_meta.table == 'SEP', :]\n",
    "# limiting the tickers to the sp500 universe\n",
    "ticker_sep_sp500 = ticker_sep.loc[ticker_sep['ticker'].isin(sp500['ticker'].tolist())  , :]\n",
    "pricing_db_sp500 = px_db.loc[px_db['ticker'].isin(sp500['ticker'].tolist())  , :]\n",
    "# joining meta-data and historical prices\n",
    "pricing_db_sp500 = pd.merge(pricing_db_sp500, ticker_sep_sp500[['ticker','sector','scalemarketcap','exchange']], on = 'ticker', how = 'left')\n",
    "# construct the variable year\n",
    "pricing_db_sp500['year'] = [str(s)[0:4] for s in pricing_db_sp500.date]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kcctk0v1srHv"
   },
   "outputs": [],
   "source": [
    "idx = (pricing_db_sp500['year'].isin(['2015','2016','2017', '2018', '2019']) ) & (pricing_db_sp500['exchange'].isin(['NYSE', 'NASDAQ']) ) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-2pnQC-esrJz"
   },
   "outputs": [],
   "source": [
    "sample_db_sp500 = pricing_db_sp500.loc[idx,:]\n",
    "sample_db_sp500.reset_index(drop = True, inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "YlfquB1WsrL2",
    "outputId": "e8d836ec-d0dc-4fdc-ee33-8d4edc34e83a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(827165, 14)"
      ]
     },
     "execution_count": 15,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_db_sp500.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_l_mZz5Hs0Pc"
   },
   "source": [
    "## Feature Engineering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "tEJCWbRdsyFk"
   },
   "outputs": [],
   "source": [
    "eps = 1e-8\n",
    "def get_change_perc(X, prior_X, eps):\n",
    "    return 100*(X-prior_X)/(np.abs(prior_X)+eps)\n",
    "def get_change(X, prior_X, eps):\n",
    "    return 10000* (np.log(X) - np.log(prior_X+eps) )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fEK5tOBjsyH5"
   },
   "outputs": [],
   "source": [
    "def extract_features(px_db_sample, etf_db):\n",
    "    \n",
    "    etf_db['TradeDtKey'] = [ int(str(s)[0:10].replace('-','')) for s in etf_db['Date'] ] \n",
    "    etf_db.sort_values(by ='Date', ascending=True, inplace=True)\n",
    "\n",
    "    etf_db['next_price_close'] = etf_db.groupby('etf')['Adj Close'].shift(-1)\n",
    "    etf_db['next_price_close_2'] = etf_db.groupby('etf')['Adj Close'].shift(-2)\n",
    "    etf_db['next_price_close_5'] = etf_db.groupby('etf')['Adj Close'].shift(-5)\n",
    "    etf_db['next_price_close_10'] = etf_db.groupby('etf')['Adj Close'].shift(-10)\n",
    "    etf_db['next_price_close_20'] = etf_db.groupby('etf')['Adj Close'].shift(-20)\n",
    "\n",
    "    etf_db['prior_price_close'] = etf_db.groupby('etf')['Adj Close'].shift(1)\n",
    "    etf_db['prior_price_close_20'] = etf_db.groupby('etf')['Adj Close'].shift(20)\n",
    "    etf_db['prior_price_close_60'] = etf_db.groupby('etf')['Adj Close'].shift(60)\n",
    "    etf_db['prior_price_close_120'] = etf_db.groupby('etf')['Adj Close'].shift(120)\n",
    "    etf_db['prior_price_close_250'] = etf_db.groupby('etf')['Adj Close'].shift(250)\n",
    "    \n",
    "\n",
    "    etf_db['prior_ret'] = get_change(etf_db['Adj Close'] , etf_db['prior_price_close'] , eps)\n",
    "    etf_db['prior_ret_20'] = get_change(etf_db['Adj Close'] , etf_db['prior_price_close_20'] , eps)\n",
    "    etf_db['prior_ret_60'] = get_change(etf_db['Adj Close'] , etf_db['prior_price_close_60'] , eps)\n",
    "    etf_db['prior_ret_120'] = get_change(etf_db['Adj Close'] , etf_db['prior_price_close_120'] , eps)\n",
    "    etf_db['prior_ret_250'] = get_change(etf_db['Adj Close'] , etf_db['prior_price_close_250'] , eps)\n",
    "\n",
    "    etf_db['next_ret'] = get_change( etf_db['next_price_close'] ,etf_db['Adj Close'] , eps)\n",
    "    etf_db['next_ret_2'] = get_change(etf_db['next_price_close_2'] , etf_db['Adj Close'] , eps)\n",
    "    etf_db['next_ret_5'] = get_change(etf_db['next_price_close_5'] ,etf_db['Adj Close'] ,  eps)\n",
    "    etf_db['next_ret_10'] = get_change(etf_db['next_price_close_10'] ,etf_db['Adj Close'] ,  eps)\n",
    "    etf_db['next_ret_20'] = get_change(etf_db['next_price_close_20'] , etf_db['Adj Close'] , eps)\n",
    "    \n",
    "    \n",
    "    \n",
    "    px_db_sample['Date'] = [ int(str(s)[0:10].replace('-','')) for s in px_db_sample.date ] \n",
    "    px_db_sample.sort_values(by ='Date', ascending=True, inplace=True)\n",
    "    px_db_sample = px_db_sample.loc[px_db_sample['exchange'].isin(['NASDAQ', 'NYSE']), :]\n",
    "    px_db_sample = px_db_sample.loc[px_db_sample['scalemarketcap'].isin(['3 - Small', '4 - Mid', '5 - Large', '6 - Mega']), :]\n",
    "\n",
    "\n",
    "    px_db_sample.reset_index(drop = True, inplace =True)\n",
    "    px_db_sample['px_rank'] = px_db_sample.groupby('Date')['close'].rank(ascending = False, method = 'dense', pct= True)\n",
    "\n",
    "    px_db_sample['next_px_rank'] = px_db_sample.groupby('ticker')['px_rank'].shift(-1)\n",
    "    px_db_sample['next_price_close'] = px_db_sample.groupby('ticker')['close'].shift(-1)\n",
    "    px_db_sample['next_price_close_2'] = px_db_sample.groupby('ticker')['close'].shift(-2)\n",
    "    px_db_sample['next_price_close_5'] = px_db_sample.groupby('ticker')['close'].shift(-5)\n",
    "    px_db_sample['next_price_close_10'] = px_db_sample.groupby('ticker')['close'].shift(-10)\n",
    "    px_db_sample['next_price_close_20'] = px_db_sample.groupby('ticker')['close'].shift(-20)\n",
    "\n",
    "    px_db_sample['prior_price_close'] = px_db_sample.groupby('ticker')['close'].shift(1)\n",
    "    px_db_sample['prior_price_close_20'] = px_db_sample.groupby('ticker')['close'].shift(20)\n",
    "    px_db_sample['prior_price_close_60'] = px_db_sample.groupby('ticker')['close'].shift(60)\n",
    "    px_db_sample['prior_price_close_120'] = px_db_sample.groupby('ticker')['close'].shift(120)\n",
    "    px_db_sample['prior_price_close_250'] = px_db_sample.groupby('ticker')['close'].shift(250)\n",
    "\n",
    "\n",
    "    px_db_sample['range'] = (px_db_sample['high'] - px_db_sample['low'])/ (px_db_sample['close']+eps)\n",
    "    px_db_sample['px_rank_chg'] = px_db_sample['next_px_rank'] - px_db_sample['px_rank'] \n",
    "\n",
    "\n",
    "    for w in [5,10,20,60]:\n",
    "        px_db_sample['90PctileRank'+'_'+str(w)] = px_db_sample.groupby('ticker')['px_rank'].apply(lambda x:x.rolling(window=w, min_periods=w).quantile(0.9))\n",
    "        px_db_sample['10PctileRank'+'_'+str(w)] = px_db_sample.groupby('ticker')['px_rank'].apply(lambda x:x.rolling(window=w, min_periods=w).quantile(0.1))\n",
    "        px_db_sample['MnRank'+'_'+str(w)] = 0.5*(px_db_sample['90PctileRank'+'_'+str(w)]  + px_db_sample['10PctileRank'+'_'+str(w)])\n",
    "        px_db_sample['RangeRank'+'_'+str(w)] = (px_db_sample['90PctileRank'+'_'+str(w)]  - px_db_sample['10PctileRank'+'_'+str(w)])\n",
    "        px_db_sample['Rank_Zscore'+'_'+str(w)] =  (px_db_sample['px_rank'] - px_db_sample['MnRank'+'_'+str(w)] )/px_db_sample['RangeRank'+'_'+str(w)]\n",
    "\n",
    "\n",
    "    px_db_sample['prior_ret'] = get_change(px_db_sample['close'] , px_db_sample['prior_price_close'] , eps)\n",
    "    px_db_sample['prior_ret_20'] = get_change(px_db_sample['close'] , px_db_sample['prior_price_close_20'] , eps)\n",
    "    px_db_sample['prior_ret_60'] = get_change(px_db_sample['close'] , px_db_sample['prior_price_close_60'] , eps)\n",
    "    px_db_sample['prior_ret_120'] = get_change(px_db_sample['close'] , px_db_sample['prior_price_close_120'] , eps)\n",
    "    px_db_sample['prior_ret_250'] = get_change(px_db_sample['close'] , px_db_sample['prior_price_close_250'] , eps)\n",
    "\n",
    "    px_db_sample['next_ret'] = get_change( px_db_sample['next_price_close'] ,px_db_sample['close'] , eps)\n",
    "    px_db_sample['next_ret_2'] = get_change(px_db_sample['next_price_close_2'] , px_db_sample['close'] , eps)\n",
    "    px_db_sample['next_ret_5'] = get_change(px_db_sample['next_price_close_5'] ,px_db_sample['close'] ,  eps)\n",
    "    px_db_sample['next_ret_10'] = get_change(px_db_sample['next_price_close_10'] ,px_db_sample['close'] ,  eps)\n",
    "    px_db_sample['next_ret_20'] = get_change(px_db_sample['next_price_close_20'] , px_db_sample['close'] , eps)\n",
    "    \n",
    "    \n",
    "    px_db_sample['vol_20'] = px_db_sample.groupby('ticker')['prior_ret'].apply(lambda x : x.rolling(20, min_periods = 1).std())\n",
    "    px_db_sample['vol_60'] = px_db_sample.groupby('ticker')['prior_ret'].apply(lambda x : x.rolling( 60, min_periods = 1).std())\n",
    "    px_db_sample['vol_90'] = px_db_sample.groupby('ticker')['prior_ret'].apply(lambda x : x.rolling( 90, min_periods = 1).std())\n",
    "    px_db_sample['vol_120'] = px_db_sample.groupby('ticker')['prior_ret'].apply(lambda x : x.rolling( 120, min_periods = 1).std())\n",
    "    px_db_sample['vol_250'] = px_db_sample.groupby('ticker')['prior_ret'].apply(lambda x : x.rolling( 250, min_periods = 1).std())\n",
    "\n",
    "\n",
    "\n",
    "    px_db_sample['DollarVolume'] = px_db_sample['close'] * px_db_sample['volume']\n",
    "    px_db_sample['adv90'] = px_db_sample.groupby('ticker')['volume'].apply(lambda x : x.rolling( 90, min_periods = 1).mean())/1000000\n",
    "    px_db_sample['dollar_adv90'] = px_db_sample.groupby('ticker')['DollarVolume'].apply(lambda x: x.rolling(90, min_periods = 1).mean())/1000000\n",
    "\n",
    "    px_db_sample['FiftyTwoWk_High'] = px_db_sample.groupby('ticker')['close'].apply(lambda x: x.rolling(window= 260, min_periods = 5).max()) \n",
    "    px_db_sample['FiftyTwoWk_Low'] = px_db_sample.groupby('ticker')['close'].apply(lambda x: x.rolling(window= 260, min_periods = 5).min()) \n",
    "\n",
    "    px_db_sample['px_to_52wk_high'] = get_change(px_db_sample['close'] , px_db_sample['FiftyTwoWk_High'] , eps)\n",
    "    px_db_sample['px_to_52wk_low'] = get_change(px_db_sample['close'] , px_db_sample['FiftyTwoWk_Low'] , eps)\n",
    "    \n",
    "    etf_db_cols = [ 'sector',\n",
    "                     'etf',\n",
    "                     \n",
    "                     'TradeDtKey',\n",
    "                     'prior_ret',\n",
    "                     'prior_ret_20',\n",
    "                     'prior_ret_60',\n",
    "                     'prior_ret_120',\n",
    "                     'prior_ret_250',\n",
    "                     'next_ret',\n",
    "                     'next_ret_2',\n",
    "                     'next_ret_5',\n",
    "                     'next_ret_10',\n",
    "                     'next_ret_20']\n",
    "    market_db = etf_db.loc[etf_db['etf'].isin(['SPY']),:][etf_db_cols]\n",
    "    market_db.rename(columns = dict(zip(etf_db_cols[3:], ['market_'+s for s in etf_db_cols[3:] ])), inplace = True)\n",
    "    \n",
    "    sector_db = etf_db.loc[etf_db['etf'].isin(sector_etf_list),:][etf_db_cols]\n",
    "    sector_db.rename(columns = dict(zip(etf_db_cols[3:], ['sector_'+s for s in etf_db_cols[3:] ])), inplace = True)    \n",
    "    \n",
    "    px_db_sample = pd.merge(px_db_sample,\n",
    "                            sector_db , \n",
    "                            left_on = ['Date' , 'sector'], \n",
    "                            right_on = ['TradeDtKey', 'sector'] ,\n",
    "                           how = 'left')\n",
    "    px_db_sample = pd.merge(px_db_sample,\n",
    "                            market_db[['TradeDtKey',\n",
    "                                     'market_prior_ret',\n",
    "                                     'market_prior_ret_20',\n",
    "                                     'market_prior_ret_60',\n",
    "                                     'market_prior_ret_120',\n",
    "                                     'market_prior_ret_250',\n",
    "                                     'market_next_ret',\n",
    "                                     'market_next_ret_2',\n",
    "                                     'market_next_ret_5',\n",
    "                                     'market_next_ret_10',\n",
    "                                     'market_next_ret_20']] , \n",
    "                            left_on = ['TradeDtKey'  ], \n",
    "                            right_on = ['TradeDtKey' ] ,\n",
    "                           how = 'left')    \n",
    "    \n",
    "    feat_db_gp = px_db_sample.groupby('ticker')\n",
    "    \n",
    "    Beta_trail = []\n",
    "    \n",
    "    for k,v in feat_db_gp:\n",
    "        v_tmp = v[['TradeDtKey', 'ticker', 'prior_ret', 'market_prior_ret', 'sector_prior_ret']].copy()\n",
    "        v_tmp.dropna(inplace = True)\n",
    "        v_tmp.reset_index(drop = True, inplace = True)\n",
    "        N = np.minimum(250, v_tmp.shape[0])\n",
    "        \n",
    "        if N>10:\n",
    "            beta = v['prior_ret'].rolling(window = N, min_periods=10).cov(v['market_prior_ret'])/v['market_prior_ret'].rolling(window = N, min_periods=10).var()\n",
    "            sec_beta = v['prior_ret'].rolling(window = N, min_periods=10).cov(v['sector_prior_ret'])/v['sector_prior_ret'].rolling(window = N, min_periods=10).var()\n",
    "        else:\n",
    "            beta = v['prior_ret'] * 0\n",
    "            sec_beta = v['prior_ret'] * 0\n",
    "        v_df = pd.DataFrame({'ticker': k,\n",
    "                            'TradeDtKey': v['TradeDtKey'].tolist() ,\n",
    "                            'Beta': beta.tolist(),\n",
    "                            'SecBeta':sec_beta.tolist() })\n",
    "        \n",
    "        Beta_trail.append(v_df)\n",
    "        \n",
    "    Beta_trail_df  = pd.concat(Beta_trail)\n",
    "    Beta_trail_df.reset_index(drop = True, inplace = True)\n",
    "    \n",
    "    px_db_sample = pd.merge(px_db_sample, \n",
    "                            Beta_trail_df,\n",
    "                           \n",
    "                            left_on = ['ticker' , 'TradeDtKey'], \n",
    "                            right_on = ['ticker', 'TradeDtKey'] ,\n",
    "                           how = 'left')    \n",
    "    \n",
    "    px_db_sample['market_relative_prior_ret'] = px_db_sample['prior_ret'] -px_db_sample['Beta']*px_db_sample['market_prior_ret']\n",
    "    px_db_sample['market_relative_prior_ret_20'] = px_db_sample['prior_ret_20'] -px_db_sample['Beta']*px_db_sample['market_prior_ret_20']\n",
    "    px_db_sample['market_relative_prior_ret_60'] = px_db_sample['prior_ret_60'] -px_db_sample['Beta']*px_db_sample['market_prior_ret_60']\n",
    "    px_db_sample['market_relative_prior_ret_120'] = px_db_sample['prior_ret_120'] -px_db_sample['Beta']*px_db_sample['market_prior_ret_120']\n",
    "    px_db_sample['market_relative_prior_ret_250'] = px_db_sample['prior_ret_250'] -px_db_sample['Beta']*px_db_sample['market_prior_ret_250']\n",
    "\n",
    "    px_db_sample['sector_relative_prior_ret'] = px_db_sample['prior_ret'] -px_db_sample['SecBeta']*px_db_sample['sector_prior_ret']\n",
    "    px_db_sample['sector_relative_prior_ret_20'] = px_db_sample['prior_ret_20'] -px_db_sample['SecBeta']*px_db_sample['sector_prior_ret_20']\n",
    "    px_db_sample['sector_relative_prior_ret_60'] = px_db_sample['prior_ret_60'] -px_db_sample['SecBeta']*px_db_sample['sector_prior_ret_60']\n",
    "    px_db_sample['sector_relative_prior_ret_120'] = px_db_sample['prior_ret_120'] -px_db_sample['SecBeta']*px_db_sample['sector_prior_ret_120']\n",
    "    px_db_sample['sector_relative_prior_ret_250'] = px_db_sample['prior_ret_250'] -px_db_sample['SecBeta']*px_db_sample['sector_prior_ret_250']\n",
    "    \n",
    "    \n",
    "    \n",
    "    \n",
    "    return px_db_sample\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "r8X0UIv3syJ-",
    "outputId": "228d2b34-a685-4bd7-f76a-624101916817"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:33: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:34: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n"
     ]
    }
   ],
   "source": [
    "feat_db = extract_features(sample_db_sp500, etf_db) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_gkpPO7UsyMJ"
   },
   "outputs": [],
   "source": [
    "feat_db.fillna(0, inplace = True)\n",
    "feat_db.replace(+np.Inf,0,inplace=True)\n",
    "feat_db.replace(-np.Inf,0,inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UCESflOcs9UR"
   },
   "source": [
    "### Potential features "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RptYBGkzs_GE"
   },
   "source": [
    "Potential features/drivers that can be used to predict/forecast future return may include:\n",
    "\n",
    "\n",
    " \n",
    " 'MnRank_5',\n",
    " 'RangeRank_5',\n",
    " 'Rank_Zscore_5',\n",
    " \n",
    " 'MnRank_10',\n",
    " 'RangeRank_10',\n",
    " 'Rank_Zscore_10',\n",
    " \n",
    " 'MnRank_20',\n",
    " 'RangeRank_20',\n",
    " 'Rank_Zscore_20',\n",
    " \n",
    " 'MnRank_60',\n",
    " 'RangeRank_60',\n",
    " 'Rank_Zscore_60',\n",
    " 'prior_ret',\n",
    " 'prior_ret_20',\n",
    " 'prior_ret_60',\n",
    " 'prior_ret_120',\n",
    " 'prior_ret_250',\n",
    " \n",
    " 'vol_20',\n",
    " 'vol_60',\n",
    " 'vol_90',\n",
    " 'vol_120',\n",
    " 'vol_250',\n",
    " \n",
    " 'adv90',\n",
    " 'dollar_adv90',\n",
    " \n",
    " 'px_to_52wk_high',\n",
    " 'px_to_52wk_low',\n",
    " \n",
    " \n",
    " 'sector_prior_ret',\n",
    " 'sector_prior_ret_20',\n",
    " 'sector_prior_ret_60',\n",
    " 'sector_prior_ret_120',\n",
    " 'sector_prior_ret_250',\n",
    " \n",
    " \n",
    " \n",
    " 'market_prior_ret',\n",
    " 'market_prior_ret_20',\n",
    " 'market_prior_ret_60',\n",
    " 'market_prior_ret_120',\n",
    " 'market_prior_ret_250',\n",
    " \n",
    " 'Beta',\n",
    " 'SecBeta',\n",
    " 'market_relative_prior_ret',\n",
    " 'market_relative_prior_ret_20',\n",
    " 'market_relative_prior_ret_60',\n",
    " 'market_relative_prior_ret_120',\n",
    " 'market_relative_prior_ret_250',\n",
    " 'sector_relative_prior_ret',\n",
    " 'sector_relative_prior_ret_20',\n",
    " 'sector_relative_prior_ret_60',\n",
    " 'sector_relative_prior_ret_120',\n",
    " 'sector_relative_prior_ret_250'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "55cxQnzmtBK2"
   },
   "source": [
    "### Potential response variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QJjhNKfUtCKP"
   },
   "source": [
    "Potential response variables may include:\n",
    "    \n",
    "'next_ret',\n",
    "'next_ret_2',\n",
    "'next_ret_5',\n",
    "'next_ret_10',\n",
    "'next_ret_20'\n",
    "    \n",
    "But can also include market relative returns or sector relative returns.    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-cxb27vmBBDQ"
   },
   "source": [
    "## Jumps Identification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LtvNPdJgKWtE"
   },
   "source": [
    "#### Close2Open returns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kMGxAWk7A_Ua"
   },
   "outputs": [],
   "source": [
    "def close2open(df):\n",
    "  s=np.log(df[\"open\"]/df.shift(1)[\"close\"])\n",
    "  s.rename(\"Close2Open\",inplace=True)\n",
    "  return s\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "zOuBO7dfMfES"
   },
   "outputs": [],
   "source": [
    "feat_db[\"Close2Open\"]=np.nan\n",
    "\n",
    "for ticker in feat_db[\"ticker\"].unique():\n",
    "  df=close2open(feat_db[feat_db[\"ticker\"]==ticker])\n",
    "  feat_db.update(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0e5XXehLNZmY"
   },
   "source": [
    "#### PV30"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "iKgTDuQpNY-T"
   },
   "outputs": [],
   "source": [
    "def log_high_low(df):\n",
    "  ind=df.index\n",
    "  return pd.DataFrame(np.log(df[\"high\"]/df[\"low\"]),index=ind)\n",
    "\n",
    "def PV30(df):\n",
    "  df_ret=log_high_low(df).rolling(30,min_periods = 1).std()\n",
    "  df_ret.columns=[\"PV30\"]\n",
    "  return df_ret"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7DkktqBeNZEG"
   },
   "outputs": [],
   "source": [
    "feat_db[\"PV30\"]=np.nan\n",
    "\n",
    "for ticker in feat_db[\"ticker\"].unique():\n",
    "  df=PV30(feat_db[feat_db[\"ticker\"]==ticker])\n",
    "  feat_db.update(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FV2W1yRkQKZY"
   },
   "source": [
    "#### Median PV30"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "UAw37BKAQZGg"
   },
   "outputs": [],
   "source": [
    "def medianPV30(df):\n",
    "  df_ret=df[\"PV30\"].rolling(30,min_periods = 1).median()\n",
    "  df_ret=pd.DataFrame(df_ret)\n",
    "  df_ret.columns=[\"median PV30\"]\n",
    "  return df_ret"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "TJcfEIxdQZIh"
   },
   "outputs": [],
   "source": [
    "feat_db[\"median PV30\"]=np.nan\n",
    "\n",
    "for ticker in feat_db[\"ticker\"].unique():\n",
    "  df=medianPV30(feat_db[feat_db[\"ticker\"]==ticker])\n",
    "  feat_db.update(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2Pxvo2mdQ3oA"
   },
   "source": [
    "#### ^VIX"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 238
    },
    "id": "4_i1hYhhQZKm",
    "outputId": "c00dcb83-521d-471a-a39b-41b58f45954b"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Open</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "      <th>Adj Close</th>\n",
       "      <th>date</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2010-01-04</th>\n",
       "      <td>21.680000</td>\n",
       "      <td>20.030001</td>\n",
       "      <td>21.680000</td>\n",
       "      <td>20.040001</td>\n",
       "      <td>0</td>\n",
       "      <td>20.040001</td>\n",
       "      <td>2010-01-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-05</th>\n",
       "      <td>20.129999</td>\n",
       "      <td>19.340000</td>\n",
       "      <td>20.049999</td>\n",
       "      <td>19.350000</td>\n",
       "      <td>0</td>\n",
       "      <td>19.350000</td>\n",
       "      <td>2010-01-05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-06</th>\n",
       "      <td>19.680000</td>\n",
       "      <td>18.770000</td>\n",
       "      <td>19.590000</td>\n",
       "      <td>19.160000</td>\n",
       "      <td>0</td>\n",
       "      <td>19.160000</td>\n",
       "      <td>2010-01-06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-07</th>\n",
       "      <td>19.709999</td>\n",
       "      <td>18.700001</td>\n",
       "      <td>19.680000</td>\n",
       "      <td>19.059999</td>\n",
       "      <td>0</td>\n",
       "      <td>19.059999</td>\n",
       "      <td>2010-01-07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-08</th>\n",
       "      <td>19.270000</td>\n",
       "      <td>18.110001</td>\n",
       "      <td>19.270000</td>\n",
       "      <td>18.129999</td>\n",
       "      <td>0</td>\n",
       "      <td>18.129999</td>\n",
       "      <td>2010-01-08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 High        Low       Open  ...  Volume  Adj Close       date\n",
       "Date                                         ...                              \n",
       "2010-01-04  21.680000  20.030001  21.680000  ...       0  20.040001 2010-01-04\n",
       "2010-01-05  20.129999  19.340000  20.049999  ...       0  19.350000 2010-01-05\n",
       "2010-01-06  19.680000  18.770000  19.590000  ...       0  19.160000 2010-01-06\n",
       "2010-01-07  19.709999  18.700001  19.680000  ...       0  19.059999 2010-01-07\n",
       "2010-01-08  19.270000  18.110001  19.270000  ...       0  18.129999 2010-01-08\n",
       "\n",
       "[5 rows x 7 columns]"
      ]
     },
     "execution_count": 26,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "start = datetime.datetime(2010, 1, 1)\n",
    "end = datetime.date.today()\n",
    "VIX = web.DataReader(\"^VIX\",'yahoo',  start, end)\n",
    "VIX.index=pd.to_datetime(VIX.index, format='%Y-%m-%d', errors='ignore')\n",
    "VIX[\"date\"]=VIX.index\n",
    "VIX.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "k-eH7RszRoPw"
   },
   "outputs": [],
   "source": [
    "VIX[\"median VIX 30 days\"]=VIX[\"Adj Close\"].rolling(30,min_periods = 1).median()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "wRYovy1DTjL0"
   },
   "outputs": [],
   "source": [
    "VIX_subset=VIX[[\"date\",\"Close\",\"median VIX 30 days\"]]\n",
    "VIX_subset.columns=[\"date\",\"VIX\",\"median VIX 30 days\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "B3yCE1ESSnsC"
   },
   "outputs": [],
   "source": [
    "feat_db=pd.merge(feat_db, VIX_subset, how='outer', on='date')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Rk7HirBwrMii"
   },
   "outputs": [],
   "source": [
    "feat_db.dropna(inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 343
    },
    "id": "LrnyoFyZeQls",
    "outputId": "1877cc9c-b3a3-488b-fabb-7017ca39602a"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ticker</th>\n",
       "      <th>date</th>\n",
       "      <th>open</th>\n",
       "      <th>high</th>\n",
       "      <th>low</th>\n",
       "      <th>close</th>\n",
       "      <th>volume</th>\n",
       "      <th>dividends</th>\n",
       "      <th>closeunadj</th>\n",
       "      <th>lastupdated</th>\n",
       "      <th>sector</th>\n",
       "      <th>scalemarketcap</th>\n",
       "      <th>exchange</th>\n",
       "      <th>year</th>\n",
       "      <th>Date</th>\n",
       "      <th>px_rank</th>\n",
       "      <th>next_px_rank</th>\n",
       "      <th>next_price_close</th>\n",
       "      <th>next_price_close_2</th>\n",
       "      <th>next_price_close_5</th>\n",
       "      <th>next_price_close_10</th>\n",
       "      <th>next_price_close_20</th>\n",
       "      <th>prior_price_close</th>\n",
       "      <th>prior_price_close_20</th>\n",
       "      <th>prior_price_close_60</th>\n",
       "      <th>prior_price_close_120</th>\n",
       "      <th>prior_price_close_250</th>\n",
       "      <th>range</th>\n",
       "      <th>px_rank_chg</th>\n",
       "      <th>90PctileRank_5</th>\n",
       "      <th>10PctileRank_5</th>\n",
       "      <th>MnRank_5</th>\n",
       "      <th>RangeRank_5</th>\n",
       "      <th>Rank_Zscore_5</th>\n",
       "      <th>90PctileRank_10</th>\n",
       "      <th>10PctileRank_10</th>\n",
       "      <th>MnRank_10</th>\n",
       "      <th>RangeRank_10</th>\n",
       "      <th>Rank_Zscore_10</th>\n",
       "      <th>90PctileRank_20</th>\n",
       "      <th>...</th>\n",
       "      <th>px_to_52wk_low</th>\n",
       "      <th>etf</th>\n",
       "      <th>TradeDtKey</th>\n",
       "      <th>sector_prior_ret</th>\n",
       "      <th>sector_prior_ret_20</th>\n",
       "      <th>sector_prior_ret_60</th>\n",
       "      <th>sector_prior_ret_120</th>\n",
       "      <th>sector_prior_ret_250</th>\n",
       "      <th>sector_next_ret</th>\n",
       "      <th>sector_next_ret_2</th>\n",
       "      <th>sector_next_ret_5</th>\n",
       "      <th>sector_next_ret_10</th>\n",
       "      <th>sector_next_ret_20</th>\n",
       "      <th>market_prior_ret</th>\n",
       "      <th>market_prior_ret_20</th>\n",
       "      <th>market_prior_ret_60</th>\n",
       "      <th>market_prior_ret_120</th>\n",
       "      <th>market_prior_ret_250</th>\n",
       "      <th>market_next_ret</th>\n",
       "      <th>market_next_ret_2</th>\n",
       "      <th>market_next_ret_5</th>\n",
       "      <th>market_next_ret_10</th>\n",
       "      <th>market_next_ret_20</th>\n",
       "      <th>Beta</th>\n",
       "      <th>SecBeta</th>\n",
       "      <th>market_relative_prior_ret</th>\n",
       "      <th>market_relative_prior_ret_20</th>\n",
       "      <th>market_relative_prior_ret_60</th>\n",
       "      <th>market_relative_prior_ret_120</th>\n",
       "      <th>market_relative_prior_ret_250</th>\n",
       "      <th>sector_relative_prior_ret</th>\n",
       "      <th>sector_relative_prior_ret_20</th>\n",
       "      <th>sector_relative_prior_ret_60</th>\n",
       "      <th>sector_relative_prior_ret_120</th>\n",
       "      <th>sector_relative_prior_ret_250</th>\n",
       "      <th>Close2Open</th>\n",
       "      <th>PV30</th>\n",
       "      <th>median PV30</th>\n",
       "      <th>VIX</th>\n",
       "      <th>median VIX 30 days</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2153701</th>\n",
       "      <td>FCX</td>\n",
       "      <td>2019-12-31</td>\n",
       "      <td>12.98</td>\n",
       "      <td>13.15</td>\n",
       "      <td>12.92</td>\n",
       "      <td>13.12</td>\n",
       "      <td>11716373.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13.12</td>\n",
       "      <td>2020-05-01</td>\n",
       "      <td>Basic Materials</td>\n",
       "      <td>5 - Large</td>\n",
       "      <td>NYSE</td>\n",
       "      <td>2019</td>\n",
       "      <td>20191231.0</td>\n",
       "      <td>0.948590</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13.01</td>\n",
       "      <td>11.45</td>\n",
       "      <td>8.83</td>\n",
       "      <td>11.10</td>\n",
       "      <td>10.07</td>\n",
       "      <td>0.017530</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.948743</td>\n",
       "      <td>0.947107</td>\n",
       "      <td>0.947925</td>\n",
       "      <td>0.001637</td>\n",
       "      <td>0.406691</td>\n",
       "      <td>0.949010</td>\n",
       "      <td>0.946932</td>\n",
       "      <td>0.947971</td>\n",
       "      <td>0.002078</td>\n",
       "      <td>0.298104</td>\n",
       "      <td>0.949010</td>\n",
       "      <td>...</td>\n",
       "      <td>4328.958398</td>\n",
       "      <td>XLB</td>\n",
       "      <td>20191231.0</td>\n",
       "      <td>73.534275</td>\n",
       "      <td>382.657263</td>\n",
       "      <td>923.609523</td>\n",
       "      <td>712.595269</td>\n",
       "      <td>2397.861093</td>\n",
       "      <td>-117.917207</td>\n",
       "      <td>-280.684429</td>\n",
       "      <td>-300.798358</td>\n",
       "      <td>-149.268548</td>\n",
       "      <td>-413.851665</td>\n",
       "      <td>24.263135</td>\n",
       "      <td>371.723568</td>\n",
       "      <td>942.515524</td>\n",
       "      <td>821.4874</td>\n",
       "      <td>2948.469634</td>\n",
       "      <td>93.084108</td>\n",
       "      <td>17.074148</td>\n",
       "      <td>80.149258</td>\n",
       "      <td>194.760548</td>\n",
       "      <td>179.208913</td>\n",
       "      <td>1.868948</td>\n",
       "      <td>1.603378</td>\n",
       "      <td>38.848367</td>\n",
       "      <td>666.748546</td>\n",
       "      <td>2198.315272</td>\n",
       "      <td>136.609595</td>\n",
       "      <td>-2864.765362</td>\n",
       "      <td>-33.708354</td>\n",
       "      <td>747.936194</td>\n",
       "      <td>2478.932257</td>\n",
       "      <td>529.366988</td>\n",
       "      <td>-1198.907564</td>\n",
       "      <td>-0.002309</td>\n",
       "      <td>0.010229</td>\n",
       "      <td>0.012756</td>\n",
       "      <td>13.78</td>\n",
       "      <td>12.725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2153702</th>\n",
       "      <td>FDX</td>\n",
       "      <td>2019-12-31</td>\n",
       "      <td>149.22</td>\n",
       "      <td>151.57</td>\n",
       "      <td>148.75</td>\n",
       "      <td>151.21</td>\n",
       "      <td>2153367.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>151.21</td>\n",
       "      <td>2020-05-01</td>\n",
       "      <td>Industrials</td>\n",
       "      <td>5 - Large</td>\n",
       "      <td>NYSE</td>\n",
       "      <td>2019</td>\n",
       "      <td>20191231.0</td>\n",
       "      <td>0.223881</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>150.14</td>\n",
       "      <td>158.03</td>\n",
       "      <td>143.00</td>\n",
       "      <td>162.60</td>\n",
       "      <td>157.19</td>\n",
       "      <td>0.018650</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.225092</td>\n",
       "      <td>0.219250</td>\n",
       "      <td>0.222171</td>\n",
       "      <td>0.005842</td>\n",
       "      <td>0.292590</td>\n",
       "      <td>0.229500</td>\n",
       "      <td>0.215443</td>\n",
       "      <td>0.222471</td>\n",
       "      <td>0.014057</td>\n",
       "      <td>0.100258</td>\n",
       "      <td>0.227415</td>\n",
       "      <td>...</td>\n",
       "      <td>885.938128</td>\n",
       "      <td>XLI</td>\n",
       "      <td>20191231.0</td>\n",
       "      <td>-6.135062</td>\n",
       "      <td>145.791067</td>\n",
       "      <td>796.051492</td>\n",
       "      <td>661.548192</td>\n",
       "      <td>2810.892954</td>\n",
       "      <td>187.261877</td>\n",
       "      <td>167.968265</td>\n",
       "      <td>184.852911</td>\n",
       "      <td>254.496949</td>\n",
       "      <td>189.671196</td>\n",
       "      <td>24.263135</td>\n",
       "      <td>371.723568</td>\n",
       "      <td>942.515524</td>\n",
       "      <td>821.4874</td>\n",
       "      <td>2948.469634</td>\n",
       "      <td>93.084108</td>\n",
       "      <td>17.074148</td>\n",
       "      <td>80.149258</td>\n",
       "      <td>194.760548</td>\n",
       "      <td>179.208913</td>\n",
       "      <td>1.469400</td>\n",
       "      <td>1.371427</td>\n",
       "      <td>35.361816</td>\n",
       "      <td>-987.363544</td>\n",
       "      <td>-826.682722</td>\n",
       "      <td>-1933.329652</td>\n",
       "      <td>-4720.338246</td>\n",
       "      <td>79.427856</td>\n",
       "      <td>-641.094641</td>\n",
       "      <td>-533.476502</td>\n",
       "      <td>-1633.500768</td>\n",
       "      <td>-4242.790022</td>\n",
       "      <td>-0.006146</td>\n",
       "      <td>0.009344</td>\n",
       "      <td>0.006871</td>\n",
       "      <td>13.78</td>\n",
       "      <td>12.725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2153703</th>\n",
       "      <td>FE</td>\n",
       "      <td>2019-12-31</td>\n",
       "      <td>48.49</td>\n",
       "      <td>48.61</td>\n",
       "      <td>48.18</td>\n",
       "      <td>48.60</td>\n",
       "      <td>2929696.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>48.60</td>\n",
       "      <td>2020-05-01</td>\n",
       "      <td>Utilities</td>\n",
       "      <td>5 - Large</td>\n",
       "      <td>NYSE</td>\n",
       "      <td>2019</td>\n",
       "      <td>20191231.0</td>\n",
       "      <td>0.688226</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>48.49</td>\n",
       "      <td>47.52</td>\n",
       "      <td>48.23</td>\n",
       "      <td>43.59</td>\n",
       "      <td>36.71</td>\n",
       "      <td>0.008848</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.695228</td>\n",
       "      <td>0.688226</td>\n",
       "      <td>0.691727</td>\n",
       "      <td>0.007002</td>\n",
       "      <td>-0.500000</td>\n",
       "      <td>0.693429</td>\n",
       "      <td>0.686313</td>\n",
       "      <td>0.689871</td>\n",
       "      <td>0.007116</td>\n",
       "      <td>-0.231201</td>\n",
       "      <td>0.693069</td>\n",
       "      <td>...</td>\n",
       "      <td>3045.589425</td>\n",
       "      <td>XLU</td>\n",
       "      <td>20191231.0</td>\n",
       "      <td>38.762464</td>\n",
       "      <td>388.995982</td>\n",
       "      <td>54.612029</td>\n",
       "      <td>729.698826</td>\n",
       "      <td>2481.041029</td>\n",
       "      <td>-126.140747</td>\n",
       "      <td>-105.789247</td>\n",
       "      <td>-115.176770</td>\n",
       "      <td>194.627054</td>\n",
       "      <td>697.766052</td>\n",
       "      <td>24.263135</td>\n",
       "      <td>371.723568</td>\n",
       "      <td>942.515524</td>\n",
       "      <td>821.4874</td>\n",
       "      <td>2948.469634</td>\n",
       "      <td>93.084108</td>\n",
       "      <td>17.074148</td>\n",
       "      <td>80.149258</td>\n",
       "      <td>194.760548</td>\n",
       "      <td>179.208913</td>\n",
       "      <td>0.312841</td>\n",
       "      <td>0.979639</td>\n",
       "      <td>15.068886</td>\n",
       "      <td>108.438083</td>\n",
       "      <td>-218.434789</td>\n",
       "      <td>830.962481</td>\n",
       "      <td>1883.340341</td>\n",
       "      <td>-15.313836</td>\n",
       "      <td>-156.347186</td>\n",
       "      <td>22.922878</td>\n",
       "      <td>373.116013</td>\n",
       "      <td>375.218091</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.003407</td>\n",
       "      <td>0.004588</td>\n",
       "      <td>13.78</td>\n",
       "      <td>12.725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2153704</th>\n",
       "      <td>F</td>\n",
       "      <td>2019-12-31</td>\n",
       "      <td>9.25</td>\n",
       "      <td>9.33</td>\n",
       "      <td>9.24</td>\n",
       "      <td>9.30</td>\n",
       "      <td>32342009.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.30</td>\n",
       "      <td>2020-05-01</td>\n",
       "      <td>Consumer Cyclical</td>\n",
       "      <td>5 - Large</td>\n",
       "      <td>NYSE</td>\n",
       "      <td>2019</td>\n",
       "      <td>20191231.0</td>\n",
       "      <td>0.965174</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.25</td>\n",
       "      <td>9.01</td>\n",
       "      <td>8.74</td>\n",
       "      <td>10.19</td>\n",
       "      <td>7.78</td>\n",
       "      <td>0.009677</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.965174</td>\n",
       "      <td>0.963744</td>\n",
       "      <td>0.964459</td>\n",
       "      <td>0.001430</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.965191</td>\n",
       "      <td>0.962084</td>\n",
       "      <td>0.963637</td>\n",
       "      <td>0.003108</td>\n",
       "      <td>0.494452</td>\n",
       "      <td>0.965174</td>\n",
       "      <td>...</td>\n",
       "      <td>1979.265536</td>\n",
       "      <td>XLY</td>\n",
       "      <td>20191231.0</td>\n",
       "      <td>14.362293</td>\n",
       "      <td>355.391800</td>\n",
       "      <td>539.890839</td>\n",
       "      <td>287.735919</td>\n",
       "      <td>2642.423209</td>\n",
       "      <td>118.101130</td>\n",
       "      <td>32.636667</td>\n",
       "      <td>74.668289</td>\n",
       "      <td>78.625087</td>\n",
       "      <td>-33.543752</td>\n",
       "      <td>24.263135</td>\n",
       "      <td>371.723568</td>\n",
       "      <td>942.515524</td>\n",
       "      <td>821.4874</td>\n",
       "      <td>2948.469634</td>\n",
       "      <td>93.084108</td>\n",
       "      <td>17.074148</td>\n",
       "      <td>80.149258</td>\n",
       "      <td>194.760548</td>\n",
       "      <td>179.208913</td>\n",
       "      <td>1.072312</td>\n",
       "      <td>1.038522</td>\n",
       "      <td>27.890836</td>\n",
       "      <td>-81.810192</td>\n",
       "      <td>-389.628164</td>\n",
       "      <td>-1794.814886</td>\n",
       "      <td>-1377.097360</td>\n",
       "      <td>38.992912</td>\n",
       "      <td>-52.289085</td>\n",
       "      <td>60.353342</td>\n",
       "      <td>-1212.744688</td>\n",
       "      <td>-959.635189</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.005096</td>\n",
       "      <td>0.005374</td>\n",
       "      <td>13.78</td>\n",
       "      <td>12.725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2153705</th>\n",
       "      <td>DG</td>\n",
       "      <td>2019-12-31</td>\n",
       "      <td>157.17</td>\n",
       "      <td>157.45</td>\n",
       "      <td>155.23</td>\n",
       "      <td>155.98</td>\n",
       "      <td>1097189.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>155.98</td>\n",
       "      <td>2020-05-01</td>\n",
       "      <td>Consumer Defensive</td>\n",
       "      <td>5 - Large</td>\n",
       "      <td>NYSE</td>\n",
       "      <td>2019</td>\n",
       "      <td>20191231.0</td>\n",
       "      <td>0.203980</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>157.35</td>\n",
       "      <td>154.68</td>\n",
       "      <td>160.97</td>\n",
       "      <td>140.36</td>\n",
       "      <td>107.84</td>\n",
       "      <td>0.014233</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.205355</td>\n",
       "      <td>0.203980</td>\n",
       "      <td>0.204668</td>\n",
       "      <td>0.001375</td>\n",
       "      <td>-0.500000</td>\n",
       "      <td>0.205702</td>\n",
       "      <td>0.200564</td>\n",
       "      <td>0.203133</td>\n",
       "      <td>0.005138</td>\n",
       "      <td>0.164909</td>\n",
       "      <td>0.206271</td>\n",
       "      <td>...</td>\n",
       "      <td>4561.242442</td>\n",
       "      <td>XLP</td>\n",
       "      <td>20191231.0</td>\n",
       "      <td>9.531811</td>\n",
       "      <td>215.853029</td>\n",
       "      <td>335.653213</td>\n",
       "      <td>682.121515</td>\n",
       "      <td>2541.147785</td>\n",
       "      <td>-79.706836</td>\n",
       "      <td>-95.725099</td>\n",
       "      <td>-114.980822</td>\n",
       "      <td>88.524301</td>\n",
       "      <td>145.021778</td>\n",
       "      <td>24.263135</td>\n",
       "      <td>371.723568</td>\n",
       "      <td>942.515524</td>\n",
       "      <td>821.4874</td>\n",
       "      <td>2948.469634</td>\n",
       "      <td>93.084108</td>\n",
       "      <td>17.074148</td>\n",
       "      <td>80.149258</td>\n",
       "      <td>194.760548</td>\n",
       "      <td>179.208913</td>\n",
       "      <td>0.637505</td>\n",
       "      <td>0.583349</td>\n",
       "      <td>-102.916177</td>\n",
       "      <td>-153.282524</td>\n",
       "      <td>-915.760952</td>\n",
       "      <td>531.469750</td>\n",
       "      <td>1811.126053</td>\n",
       "      <td>-93.008667</td>\n",
       "      <td>-42.224337</td>\n",
       "      <td>-510.705086</td>\n",
       "      <td>657.257657</td>\n",
       "      <td>2208.415930</td>\n",
       "      <td>-0.001145</td>\n",
       "      <td>0.005840</td>\n",
       "      <td>0.007188</td>\n",
       "      <td>13.78</td>\n",
       "      <td>12.725</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 110 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        ticker       date    open  ...  median PV30    VIX  median VIX 30 days\n",
       "2153701    FCX 2019-12-31   12.98  ...     0.012756  13.78              12.725\n",
       "2153702    FDX 2019-12-31  149.22  ...     0.006871  13.78              12.725\n",
       "2153703     FE 2019-12-31   48.49  ...     0.004588  13.78              12.725\n",
       "2153704      F 2019-12-31    9.25  ...     0.005374  13.78              12.725\n",
       "2153705     DG 2019-12-31  157.17  ...     0.007188  13.78              12.725\n",
       "\n",
       "[5 rows x 110 columns]"
      ]
     },
     "execution_count": 31,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feat_db.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4ZrUbokWR6yB"
   },
   "source": [
    "#### JUMP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "FTBS-djUFAgq"
   },
   "outputs": [],
   "source": [
    "#feat_db=pd.read_csv(\"/content/drive/MyDrive/data/feat_db.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0YsfdXzGR-NO"
   },
   "outputs": [],
   "source": [
    "difference=pd.DataFrame(feat_db[\"Close2Open\"]-feat_db[\"median PV30\"]*feat_db[\"median VIX 30 days\"]/feat_db[\"VIX\"].shift(1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "UGNg8xaSUWgw"
   },
   "outputs": [],
   "source": [
    "feat_db[\"JUMP\"]=np.sign(difference)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Kkf1iiHfVMQC"
   },
   "outputs": [],
   "source": [
    "JUMP_subset=feat_db[feat_db[\"JUMP\"]==1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 377
    },
    "id": "4j_MclTiRs9J",
    "outputId": "54dab7de-9554-4a05-be71-206a44984ae1"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ticker</th>\n",
       "      <th>date</th>\n",
       "      <th>open</th>\n",
       "      <th>high</th>\n",
       "      <th>low</th>\n",
       "      <th>close</th>\n",
       "      <th>volume</th>\n",
       "      <th>dividends</th>\n",
       "      <th>closeunadj</th>\n",
       "      <th>lastupdated</th>\n",
       "      <th>sector</th>\n",
       "      <th>scalemarketcap</th>\n",
       "      <th>exchange</th>\n",
       "      <th>year</th>\n",
       "      <th>Date</th>\n",
       "      <th>px_rank</th>\n",
       "      <th>next_px_rank</th>\n",
       "      <th>next_price_close</th>\n",
       "      <th>next_price_close_2</th>\n",
       "      <th>next_price_close_5</th>\n",
       "      <th>next_price_close_10</th>\n",
       "      <th>next_price_close_20</th>\n",
       "      <th>prior_price_close</th>\n",
       "      <th>prior_price_close_20</th>\n",
       "      <th>prior_price_close_60</th>\n",
       "      <th>prior_price_close_120</th>\n",
       "      <th>prior_price_close_250</th>\n",
       "      <th>range</th>\n",
       "      <th>px_rank_chg</th>\n",
       "      <th>90PctileRank_5</th>\n",
       "      <th>10PctileRank_5</th>\n",
       "      <th>MnRank_5</th>\n",
       "      <th>RangeRank_5</th>\n",
       "      <th>Rank_Zscore_5</th>\n",
       "      <th>90PctileRank_10</th>\n",
       "      <th>10PctileRank_10</th>\n",
       "      <th>MnRank_10</th>\n",
       "      <th>RangeRank_10</th>\n",
       "      <th>Rank_Zscore_10</th>\n",
       "      <th>90PctileRank_20</th>\n",
       "      <th>...</th>\n",
       "      <th>etf</th>\n",
       "      <th>TradeDtKey</th>\n",
       "      <th>sector_prior_ret</th>\n",
       "      <th>sector_prior_ret_20</th>\n",
       "      <th>sector_prior_ret_60</th>\n",
       "      <th>sector_prior_ret_120</th>\n",
       "      <th>sector_prior_ret_250</th>\n",
       "      <th>sector_next_ret</th>\n",
       "      <th>sector_next_ret_2</th>\n",
       "      <th>sector_next_ret_5</th>\n",
       "      <th>sector_next_ret_10</th>\n",
       "      <th>sector_next_ret_20</th>\n",
       "      <th>market_prior_ret</th>\n",
       "      <th>market_prior_ret_20</th>\n",
       "      <th>market_prior_ret_60</th>\n",
       "      <th>market_prior_ret_120</th>\n",
       "      <th>market_prior_ret_250</th>\n",
       "      <th>market_next_ret</th>\n",
       "      <th>market_next_ret_2</th>\n",
       "      <th>market_next_ret_5</th>\n",
       "      <th>market_next_ret_10</th>\n",
       "      <th>market_next_ret_20</th>\n",
       "      <th>Beta</th>\n",
       "      <th>SecBeta</th>\n",
       "      <th>market_relative_prior_ret</th>\n",
       "      <th>market_relative_prior_ret_20</th>\n",
       "      <th>market_relative_prior_ret_60</th>\n",
       "      <th>market_relative_prior_ret_120</th>\n",
       "      <th>market_relative_prior_ret_250</th>\n",
       "      <th>sector_relative_prior_ret</th>\n",
       "      <th>sector_relative_prior_ret_20</th>\n",
       "      <th>sector_relative_prior_ret_60</th>\n",
       "      <th>sector_relative_prior_ret_120</th>\n",
       "      <th>sector_relative_prior_ret_250</th>\n",
       "      <th>Close2Open</th>\n",
       "      <th>PV30</th>\n",
       "      <th>median PV30</th>\n",
       "      <th>VIX</th>\n",
       "      <th>median VIX 30 days</th>\n",
       "      <th>JUMP</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>811</th>\n",
       "      <td>EQIX</td>\n",
       "      <td>2015-01-02</td>\n",
       "      <td>228.51</td>\n",
       "      <td>229.56</td>\n",
       "      <td>224.28</td>\n",
       "      <td>226.65</td>\n",
       "      <td>383808.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>226.65</td>\n",
       "      <td>2018-06-13</td>\n",
       "      <td>Real Estate</td>\n",
       "      <td>5 - Large</td>\n",
       "      <td>NASDAQ</td>\n",
       "      <td>2015</td>\n",
       "      <td>20150102.0</td>\n",
       "      <td>0.026826</td>\n",
       "      <td>0.02651</td>\n",
       "      <td>224.32</td>\n",
       "      <td>219.43</td>\n",
       "      <td>216.89</td>\n",
       "      <td>220.24</td>\n",
       "      <td>218.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.023296</td>\n",
       "      <td>-0.000316</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.008173</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>17.790001</td>\n",
       "      <td>14.65</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>812</th>\n",
       "      <td>EQIX</td>\n",
       "      <td>2015-01-02</td>\n",
       "      <td>228.51</td>\n",
       "      <td>229.56</td>\n",
       "      <td>224.28</td>\n",
       "      <td>226.65</td>\n",
       "      <td>383808.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>226.65</td>\n",
       "      <td>2018-06-13</td>\n",
       "      <td>Real Estate</td>\n",
       "      <td>5 - Large</td>\n",
       "      <td>NASDAQ</td>\n",
       "      <td>2015</td>\n",
       "      <td>20150102.0</td>\n",
       "      <td>0.026826</td>\n",
       "      <td>0.02651</td>\n",
       "      <td>224.32</td>\n",
       "      <td>219.43</td>\n",
       "      <td>216.89</td>\n",
       "      <td>220.24</td>\n",
       "      <td>218.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.023296</td>\n",
       "      <td>-0.000316</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.008173</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>17.790001</td>\n",
       "      <td>14.65</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>813</th>\n",
       "      <td>EQIX</td>\n",
       "      <td>2015-01-02</td>\n",
       "      <td>228.51</td>\n",
       "      <td>229.56</td>\n",
       "      <td>224.28</td>\n",
       "      <td>226.65</td>\n",
       "      <td>383808.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>226.65</td>\n",
       "      <td>2018-06-13</td>\n",
       "      <td>Real Estate</td>\n",
       "      <td>5 - Large</td>\n",
       "      <td>NASDAQ</td>\n",
       "      <td>2015</td>\n",
       "      <td>20150102.0</td>\n",
       "      <td>0.026826</td>\n",
       "      <td>0.02651</td>\n",
       "      <td>224.32</td>\n",
       "      <td>219.43</td>\n",
       "      <td>216.89</td>\n",
       "      <td>220.24</td>\n",
       "      <td>218.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.023296</td>\n",
       "      <td>-0.000316</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.008173</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>17.790001</td>\n",
       "      <td>14.65</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>814</th>\n",
       "      <td>EQIX</td>\n",
       "      <td>2015-01-02</td>\n",
       "      <td>228.51</td>\n",
       "      <td>229.56</td>\n",
       "      <td>224.28</td>\n",
       "      <td>226.65</td>\n",
       "      <td>383808.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>226.65</td>\n",
       "      <td>2018-06-13</td>\n",
       "      <td>Real Estate</td>\n",
       "      <td>5 - Large</td>\n",
       "      <td>NASDAQ</td>\n",
       "      <td>2015</td>\n",
       "      <td>20150102.0</td>\n",
       "      <td>0.026826</td>\n",
       "      <td>0.02651</td>\n",
       "      <td>224.32</td>\n",
       "      <td>219.43</td>\n",
       "      <td>216.89</td>\n",
       "      <td>220.24</td>\n",
       "      <td>218.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.023296</td>\n",
       "      <td>-0.000316</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.008173</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>17.790001</td>\n",
       "      <td>14.65</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>815</th>\n",
       "      <td>EQIX</td>\n",
       "      <td>2015-01-02</td>\n",
       "      <td>228.51</td>\n",
       "      <td>229.56</td>\n",
       "      <td>224.28</td>\n",
       "      <td>226.65</td>\n",
       "      <td>383808.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>226.65</td>\n",
       "      <td>2018-06-13</td>\n",
       "      <td>Real Estate</td>\n",
       "      <td>5 - Large</td>\n",
       "      <td>NASDAQ</td>\n",
       "      <td>2015</td>\n",
       "      <td>20150102.0</td>\n",
       "      <td>0.026826</td>\n",
       "      <td>0.02651</td>\n",
       "      <td>224.32</td>\n",
       "      <td>219.43</td>\n",
       "      <td>216.89</td>\n",
       "      <td>220.24</td>\n",
       "      <td>218.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.023296</td>\n",
       "      <td>-0.000316</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.008173</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>17.790001</td>\n",
       "      <td>14.65</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 111 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    ticker       date    open  ...        VIX  median VIX 30 days  JUMP\n",
       "811   EQIX 2015-01-02  228.51  ...  17.790001               14.65   1.0\n",
       "812   EQIX 2015-01-02  228.51  ...  17.790001               14.65   1.0\n",
       "813   EQIX 2015-01-02  228.51  ...  17.790001               14.65   1.0\n",
       "814   EQIX 2015-01-02  228.51  ...  17.790001               14.65   1.0\n",
       "815   EQIX 2015-01-02  228.51  ...  17.790001               14.65   1.0\n",
       "\n",
       "[5 rows x 111 columns]"
      ]
     },
     "execution_count": 35,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "JUMP_subset.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Ej9QNF8GRJB7"
   },
   "outputs": [],
   "source": [
    "#JUMP_subset.to_csv(\"/content/drive/MyDrive/data/jump_without_returns.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "B_swaNmOZ7jN"
   },
   "outputs": [],
   "source": [
    "#JUMP_subset=pd.read_csv(\"/content/drive/MyDrive/data/jump_without_returns.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gqKQHkZLVRrD"
   },
   "source": [
    "## Predicting JUMP's Response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9sOTBtXuWywk"
   },
   "source": [
    "### Forward 1,2,5,10 and 20 day returns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "UGMRAvIvVUVb"
   },
   "outputs": [],
   "source": [
    "def forward_returns(df,window):\n",
    "  ch=\"forward returns \"+str(window)\n",
    "  df_ret=np.log(df.shift(-1*window)[\"close\"]/df[\"close\"])\n",
    "  df_ret=pd.DataFrame(df_ret)\n",
    "  df_ret.columns=[ch]\n",
    "  return df_ret"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "nSEvpGPXV-jL",
    "outputId": "9993c474-bc65-4f67-e7ee-bbee08a2292f"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n",
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  \n",
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:3: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n",
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:4: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  after removing the cwd from sys.path.\n",
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  \"\"\"\n"
     ]
    }
   ],
   "source": [
    "JUMP_subset[\"forward returns 1\"]=np.nan\n",
    "JUMP_subset[\"forward returns 2\"]=np.nan\n",
    "JUMP_subset[\"forward returns 5\"]=np.nan\n",
    "JUMP_subset[\"forward returns 10\"]=np.nan\n",
    "JUMP_subset[\"forward returns 20\"]=np.nan\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "wP_j2TXlmGR_",
    "outputId": "251ed0fc-59a6-4ec7-8e95-fe161750cc5e"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/pandas/core/frame.py:6397: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  self[col] = expressions.where(mask, this, that)\n"
     ]
    }
   ],
   "source": [
    "for ticker in JUMP_subset[\"ticker\"].unique():\n",
    "  df1=forward_returns(JUMP_subset[JUMP_subset[\"ticker\"]==ticker],1)\n",
    "\n",
    "  JUMP_subset.update(df1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "3uEVSwzqmIr_",
    "outputId": "620e2224-bb43-43f7-dfd6-ad7766e1a489"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/pandas/core/frame.py:6397: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  self[col] = expressions.where(mask, this, that)\n"
     ]
    }
   ],
   "source": [
    "for ticker in JUMP_subset[\"ticker\"].unique():\n",
    "  df2=forward_returns(JUMP_subset[JUMP_subset[\"ticker\"]==ticker],2)\n",
    "\n",
    "  JUMP_subset.update(df2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ZaFdluUCmRDh",
    "outputId": "9924f793-82c9-451e-891e-83f91ee390a9"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/pandas/core/frame.py:6397: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  self[col] = expressions.where(mask, this, that)\n"
     ]
    }
   ],
   "source": [
    "for ticker in JUMP_subset[\"ticker\"].unique():\n",
    "  df5=forward_returns(JUMP_subset[JUMP_subset[\"ticker\"]==ticker],5)\n",
    "\n",
    "  JUMP_subset.update(df5)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "JSPCrnfBmTZZ",
    "outputId": "97d23268-789c-4d93-c951-ff99105f951f"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/pandas/core/frame.py:6397: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  self[col] = expressions.where(mask, this, that)\n"
     ]
    }
   ],
   "source": [
    "for ticker in JUMP_subset[\"ticker\"].unique():\n",
    "  df10=forward_returns(JUMP_subset[JUMP_subset[\"ticker\"]==ticker],10)\n",
    "\n",
    "  JUMP_subset.update(df10)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "LmoY5CYTmVgW",
    "outputId": "ae7cdbf8-6c03-4488-bbe6-63032bf313a4"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/pandas/core/frame.py:6397: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  self[col] = expressions.where(mask, this, that)\n"
     ]
    }
   ],
   "source": [
    "for ticker in JUMP_subset[\"ticker\"].unique():\n",
    "  df20=forward_returns(JUMP_subset[JUMP_subset[\"ticker\"]==ticker],20)\n",
    "\n",
    "  JUMP_subset.update(df20)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "YxepUrOluWeF",
    "outputId": "750d943c-7f57-4214-de00-f4017a499cd2"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    }
   ],
   "source": [
    "JUMP_subset.dropna(inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "t-gm-GPZXMuk"
   },
   "source": [
    "### Categorization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "PNLKb4zmWxrV",
    "outputId": "cb5f4a8b-5419-4fb1-d73c-336a42d6a210"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n",
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  \n",
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:3: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n",
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:4: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  after removing the cwd from sys.path.\n",
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  \"\"\"\n"
     ]
    }
   ],
   "source": [
    "JUMP_subset[\"forward returns 1\"]=np.sign(JUMP_subset[\"forward returns 1\"])\n",
    "JUMP_subset[\"forward returns 2\"]=np.sign(JUMP_subset[\"forward returns 2\"])\n",
    "JUMP_subset[\"forward returns 5\"]=np.sign(JUMP_subset[\"forward returns 5\"])\n",
    "JUMP_subset[\"forward returns 10\"]=np.sign(JUMP_subset[\"forward returns 10\"])\n",
    "JUMP_subset[\"forward returns 20\"]=np.sign(JUMP_subset[\"forward returns 20\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 600
    },
    "id": "N6pPKVppkTue",
    "outputId": "9640664e-2563-4a56-f444-2b37c322210b"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ticker</th>\n",
       "      <th>date</th>\n",
       "      <th>open</th>\n",
       "      <th>high</th>\n",
       "      <th>low</th>\n",
       "      <th>close</th>\n",
       "      <th>volume</th>\n",
       "      <th>dividends</th>\n",
       "      <th>closeunadj</th>\n",
       "      <th>lastupdated</th>\n",
       "      <th>sector</th>\n",
       "      <th>scalemarketcap</th>\n",
       "      <th>exchange</th>\n",
       "      <th>year</th>\n",
       "      <th>Date</th>\n",
       "      <th>px_rank</th>\n",
       "      <th>next_px_rank</th>\n",
       "      <th>next_price_close</th>\n",
       "      <th>next_price_close_2</th>\n",
       "      <th>next_price_close_5</th>\n",
       "      <th>next_price_close_10</th>\n",
       "      <th>next_price_close_20</th>\n",
       "      <th>prior_price_close</th>\n",
       "      <th>prior_price_close_20</th>\n",
       "      <th>prior_price_close_60</th>\n",
       "      <th>prior_price_close_120</th>\n",
       "      <th>prior_price_close_250</th>\n",
       "      <th>range</th>\n",
       "      <th>px_rank_chg</th>\n",
       "      <th>90PctileRank_5</th>\n",
       "      <th>10PctileRank_5</th>\n",
       "      <th>MnRank_5</th>\n",
       "      <th>RangeRank_5</th>\n",
       "      <th>Rank_Zscore_5</th>\n",
       "      <th>90PctileRank_10</th>\n",
       "      <th>10PctileRank_10</th>\n",
       "      <th>MnRank_10</th>\n",
       "      <th>RangeRank_10</th>\n",
       "      <th>Rank_Zscore_10</th>\n",
       "      <th>90PctileRank_20</th>\n",
       "      <th>...</th>\n",
       "      <th>sector_prior_ret_120</th>\n",
       "      <th>sector_prior_ret_250</th>\n",
       "      <th>sector_next_ret</th>\n",
       "      <th>sector_next_ret_2</th>\n",
       "      <th>sector_next_ret_5</th>\n",
       "      <th>sector_next_ret_10</th>\n",
       "      <th>sector_next_ret_20</th>\n",
       "      <th>market_prior_ret</th>\n",
       "      <th>market_prior_ret_20</th>\n",
       "      <th>market_prior_ret_60</th>\n",
       "      <th>market_prior_ret_120</th>\n",
       "      <th>market_prior_ret_250</th>\n",
       "      <th>market_next_ret</th>\n",
       "      <th>market_next_ret_2</th>\n",
       "      <th>market_next_ret_5</th>\n",
       "      <th>market_next_ret_10</th>\n",
       "      <th>market_next_ret_20</th>\n",
       "      <th>Beta</th>\n",
       "      <th>SecBeta</th>\n",
       "      <th>market_relative_prior_ret</th>\n",
       "      <th>market_relative_prior_ret_20</th>\n",
       "      <th>market_relative_prior_ret_60</th>\n",
       "      <th>market_relative_prior_ret_120</th>\n",
       "      <th>market_relative_prior_ret_250</th>\n",
       "      <th>sector_relative_prior_ret</th>\n",
       "      <th>sector_relative_prior_ret_20</th>\n",
       "      <th>sector_relative_prior_ret_60</th>\n",
       "      <th>sector_relative_prior_ret_120</th>\n",
       "      <th>sector_relative_prior_ret_250</th>\n",
       "      <th>Close2Open</th>\n",
       "      <th>PV30</th>\n",
       "      <th>median PV30</th>\n",
       "      <th>VIX</th>\n",
       "      <th>median VIX 30 days</th>\n",
       "      <th>JUMP</th>\n",
       "      <th>forward returns 1</th>\n",
       "      <th>forward returns 2</th>\n",
       "      <th>forward returns 5</th>\n",
       "      <th>forward returns 10</th>\n",
       "      <th>forward returns 20</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>811</th>\n",
       "      <td>EQIX</td>\n",
       "      <td>2015-01-02</td>\n",
       "      <td>228.51</td>\n",
       "      <td>229.560</td>\n",
       "      <td>224.280</td>\n",
       "      <td>226.65</td>\n",
       "      <td>383808.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>226.65</td>\n",
       "      <td>2018-06-13</td>\n",
       "      <td>Real Estate</td>\n",
       "      <td>5 - Large</td>\n",
       "      <td>NASDAQ</td>\n",
       "      <td>2015</td>\n",
       "      <td>20150102.0</td>\n",
       "      <td>0.026826</td>\n",
       "      <td>0.026510</td>\n",
       "      <td>224.32</td>\n",
       "      <td>219.43</td>\n",
       "      <td>216.89</td>\n",
       "      <td>220.24</td>\n",
       "      <td>218.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.023296</td>\n",
       "      <td>-0.000316</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.008173</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>17.790001</td>\n",
       "      <td>14.650</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>812</th>\n",
       "      <td>EQIX</td>\n",
       "      <td>2015-01-02</td>\n",
       "      <td>228.51</td>\n",
       "      <td>229.560</td>\n",
       "      <td>224.280</td>\n",
       "      <td>226.65</td>\n",
       "      <td>383808.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>226.65</td>\n",
       "      <td>2018-06-13</td>\n",
       "      <td>Real Estate</td>\n",
       "      <td>5 - Large</td>\n",
       "      <td>NASDAQ</td>\n",
       "      <td>2015</td>\n",
       "      <td>20150102.0</td>\n",
       "      <td>0.026826</td>\n",
       "      <td>0.026510</td>\n",
       "      <td>224.32</td>\n",
       "      <td>219.43</td>\n",
       "      <td>216.89</td>\n",
       "      <td>220.24</td>\n",
       "      <td>218.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.023296</td>\n",
       "      <td>-0.000316</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.008173</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>17.790001</td>\n",
       "      <td>14.650</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>813</th>\n",
       "      <td>EQIX</td>\n",
       "      <td>2015-01-02</td>\n",
       "      <td>228.51</td>\n",
       "      <td>229.560</td>\n",
       "      <td>224.280</td>\n",
       "      <td>226.65</td>\n",
       "      <td>383808.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>226.65</td>\n",
       "      <td>2018-06-13</td>\n",
       "      <td>Real Estate</td>\n",
       "      <td>5 - Large</td>\n",
       "      <td>NASDAQ</td>\n",
       "      <td>2015</td>\n",
       "      <td>20150102.0</td>\n",
       "      <td>0.026826</td>\n",
       "      <td>0.026510</td>\n",
       "      <td>224.32</td>\n",
       "      <td>219.43</td>\n",
       "      <td>216.89</td>\n",
       "      <td>220.24</td>\n",
       "      <td>218.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.023296</td>\n",
       "      <td>-0.000316</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.008173</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>17.790001</td>\n",
       "      <td>14.650</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>814</th>\n",
       "      <td>EQIX</td>\n",
       "      <td>2015-01-02</td>\n",
       "      <td>228.51</td>\n",
       "      <td>229.560</td>\n",
       "      <td>224.280</td>\n",
       "      <td>226.65</td>\n",
       "      <td>383808.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>226.65</td>\n",
       "      <td>2018-06-13</td>\n",
       "      <td>Real Estate</td>\n",
       "      <td>5 - Large</td>\n",
       "      <td>NASDAQ</td>\n",
       "      <td>2015</td>\n",
       "      <td>20150102.0</td>\n",
       "      <td>0.026826</td>\n",
       "      <td>0.026510</td>\n",
       "      <td>224.32</td>\n",
       "      <td>219.43</td>\n",
       "      <td>216.89</td>\n",
       "      <td>220.24</td>\n",
       "      <td>218.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.023296</td>\n",
       "      <td>-0.000316</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.008173</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>17.790001</td>\n",
       "      <td>14.650</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>815</th>\n",
       "      <td>EQIX</td>\n",
       "      <td>2015-01-02</td>\n",
       "      <td>228.51</td>\n",
       "      <td>229.560</td>\n",
       "      <td>224.280</td>\n",
       "      <td>226.65</td>\n",
       "      <td>383808.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>226.65</td>\n",
       "      <td>2018-06-13</td>\n",
       "      <td>Real Estate</td>\n",
       "      <td>5 - Large</td>\n",
       "      <td>NASDAQ</td>\n",
       "      <td>2015</td>\n",
       "      <td>20150102.0</td>\n",
       "      <td>0.026826</td>\n",
       "      <td>0.026510</td>\n",
       "      <td>224.32</td>\n",
       "      <td>219.43</td>\n",
       "      <td>216.89</td>\n",
       "      <td>220.24</td>\n",
       "      <td>218.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.023296</td>\n",
       "      <td>-0.000316</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.008173</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>17.790001</td>\n",
       "      <td>14.650</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2119795</th>\n",
       "      <td>RDS.A</td>\n",
       "      <td>2019-10-11</td>\n",
       "      <td>57.80</td>\n",
       "      <td>58.205</td>\n",
       "      <td>57.770</td>\n",
       "      <td>57.85</td>\n",
       "      <td>2365519.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>57.85</td>\n",
       "      <td>2020-05-01</td>\n",
       "      <td>Energy</td>\n",
       "      <td>6 - Mega</td>\n",
       "      <td>NYSE</td>\n",
       "      <td>2019</td>\n",
       "      <td>20191011.0</td>\n",
       "      <td>0.586207</td>\n",
       "      <td>0.588816</td>\n",
       "      <td>57.82</td>\n",
       "      <td>57.54</td>\n",
       "      <td>57.72</td>\n",
       "      <td>59.68</td>\n",
       "      <td>60.25</td>\n",
       "      <td>57.26</td>\n",
       "      <td>56.87</td>\n",
       "      <td>63.13</td>\n",
       "      <td>65.00</td>\n",
       "      <td>65.26</td>\n",
       "      <td>0.007519</td>\n",
       "      <td>0.002609</td>\n",
       "      <td>0.584511</td>\n",
       "      <td>0.580434</td>\n",
       "      <td>0.582472</td>\n",
       "      <td>0.004077</td>\n",
       "      <td>0.915914</td>\n",
       "      <td>0.586264</td>\n",
       "      <td>0.580269</td>\n",
       "      <td>0.583266</td>\n",
       "      <td>0.005995</td>\n",
       "      <td>0.490493</td>\n",
       "      <td>0.591292</td>\n",
       "      <td>...</td>\n",
       "      <td>-1500.983713</td>\n",
       "      <td>-1966.011695</td>\n",
       "      <td>-5.183723</td>\n",
       "      <td>36.209562</td>\n",
       "      <td>-149.672408</td>\n",
       "      <td>277.676756</td>\n",
       "      <td>472.342478</td>\n",
       "      <td>103.134715</td>\n",
       "      <td>-114.971045</td>\n",
       "      <td>-39.627103</td>\n",
       "      <td>210.011546</td>\n",
       "      <td>907.620814</td>\n",
       "      <td>-11.142959</td>\n",
       "      <td>87.373575</td>\n",
       "      <td>56.879273</td>\n",
       "      <td>177.968303</td>\n",
       "      <td>418.423550</td>\n",
       "      <td>0.698609</td>\n",
       "      <td>0.678014</td>\n",
       "      <td>30.460689</td>\n",
       "      <td>251.174738</td>\n",
       "      <td>-845.742535</td>\n",
       "      <td>-1312.054138</td>\n",
       "      <td>-1839.330524</td>\n",
       "      <td>11.723239</td>\n",
       "      <td>472.812393</td>\n",
       "      <td>-418.902127</td>\n",
       "      <td>-147.650060</td>\n",
       "      <td>127.725250</td>\n",
       "      <td>0.009386</td>\n",
       "      <td>0.003576</td>\n",
       "      <td>0.004273</td>\n",
       "      <td>15.580000</td>\n",
       "      <td>16.155</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2121524</th>\n",
       "      <td>UN</td>\n",
       "      <td>2019-10-16</td>\n",
       "      <td>59.50</td>\n",
       "      <td>59.785</td>\n",
       "      <td>59.300</td>\n",
       "      <td>59.72</td>\n",
       "      <td>1422886.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>59.72</td>\n",
       "      <td>2020-05-01</td>\n",
       "      <td>Consumer Defensive</td>\n",
       "      <td>5 - Large</td>\n",
       "      <td>NYSE</td>\n",
       "      <td>2019</td>\n",
       "      <td>20191016.0</td>\n",
       "      <td>0.577741</td>\n",
       "      <td>0.569558</td>\n",
       "      <td>60.46</td>\n",
       "      <td>60.50</td>\n",
       "      <td>59.04</td>\n",
       "      <td>59.05</td>\n",
       "      <td>59.68</td>\n",
       "      <td>58.46</td>\n",
       "      <td>60.33</td>\n",
       "      <td>60.45</td>\n",
       "      <td>59.26</td>\n",
       "      <td>53.71</td>\n",
       "      <td>0.008121</td>\n",
       "      <td>-0.008183</td>\n",
       "      <td>0.583553</td>\n",
       "      <td>0.571523</td>\n",
       "      <td>0.577538</td>\n",
       "      <td>0.012029</td>\n",
       "      <td>0.016911</td>\n",
       "      <td>0.581086</td>\n",
       "      <td>0.570467</td>\n",
       "      <td>0.575776</td>\n",
       "      <td>0.010618</td>\n",
       "      <td>0.185055</td>\n",
       "      <td>0.580645</td>\n",
       "      <td>...</td>\n",
       "      <td>707.863830</td>\n",
       "      <td>1476.305929</td>\n",
       "      <td>46.296730</td>\n",
       "      <td>66.072022</td>\n",
       "      <td>148.054445</td>\n",
       "      <td>149.686489</td>\n",
       "      <td>162.739656</td>\n",
       "      <td>-16.073650</td>\n",
       "      <td>-44.002217</td>\n",
       "      <td>-8.403062</td>\n",
       "      <td>263.230781</td>\n",
       "      <td>817.163042</td>\n",
       "      <td>29.447437</td>\n",
       "      <td>-14.420653</td>\n",
       "      <td>49.475679</td>\n",
       "      <td>190.533503</td>\n",
       "      <td>352.299857</td>\n",
       "      <td>0.365802</td>\n",
       "      <td>0.683541</td>\n",
       "      <td>219.121905</td>\n",
       "      <td>-85.529090</td>\n",
       "      <td>-118.422187</td>\n",
       "      <td>-18.966104</td>\n",
       "      <td>761.757621</td>\n",
       "      <td>206.442274</td>\n",
       "      <td>-81.045124</td>\n",
       "      <td>-198.163072</td>\n",
       "      <td>-406.529954</td>\n",
       "      <td>51.561414</td>\n",
       "      <td>0.017634</td>\n",
       "      <td>0.002960</td>\n",
       "      <td>0.003011</td>\n",
       "      <td>13.680000</td>\n",
       "      <td>15.450</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2122187</th>\n",
       "      <td>RDS.A</td>\n",
       "      <td>2019-10-17</td>\n",
       "      <td>58.19</td>\n",
       "      <td>58.325</td>\n",
       "      <td>57.907</td>\n",
       "      <td>58.16</td>\n",
       "      <td>2543197.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>58.16</td>\n",
       "      <td>2020-05-01</td>\n",
       "      <td>Energy</td>\n",
       "      <td>6 - Mega</td>\n",
       "      <td>NYSE</td>\n",
       "      <td>2019</td>\n",
       "      <td>20191017.0</td>\n",
       "      <td>0.585925</td>\n",
       "      <td>0.596026</td>\n",
       "      <td>57.72</td>\n",
       "      <td>58.28</td>\n",
       "      <td>59.57</td>\n",
       "      <td>57.97</td>\n",
       "      <td>59.13</td>\n",
       "      <td>57.39</td>\n",
       "      <td>58.31</td>\n",
       "      <td>63.17</td>\n",
       "      <td>63.13</td>\n",
       "      <td>64.70</td>\n",
       "      <td>0.007187</td>\n",
       "      <td>0.010102</td>\n",
       "      <td>0.591991</td>\n",
       "      <td>0.586038</td>\n",
       "      <td>0.589014</td>\n",
       "      <td>0.005954</td>\n",
       "      <td>-0.518959</td>\n",
       "      <td>0.589345</td>\n",
       "      <td>0.580566</td>\n",
       "      <td>0.584955</td>\n",
       "      <td>0.008779</td>\n",
       "      <td>0.110415</td>\n",
       "      <td>0.589040</td>\n",
       "      <td>...</td>\n",
       "      <td>-1250.614641</td>\n",
       "      <td>-1948.869235</td>\n",
       "      <td>-52.465931</td>\n",
       "      <td>124.806976</td>\n",
       "      <td>293.911833</td>\n",
       "      <td>109.291384</td>\n",
       "      <td>391.670649</td>\n",
       "      <td>29.447437</td>\n",
       "      <td>-13.890498</td>\n",
       "      <td>-25.840839</td>\n",
       "      <td>277.013955</td>\n",
       "      <td>992.074153</td>\n",
       "      <td>-43.868090</td>\n",
       "      <td>23.694564</td>\n",
       "      <td>36.355074</td>\n",
       "      <td>134.416576</td>\n",
       "      <td>337.399367</td>\n",
       "      <td>0.703705</td>\n",
       "      <td>0.675079</td>\n",
       "      <td>112.555315</td>\n",
       "      <td>-15.982916</td>\n",
       "      <td>-808.132403</td>\n",
       "      <td>-1014.918567</td>\n",
       "      <td>-1763.760754</td>\n",
       "      <td>128.566703</td>\n",
       "      <td>418.533839</td>\n",
       "      <td>-198.803172</td>\n",
       "      <td>24.280495</td>\n",
       "      <td>250.006085</td>\n",
       "      <td>0.013843</td>\n",
       "      <td>0.003486</td>\n",
       "      <td>0.004207</td>\n",
       "      <td>13.790000</td>\n",
       "      <td>15.295</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2122301</th>\n",
       "      <td>UN</td>\n",
       "      <td>2019-10-17</td>\n",
       "      <td>60.63</td>\n",
       "      <td>60.630</td>\n",
       "      <td>60.340</td>\n",
       "      <td>60.46</td>\n",
       "      <td>3920417.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>60.46</td>\n",
       "      <td>2020-05-01</td>\n",
       "      <td>Consumer Defensive</td>\n",
       "      <td>5 - Large</td>\n",
       "      <td>NYSE</td>\n",
       "      <td>2019</td>\n",
       "      <td>20191017.0</td>\n",
       "      <td>0.569558</td>\n",
       "      <td>0.577815</td>\n",
       "      <td>60.50</td>\n",
       "      <td>59.50</td>\n",
       "      <td>59.03</td>\n",
       "      <td>59.24</td>\n",
       "      <td>59.12</td>\n",
       "      <td>59.72</td>\n",
       "      <td>60.34</td>\n",
       "      <td>59.89</td>\n",
       "      <td>59.31</td>\n",
       "      <td>52.70</td>\n",
       "      <td>0.004797</td>\n",
       "      <td>0.008256</td>\n",
       "      <td>0.583553</td>\n",
       "      <td>0.570963</td>\n",
       "      <td>0.577258</td>\n",
       "      <td>0.012590</td>\n",
       "      <td>-0.611601</td>\n",
       "      <td>0.581086</td>\n",
       "      <td>0.570398</td>\n",
       "      <td>0.575742</td>\n",
       "      <td>0.010687</td>\n",
       "      <td>-0.578631</td>\n",
       "      <td>0.580645</td>\n",
       "      <td>...</td>\n",
       "      <td>741.882318</td>\n",
       "      <td>1535.685478</td>\n",
       "      <td>19.775291</td>\n",
       "      <td>52.648957</td>\n",
       "      <td>135.989506</td>\n",
       "      <td>88.685352</td>\n",
       "      <td>113.180910</td>\n",
       "      <td>29.447437</td>\n",
       "      <td>-13.890498</td>\n",
       "      <td>-25.840839</td>\n",
       "      <td>277.013955</td>\n",
       "      <td>992.074153</td>\n",
       "      <td>-43.868090</td>\n",
       "      <td>23.694564</td>\n",
       "      <td>36.355074</td>\n",
       "      <td>134.416576</td>\n",
       "      <td>337.399367</td>\n",
       "      <td>0.358986</td>\n",
       "      <td>0.683648</td>\n",
       "      <td>112.578934</td>\n",
       "      <td>24.854055</td>\n",
       "      <td>104.000941</td>\n",
       "      <td>92.596372</td>\n",
       "      <td>1017.524140</td>\n",
       "      <td>91.499488</td>\n",
       "      <td>7.677202</td>\n",
       "      <td>-59.318922</td>\n",
       "      <td>-315.145899</td>\n",
       "      <td>323.796680</td>\n",
       "      <td>0.015123</td>\n",
       "      <td>0.003054</td>\n",
       "      <td>0.003011</td>\n",
       "      <td>13.790000</td>\n",
       "      <td>15.295</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2123808</th>\n",
       "      <td>RDS.A</td>\n",
       "      <td>2019-10-22</td>\n",
       "      <td>58.57</td>\n",
       "      <td>59.340</td>\n",
       "      <td>58.500</td>\n",
       "      <td>58.94</td>\n",
       "      <td>3305023.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>58.94</td>\n",
       "      <td>2020-05-01</td>\n",
       "      <td>Energy</td>\n",
       "      <td>6 - Mega</td>\n",
       "      <td>NYSE</td>\n",
       "      <td>2019</td>\n",
       "      <td>20191022.0</td>\n",
       "      <td>0.589198</td>\n",
       "      <td>0.584288</td>\n",
       "      <td>59.61</td>\n",
       "      <td>59.57</td>\n",
       "      <td>59.93</td>\n",
       "      <td>60.16</td>\n",
       "      <td>58.73</td>\n",
       "      <td>58.28</td>\n",
       "      <td>57.89</td>\n",
       "      <td>63.46</td>\n",
       "      <td>63.70</td>\n",
       "      <td>63.11</td>\n",
       "      <td>0.014252</td>\n",
       "      <td>-0.004910</td>\n",
       "      <td>0.595259</td>\n",
       "      <td>0.587234</td>\n",
       "      <td>0.591247</td>\n",
       "      <td>0.008025</td>\n",
       "      <td>-0.255267</td>\n",
       "      <td>0.594300</td>\n",
       "      <td>0.581803</td>\n",
       "      <td>0.588052</td>\n",
       "      <td>0.012497</td>\n",
       "      <td>0.091742</td>\n",
       "      <td>0.592635</td>\n",
       "      <td>...</td>\n",
       "      <td>-612.239129</td>\n",
       "      <td>-1230.557096</td>\n",
       "      <td>82.988777</td>\n",
       "      <td>40.733607</td>\n",
       "      <td>104.890065</td>\n",
       "      <td>414.748822</td>\n",
       "      <td>-56.279862</td>\n",
       "      <td>-32.720793</td>\n",
       "      <td>105.566873</td>\n",
       "      <td>-35.529521</td>\n",
       "      <td>359.945722</td>\n",
       "      <td>1084.500591</td>\n",
       "      <td>29.054471</td>\n",
       "      <td>45.381303</td>\n",
       "      <td>139.486511</td>\n",
       "      <td>264.684730</td>\n",
       "      <td>423.018022</td>\n",
       "      <td>0.698236</td>\n",
       "      <td>0.672541</td>\n",
       "      <td>135.456788</td>\n",
       "      <td>106.042620</td>\n",
       "      <td>-714.090106</td>\n",
       "      <td>-1027.972846</td>\n",
       "      <td>-1440.829730</td>\n",
       "      <td>26.275013</td>\n",
       "      <td>352.513392</td>\n",
       "      <td>-422.071687</td>\n",
       "      <td>-364.889847</td>\n",
       "      <td>144.007699</td>\n",
       "      <td>0.004964</td>\n",
       "      <td>0.003341</td>\n",
       "      <td>0.004166</td>\n",
       "      <td>14.460000</td>\n",
       "      <td>15.115</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>679368 rows × 116 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        ticker       date  ...  forward returns 10  forward returns 20\n",
       "811       EQIX 2015-01-02  ...                 0.0                 0.0\n",
       "812       EQIX 2015-01-02  ...                 0.0                 0.0\n",
       "813       EQIX 2015-01-02  ...                 0.0                 0.0\n",
       "814       EQIX 2015-01-02  ...                 0.0                 0.0\n",
       "815       EQIX 2015-01-02  ...                 0.0                 0.0\n",
       "...        ...        ...  ...                 ...                 ...\n",
       "2119795  RDS.A 2019-10-11  ...                 1.0                 1.0\n",
       "2121524     UN 2019-10-16  ...                -1.0                -1.0\n",
       "2122187  RDS.A 2019-10-17  ...                 1.0                 1.0\n",
       "2122301     UN 2019-10-17  ...                -1.0                -1.0\n",
       "2123808  RDS.A 2019-10-22  ...                -1.0                 1.0\n",
       "\n",
       "[679368 rows x 116 columns]"
      ]
     },
     "execution_count": 47,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "JUMP_subset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Qu65mr7Ky5It"
   },
   "source": [
    "Our data finally!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "exP1ymcOYGPx"
   },
   "outputs": [],
   "source": [
    "#JUMP_subset.to_csv(\"/content/drive/MyDrive/data/jump_final.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TGErLAQiXQD-"
   },
   "source": [
    "## Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "p4xOzZy4aOvy"
   },
   "outputs": [],
   "source": [
    "JUMP_subset=pd.read_csv(\"/content/drive/MyDrive/jump_final.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "D-VXXUxYzzuX"
   },
   "source": [
    "#### Final touches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "wSS5jFwU1cCb"
   },
   "outputs": [],
   "source": [
    "JUMP_subset=JUMP_subset[JUMP_subset[\"forward returns 1\"]!=0]\n",
    "JUMP_subset=JUMP_subset[JUMP_subset[\"forward returns 2\"]!=0]\n",
    "JUMP_subset=JUMP_subset[JUMP_subset[\"forward returns 5\"]!=0]\n",
    "JUMP_subset=JUMP_subset[JUMP_subset[\"forward returns 10\"]!=0]\n",
    "JUMP_subset=JUMP_subset[JUMP_subset[\"forward returns 20\"]!=0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kAPf_fIjplgH"
   },
   "outputs": [],
   "source": [
    "train_val_threshold=664978"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qJq3OKWqu3tF"
   },
   "outputs": [],
   "source": [
    "to_remove=[\"ticker\",\"date\",\"sector\",\"lastupdated\",\"scalemarketcap\",\"exchange\",\"year\",\"Date\",\"JUMP\",\"etf\",\"Unnamed: 0\"]\n",
    "JUMP_subset=JUMP_subset.drop(to_remove,axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "lTsv-n3ivKdp"
   },
   "outputs": [],
   "source": [
    "y=JUMP_subset.iloc[:,-5:]\n",
    "X=JUMP_subset.iloc[:,:-5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vAPl57aHbJGT"
   },
   "outputs": [],
   "source": [
    "X_train=X.loc[:train_val_threshold]\n",
    "X_test=X.loc[train_val_threshold:]\n",
    "y_train=y.loc[:train_val_threshold]\n",
    "y_test=y.loc[train_val_threshold:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xe1Xg5tCY9kX"
   },
   "source": [
    "#### Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CiBdpVgYZHZh"
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "aXzRkwShazfX",
    "outputId": "276867fd-e4bb-407a-fbe1-f13b0b3b5e5f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report, forward_returns  1\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.00      0.00      0.00         0\n",
      "         1.0       1.00      0.57      0.73     14317\n",
      "\n",
      "    accuracy                           0.57     14317\n",
      "   macro avg       0.50      0.29      0.36     14317\n",
      "weighted avg       1.00      0.57      0.73     14317\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report, forward_returns  2\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.00      0.00      0.00         0\n",
      "         1.0       1.00      0.52      0.69     14317\n",
      "\n",
      "    accuracy                           0.52     14317\n",
      "   macro avg       0.50      0.26      0.34     14317\n",
      "weighted avg       1.00      0.52      0.69     14317\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report, forward_returns  5\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.00      0.00      0.00         1\n",
      "         1.0       1.00      0.49      0.66     14316\n",
      "\n",
      "    accuracy                           0.49     14317\n",
      "   macro avg       0.50      0.25      0.33     14317\n",
      "weighted avg       1.00      0.49      0.66     14317\n",
      "\n",
      "\n",
      "\n",
      "Classification report, forward_returns  10\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.00      1.00      0.00         1\n",
      "         1.0       1.00      0.49      0.66     14316\n",
      "\n",
      "    accuracy                           0.49     14317\n",
      "   macro avg       0.50      0.75      0.33     14317\n",
      "weighted avg       1.00      0.49      0.66     14317\n",
      "\n",
      "\n",
      "\n",
      "Classification report, forward_returns  20\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.00      1.00      0.00         2\n",
      "         1.0       1.00      0.53      0.69     14315\n",
      "\n",
      "    accuracy                           0.53     14317\n",
      "   macro avg       0.50      0.76      0.34     14317\n",
      "weighted avg       1.00      0.53      0.69     14317\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "lr=LogisticRegression()\n",
    "\n",
    "for i in range(5):\n",
    "  lr.fit(X_train,y_train.iloc[:,i])\n",
    "  pred=lr.predict(X_test)\n",
    "  print(\"Classification report, forward_returns \",[1,2,5,10,20][i])\n",
    "  print(classification_report(pred,y_test.iloc[:,i]))\n",
    "  print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ap-32SNPY_r_"
   },
   "source": [
    "#### L1-Regularized Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pe3y3y_fZH6M",
    "outputId": "bf157873-3ad7-410d-8ed4-0284d71b067e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report, forward_returns  1\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.00      0.00      0.00         0\n",
      "         1.0       1.00      0.57      0.73     12360\n",
      "\n",
      "    accuracy                           0.57     12360\n",
      "   macro avg       0.50      0.29      0.36     12360\n",
      "weighted avg       1.00      0.57      0.73     12360\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_sag.py:330: ConvergenceWarning: The max_iter was reached which means the coef_ did not converge\n",
      "  \"the coef_ did not converge\", ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_logistic.py:1501: UserWarning: l1_ratio parameter is only used when penalty is 'elasticnet'. Got (penalty=l1)\n",
      "  \"(penalty={})\".format(self.penalty))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report, forward_returns  2\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.00      0.00      0.00         0\n",
      "         1.0       1.00      0.53      0.69     12360\n",
      "\n",
      "    accuracy                           0.53     12360\n",
      "   macro avg       0.50      0.26      0.35     12360\n",
      "weighted avg       1.00      0.53      0.69     12360\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_sag.py:330: ConvergenceWarning: The max_iter was reached which means the coef_ did not converge\n",
      "  \"the coef_ did not converge\", ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_logistic.py:1501: UserWarning: l1_ratio parameter is only used when penalty is 'elasticnet'. Got (penalty=l1)\n",
      "  \"(penalty={})\".format(self.penalty))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report, forward_returns  5\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.00      0.00      0.00         0\n",
      "         1.0       1.00      0.51      0.67     12360\n",
      "\n",
      "    accuracy                           0.51     12360\n",
      "   macro avg       0.50      0.25      0.34     12360\n",
      "weighted avg       1.00      0.51      0.67     12360\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_sag.py:330: ConvergenceWarning: The max_iter was reached which means the coef_ did not converge\n",
      "  \"the coef_ did not converge\", ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_logistic.py:1501: UserWarning: l1_ratio parameter is only used when penalty is 'elasticnet'. Got (penalty=l1)\n",
      "  \"(penalty={})\".format(self.penalty))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report, forward_returns  10\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.00      0.00      0.00         0\n",
      "         1.0       1.00      0.50      0.66     12360\n",
      "\n",
      "    accuracy                           0.50     12360\n",
      "   macro avg       0.50      0.25      0.33     12360\n",
      "weighted avg       1.00      0.50      0.66     12360\n",
      "\n",
      "\n",
      "\n",
      "Classification report, forward_returns  20\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.00      0.00      0.00         0\n",
      "         1.0       1.00      0.54      0.70     12360\n",
      "\n",
      "    accuracy                           0.54     12360\n",
      "   macro avg       0.50      0.27      0.35     12360\n",
      "weighted avg       1.00      0.54      0.70     12360\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_sag.py:330: ConvergenceWarning: The max_iter was reached which means the coef_ did not converge\n",
      "  \"the coef_ did not converge\", ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    }
   ],
   "source": [
    "l1=LogisticRegression(penalty='l1',solver=\"saga\",l1_ratio=0.1)\n",
    "\n",
    "for i in range(5):\n",
    "  l1.fit(X_train,y_train.iloc[:,i])\n",
    "  pred=l1.predict(X_test)\n",
    "  print(\"Classification report, forward_returns \",[1,2,5,10,20][i])\n",
    "  print(classification_report(pred,y_test.iloc[:,i]))\n",
    "  print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TrtYbQiGZA24"
   },
   "source": [
    "#### L2-Regularized Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "29sgaKZsZJVA",
    "outputId": "3f56f50a-da8f-4061-dea4-1b40802cabe5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report, forward_returns  0\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.00      0.00      0.00         0\n",
      "         1.0       1.00      0.60      0.75       237\n",
      "\n",
      "    accuracy                           0.60       237\n",
      "   macro avg       0.50      0.30      0.37       237\n",
      "weighted avg       1.00      0.60      0.75       237\n",
      "\n",
      "\n",
      "\n",
      "Classification report, forward_returns  1\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.00      0.00      0.00         0\n",
      "         1.0       1.00      0.55      0.71       237\n",
      "\n",
      "    accuracy                           0.55       237\n",
      "   macro avg       0.50      0.27      0.35       237\n",
      "weighted avg       1.00      0.55      0.71       237\n",
      "\n",
      "\n",
      "\n",
      "Classification report, forward_returns  2\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.09      0.50      0.16        20\n",
      "         1.0       0.92      0.55      0.69       217\n",
      "\n",
      "    accuracy                           0.54       237\n",
      "   macro avg       0.51      0.52      0.42       237\n",
      "weighted avg       0.85      0.54      0.64       237\n",
      "\n",
      "\n",
      "\n",
      "Classification report, forward_returns  3\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.03      0.60      0.05         5\n",
      "         1.0       0.98      0.51      0.67       232\n",
      "\n",
      "    accuracy                           0.51       237\n",
      "   macro avg       0.50      0.56      0.36       237\n",
      "weighted avg       0.96      0.51      0.66       237\n",
      "\n",
      "\n",
      "\n",
      "Classification report, forward_returns  4\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.20      0.39      0.27        44\n",
      "         1.0       0.82      0.66      0.73       193\n",
      "\n",
      "    accuracy                           0.61       237\n",
      "   macro avg       0.51      0.52      0.50       237\n",
      "weighted avg       0.71      0.61      0.65       237\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    }
   ],
   "source": [
    "l2=LogisticRegression(penalty='l2')\n",
    "\n",
    "for i in range(5):\n",
    "  l2.fit(X_train,y_train.iloc[:,i])\n",
    "  pred=l2.predict(X_test)\n",
    "  print(\"Classification report, forward_returns \",[1,2,5,10,20][i])\n",
    "  print(classification_report(pred,y_test.iloc[:,i]))\n",
    "  print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CDo5QZ8tZA5G"
   },
   "source": [
    "#### Artificial Neural Network (Shallow)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "udcO-BQTZO3d"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from keras.callbacks import ModelCheckpoint\n",
    "from keras.models import Model, load_model, Sequential\n",
    "from keras.layers import Dense, Activation, Dropout, Input,Flatten\n",
    "from keras.layers import Reshape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ZSuz0Gv70Uh2",
    "outputId": "a6ed1427-a187-444a-9915-d53b9c1f1ff8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "flatten_1 (Flatten)          (None, 102)               0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 256)               26368     \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 1)                 257       \n",
      "=================================================================\n",
      "Total params: 26,625\n",
      "Trainable params: 26,625\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = tf.keras.models.Sequential([\n",
    "    tf.keras.layers.Flatten(input_shape=(102,)),\n",
    "    tf.keras.layers.Dense(units=256, activation=tf.nn.relu),\n",
    "    tf.keras.layers.Dense(units=1, activation=tf.nn.sigmoid)\n",
    "])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "y5aXRogk0UkC"
   },
   "outputs": [],
   "source": [
    "model.compile(optimizer='adam',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "history=model.fit(\n",
    "    X_train,\n",
    "    y_train.iloc[:,0],\n",
    "    epochs=50,\n",
    "    validation_data=(X_test,y_test.iloc[:,0])\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xSMI19WoCfxz"
   },
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FqeYNUjLZBG8"
   },
   "source": [
    "#### Pytorch Sequence model + Autoencoder for feature compression: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iQ6wjk-jxqaa"
   },
   "source": [
    "###### Setting for a sequence model & scaling:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "O06kCtNDhKzH"
   },
   "source": [
    "Here we transform the data frame to the required shape for the LSTM netowrks (batch_size,sequence length, number of features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Q5bWmrORxqab"
   },
   "outputs": [],
   "source": [
    "def add_window(data, windows_size, encoder_feat) : \n",
    "    dataset = data.copy() \n",
    "    for feat in encoder_feat : \n",
    "        for time_step in range(1,windows_size) : \n",
    "            dataset[f\"{feat}(t= N - {time_step})\"] = dataset[feat].shift(time_step)\n",
    "    return dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kF5t5CSAxqae"
   },
   "outputs": [],
   "source": [
    "def create_window(data, windows_size, encoder_feat):\n",
    "    is_start = 0\n",
    "    for feat in encoder_feat:\n",
    "        col = [f\"{feat}(t= N - {i})\" for i in range(windows_size - 1, 0, -1)]\n",
    "        x = data[col].values\n",
    "        x = np.expand_dims(x, axis=2)\n",
    "        if is_start != 0:\n",
    "            input_data = np.concatenate([input_data, x], axis=2)\n",
    "        else:\n",
    "            input_data = x\n",
    "            is_start = 1\n",
    "    return input_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cY5Dmy4Gxqaj"
   },
   "outputs": [],
   "source": [
    "window_size=12\n",
    "encoder_feat=X_train.columns\n",
    "train_data = add_window(X_train,window_size,encoder_feat)\n",
    "test_data = add_window(X_test,window_size,encoder_feat)\n",
    "train_data.dropna( inplace = True)\n",
    "test_data.dropna( inplace = True)\n",
    "train_data  = create_window(train_data,window_size,encoder_feat)\n",
    "train_labels = y_train[window_size-1:]\n",
    "test_data  = create_window(test_data,window_size,encoder_feat)\n",
    "test_labels = y_test[window_size-1:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "B-mcs7cUoqbb"
   },
   "source": [
    "Scaling features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WkTqKbeQoumw"
   },
   "source": [
    "A function to scale multi dimensional arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vr9LMwRH3QO3"
   },
   "outputs": [],
   "source": [
    "from sklearn.base import TransformerMixin\n",
    "from sklearn.preprocessing import StandardScaler,RobustScaler,MinMaxScaler\n",
    "class NDStandardScaler(TransformerMixin):\n",
    "    def __init__(self, **kwargs):\n",
    "        self._scaler = MinMaxScaler(feature_range=(0, 1))\n",
    "        self._orig_shape = None\n",
    "\n",
    "    def fit(self, X, **kwargs):\n",
    "        X = np.array(X)\n",
    "        # Save the original shape to reshape the flattened X later\n",
    "        # back to its original shape\n",
    "        if len(X.shape) > 1:\n",
    "            self._orig_shape = X.shape[1:]\n",
    "        X = self._flatten(X)\n",
    "        self._scaler.fit(X, **kwargs)\n",
    "        return self\n",
    "\n",
    "    def transform(self, X, **kwargs):\n",
    "        X = np.array(X)\n",
    "        X = self._flatten(X)\n",
    "        X = self._scaler.transform(X, **kwargs)\n",
    "        X = self._reshape(X)\n",
    "        return X\n",
    "\n",
    "    def _flatten(self, X):\n",
    "        # Reshape X to <= 2 dimensions\n",
    "        if len(X.shape) > 2:\n",
    "            n_dims = np.prod(self._orig_shape)\n",
    "            X = X.reshape(-1, n_dims)\n",
    "        return X\n",
    "\n",
    "    def _reshape(self, X):\n",
    "        # Reshape X back to it's original shape\n",
    "        if len(X.shape) >= 2:\n",
    "            X = X.reshape(-1, *self._orig_shape)\n",
    "        return X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "nNW91Yh7SQuB"
   },
   "outputs": [],
   "source": [
    "scaler = NDStandardScaler()\n",
    "scaler.fit(train_data)\n",
    "train_data=scaler.transform(train_data)\n",
    "test_data=scaler.transform(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c83e-9RpzppM"
   },
   "source": [
    "##### Modelling the Neural Network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "G7KbLhIiksbL"
   },
   "outputs": [],
   "source": [
    "pip install transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "XiyvYygJrr62"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torchvision\n",
    "import torch.nn as nn  # All neural network modules, nn.Linear, nn.Conv2d, BatchNorm, Loss functions\n",
    "import torch.optim as optim  # For all Optimization algorithms, SGD, Adam, etc.\n",
    "import torch.nn.functional as F  # All functions that don't have any parameters\n",
    "from torch.utils.data import DataLoader# Gives easier dataset managment and creates mini batches\n",
    "import torchvision.datasets as datasets  # Has standard datasets we can import in a nice way\n",
    "import torchvision.transforms as transforms  # Transformations we can perform on our dataset\n",
    "from tqdm import tqdm\n",
    "from torch.cuda import amp\n",
    "from torch.autograd import Variable\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "import transformers\n",
    "from transformers import AdamW\n",
    "from transformers import get_linear_schedule_with_warmup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "OqVrdNcr1Yjm"
   },
   "outputs": [],
   "source": [
    "class inst_dataset() : \n",
    "    def __init__(self,trg=None,data=None): \n",
    "        self.trg=trg\n",
    "        self.data=data\n",
    "        if trg is None:\n",
    "          self.feat = data\n",
    "        else:\n",
    "          self.target = trg.values \n",
    "          self.feat = data\n",
    "        \n",
    "    def __len__(self) : \n",
    "        l= self.data.shape[0]\n",
    "        return l \n",
    "    def __getitem__(self,item) : \n",
    "        out = dict()\n",
    "        if self.trg is None:\n",
    "          out['encoder_feat'] = torch.tensor(self.feat[item] ,dtype = torch.float)\n",
    "        else:\n",
    "\n",
    "          out['target'] = torch.tensor(self.target[item],dtype = torch.float)\n",
    "          out['encoder_feat'] = torch.tensor(self.feat[item] ,dtype = torch.float)\n",
    "        return out  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "azSDyjFGjaC3"
   },
   "source": [
    "Due to the Large number of features which can affect the performance of the LSTM network we use an autoencoder to compress the data and try to extract fewer features from the original ones. The Auto Encoder gives the privilege of non linearlity compared to the traditionnal PCA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Vl5M1hHu1YoJ"
   },
   "outputs": [],
   "source": [
    "class TimeDistributed(nn.Module):\n",
    "    def __init__(self, module, batch_first=False):\n",
    "        super(TimeDistributed, self).__init__()\n",
    "        self.module = module\n",
    "        self.batch_first = batch_first\n",
    "\n",
    "    def forward(self, x):\n",
    "        if len(x.size()) <= 2:\n",
    "            return self.module(x)\n",
    "        x_reshape = x.contiguous().view(-1, x.size(-1))\n",
    "        y = self.module(x_reshape)\n",
    "        if self.batch_first:\n",
    "            y = y.contiguous().view(x.size(0), -1, y.size(-1))\n",
    "        else:\n",
    "            y = y.view(-1, x.size(1), y.size(-1))\n",
    "        return y\n",
    "    \n",
    "    \n",
    "# AutoEncoder Class    \n",
    "class LSTM_AE(nn.Module):\n",
    "    def __init__(self,input_size):\n",
    "        super(LSTM_AE, self).__init__()\n",
    "        \n",
    "        self.init_batchnorm = TimeDistributed(nn.BatchNorm1d(input_size, momentum=0.01),batch_first = True)\n",
    "        self.activ = nn.Sequential(\n",
    "            nn.Linear(32,40),\n",
    "            nn.ReLU(inplace=True),\n",
    "        )\n",
    "        \n",
    "        # encoder\n",
    "        self.encoder_LSTM1 = nn.LSTM(\n",
    "            input_size=input_size, hidden_size=25, batch_first=True, bidirectional=True\n",
    "        )\n",
    "        self.encoder_LSTM2 = nn.LSTM(\n",
    "            input_size=50, hidden_size=16, batch_first=True, bidirectional=True\n",
    "        )\n",
    "        \n",
    "        # decoder\n",
    "        self.decoder_LSTM1 = nn.LSTM(\n",
    "            input_size=40, hidden_size=35, batch_first=True, bidirectional=True\n",
    "        )\n",
    "        self.decoder_LSTM2 = nn.LSTM(\n",
    "            input_size=70, hidden_size=input_size, batch_first=True, bidirectional=False\n",
    "        )\n",
    "    def forward(self, x):\n",
    "        # encoding\n",
    "        x = self.init_batchnorm(x)\n",
    "        x_hat, _ = self.encoder_LSTM1(x)\n",
    "        x_mapped, _ = self.encoder_LSTM2(x_hat)\n",
    "        x_mapped = self.activ(x_mapped)\n",
    " \n",
    "        # decoding\n",
    "        x_hat, _ = self.decoder_LSTM1(x_mapped)\n",
    "        x_hat, _ = self.decoder_LSTM2(x_hat)\n",
    "        return x_hat,x_mapped \n",
    "    \n",
    "#Base encoder with three LSTM layers    \n",
    "class StackedLSTMs(nn.Module):\n",
    "    \"\"\"\n",
    "    Computes an encoder based on LSTM layers\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, input_size, add_feature_selection=False):\n",
    "        super(StackedLSTMs, self).__init__()\n",
    "        \n",
    "        self.init_batchnorm = TimeDistributed(nn.BatchNorm1d(input_size, momentum=0.01),batch_first = True)\n",
    "        self.LSTM1 = nn.LSTM(\n",
    "            input_size=input_size, hidden_size=32, batch_first=True, bidirectional=True\n",
    "        )\n",
    "        self.LSTM2 = nn.LSTM(\n",
    "            input_size=64, hidden_size=128, batch_first=True, bidirectional=True\n",
    "        )\n",
    "        self.LSTM3 = nn.LSTM(\n",
    "            input_size=256, hidden_size=32, batch_first=True, bidirectional=True\n",
    "        )\n",
    "        \n",
    "        self.batchnorm = TimeDistributed(nn.BatchNorm1d(64, momentum=0.01),batch_first = True)\n",
    "        self.output_embedding = TimeDistributed(nn.Linear(64 , 32),batch_first = True)\n",
    "        self.Relu = nn.ReLU(inplace =True)\n",
    "    def forward(self, enc):\n",
    "        enc = self.init_batchnorm(enc)\n",
    "        x_hat, _ = self.LSTM1(enc)\n",
    "        x_hat, _ = self.LSTM2(x_hat)\n",
    "        x_hat, _ = self.LSTM3(x_hat)\n",
    "        x_hat = self.batchnorm(x_hat)\n",
    "        x_hat = self.output_embedding(x_hat)\n",
    "        x_hat = self.Relu(x_hat)\n",
    "        return x_hat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0ukQXrlO1Yzj"
   },
   "outputs": [],
   "source": [
    "class FullModel(nn.Module):\n",
    "    def __init__(self, params ):\n",
    "        super(FullModel, self).__init__()\n",
    "        \n",
    "        self.AE= LSTM_AE(len(params[\"encoder_feat\"]))\n",
    "        self.encoder  = StackedLSTMs(40)\n",
    "        self.fc = nn.Sequential(\n",
    "            nn.Linear(32,16),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.Linear(16, 5),\n",
    "            nn.Sigmoid()\n",
    "        )\n",
    "    def forward(self, enc_feat):\n",
    "        rec_feat,enc_feat = self.AE(enc_feat)\n",
    "        x_hat = self.encoder(enc_feat)\n",
    "        x_hat = self.fc(x_hat[:,-1,:])\n",
    "        return rec_feat,x_hat\n",
    "\n",
    "\n",
    "class TestModel():\n",
    "\n",
    "    def __init__(self, params):\n",
    "        self.name = \"Test_Model\"\n",
    "        self.model = FullModel(params)\n",
    "        self.model = self.model.to(params[\"device\"])\n",
    "        self.params = params\n",
    "\n",
    "    def _create_dataset(self,y=None,dataset=None):\n",
    "        return inst_dataset(y,\n",
    "            dataset\n",
    "        )\n",
    "\n",
    "    def _model_trainer(self, model, train_dataset, valid_dataset, params):\n",
    "        run(\n",
    "            model,\n",
    "            train_dataset,\n",
    "            valid_dataset,\n",
    "            params[\"lr\"],\n",
    "            params[\"epochs\"],\n",
    "            params[\"batch_size\"],\n",
    "            params[\"batch_size\"],\n",
    "            params[\"device\"],\n",
    "            params[\"save_path\"],\n",
    "            self.params[\"verbose\"],\n",
    "        )\n",
    "\n",
    "    def train(self, train_dataset, test_dataset,y_train,y_test, params):\n",
    "        train_dataset = self._create_dataset(y_train,train_dataset)\n",
    "        valid_dataset = self._create_dataset(y_test,test_dataset)\n",
    "        if self.params[\"verbose\"]:\n",
    "            print(\"Training Model 1 ...\")\n",
    "        self._model_trainer(self.model, train_dataset, valid_dataset, params)\n",
    "\n",
    "    def eval(self):\n",
    "        self.model.eval()\n",
    "\n",
    "    def __call__(self, x, device):\n",
    "        return self.model(x, device)\n",
    "\n",
    "    def predict(self, dataset):\n",
    "        test_dataset = self._create_dataset(None,dataset)\n",
    "        pred = predict(self.model, test_dataset)\n",
    "        return pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eb-MZIbPywoy"
   },
   "source": [
    "##### Engine Class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Zkvbcn6Frr9t"
   },
   "outputs": [],
   "source": [
    "class AverageMeter:\n",
    "    \"\"\"\n",
    "    Computes and stores the average and current value\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self):\n",
    "        self.reset()\n",
    "\n",
    "    def reset(self):\n",
    "        self.val = 0\n",
    "        self.avg = 0\n",
    "        self.sum = 0\n",
    "        self.count = 0\n",
    "\n",
    "    def update(self, val, n=1):\n",
    "        self.val = val\n",
    "        self.sum += val * n\n",
    "        self.count += n\n",
    "        self.avg = self.sum / self.count\n",
    "\n",
    "\n",
    "def train_fn(\n",
    "    data_loader, model, optimizer, scaler, device, verbose, epoch ):\n",
    "    \"\"\"\n",
    "    computes the model training for one epoch\n",
    "    \"\"\"\n",
    "    \n",
    "    model.train()\n",
    "    tr_loss = 0\n",
    "    counter = 0\n",
    "    if verbose:\n",
    "        losses = AverageMeter()\n",
    "        tk0 = tqdm(enumerate(data_loader), total=len(data_loader))\n",
    "    else:\n",
    "        tk0 = enumerate(data_loader)\n",
    "    for bi, d in tk0:\n",
    "        targets = d[\"target\"].to(device, dtype=torch.float)\n",
    "        enc_feat   = d[\"encoder_feat\"].to(device, dtype=torch.float)\n",
    "        criterion= nn.BCEWithLogitsLoss(reduction = 'mean')\n",
    "        optimizer.zero_grad()\n",
    "        rec_feat,outputs = model(enc_feat)\n",
    "        rec_loss = 0.00005*(torch.nn.L1Loss()(rec_feat,enc_feat))\n",
    "        loss = criterion(outputs,targets) +rec_loss\n",
    "        tr_loss += loss.item()\n",
    "        counter += 1\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        if verbose:\n",
    "            losses.update(loss.item(), targets.size(0))\n",
    "            tk0.set_postfix(loss=losses.avg)\n",
    "    return tr_loss / counter\n",
    "\n",
    "\n",
    "def eval_fn(data_loader, model, device, verbose, epoch):\n",
    "    \"\"\"\n",
    "    computes the model evaluation for one epoch\n",
    "    \"\"\"\n",
    "    model.eval()\n",
    "    fin_loss = 0\n",
    "    counter = 0\n",
    "    if verbose:\n",
    "        losses = AverageMeter()\n",
    "        tk0 = tqdm(enumerate(data_loader), total=len(data_loader))\n",
    "    else:\n",
    "        tk0 = enumerate(data_loader)\n",
    "    with torch.no_grad():\n",
    "        for bi, d in tk0:\n",
    "            targets = d[\"target\"].to(device, dtype=torch.float)\n",
    "            enc_feat   = d[\"encoder_feat\"].to(device, dtype=torch.float)\n",
    "            criterion= nn.BCEWithLogitsLoss(reduction = 'mean')\n",
    "            rec_feat,outputs = model(enc_feat)\n",
    "            rec_loss = 0.00005*(torch.nn.L1Loss()(rec_feat,enc_feat))\n",
    "            loss = criterion(outputs,targets) +rec_loss\n",
    "            if verbose:\n",
    "                losses.update(loss.item(), targets.size(0))\n",
    "                tk0.set_postfix(loss=losses.avg)\n",
    "            fin_loss += loss.item()\n",
    "            counter += 1   \n",
    "        return fin_loss / counter\n",
    "\n",
    "\n",
    "def run(\n",
    "    model,\n",
    "    train_dataset,\n",
    "    valid_dataset,\n",
    "    lr,\n",
    "    EPOCHS,\n",
    "    TRAIN_BATCH_SIZE,\n",
    "    VALID_BATCH_SIZE,\n",
    "    device,\n",
    "    path,\n",
    "    verbose=True,\n",
    "):\n",
    "    \"\"\"\n",
    "    trains a given model for a given number of epochs and paramters\n",
    "    \"\"\"\n",
    "    train_data_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=TRAIN_BATCH_SIZE, shuffle=True,num_workers=4)\n",
    "    \n",
    "    valid_data_loader = DataLoader(dataset=valid_dataset, batch_size=VALID_BATCH_SIZE,num_workers=4, shuffle=False)\n",
    "    \n",
    "    optimizer = AdamW(model.parameters(), lr=lr,weight_decay=5e-2)\n",
    "    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(\n",
    "        optimizer, factor=0.2, patience=3, verbose=verbose\n",
    "    )\n",
    "    \n",
    "    scaler = amp.GradScaler()\n",
    "    train_loss = []\n",
    "    val_loss = []\n",
    "    best = 50000\n",
    "    patience = 0\n",
    "    for epoch in range(EPOCHS):\n",
    "        if verbose:\n",
    "            print(f\"--------- Epoch {epoch} ---------\")\n",
    "        tr_loss = train_fn(\n",
    "            train_data_loader,\n",
    "            model,\n",
    "            optimizer,\n",
    "            scaler,\n",
    "            device,\n",
    "            verbose,\n",
    "            epoch,       \n",
    "        )\n",
    "        train_loss.append(tr_loss)\n",
    "        \n",
    "        if verbose:\n",
    "            print(f\" train_loss  = {tr_loss}\")\n",
    "        val = eval_fn(\n",
    "            valid_data_loader, model, device, verbose, epoch\n",
    "        )\n",
    "        \n",
    "        val_loss.append(val)\n",
    "        scheduler.step(val)\n",
    "            \n",
    "        if verbose:\n",
    "            print(f\" val_loss  = {val}\")\n",
    "        if val < best:\n",
    "            best = val\n",
    "            patience = 0\n",
    "            torch.save(model.state_dict(), path)\n",
    "        else:\n",
    "            patience += 1\n",
    "        if patience > 2:\n",
    "            print(f\"Eraly Stopping on Epoch {epoch}\")\n",
    "            print(f\"Best Loss =  {best}\")\n",
    "            break\n",
    "    model.load_state_dict(torch.load(path), strict=False)\n",
    "    return val_loss, train_loss\n",
    "\n",
    "\n",
    "def predict(model, dataset, device=torch.device(\"cuda\"), is_diff=False):\n",
    "    \"\"\"\n",
    "    computes the prediction a given model and data\n",
    "    \"\"\"\n",
    "    model.eval()\n",
    "    data_loader = torch.utils.data.DataLoader(\n",
    "        dataset, batch_size=64, num_workers=4, shuffle=False\n",
    "    )\n",
    "    losses = AverageMeter()\n",
    "    rmse = AverageMeter()\n",
    "    tk0 = tqdm(enumerate(data_loader), total=len(data_loader))\n",
    "    with torch.no_grad():\n",
    "        for bi, d in tk0:\n",
    "            enc_feat   = d[\"encoder_feat\"].to(device, dtype=torch.float)\n",
    "            _,outputs = model(enc_feat)\n",
    "            if bi == 0:\n",
    "                out = outputs\n",
    "            else:\n",
    "                out = torch.cat([out, outputs], dim=0)\n",
    "    return out.cpu().detach().numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "diey48Y3rsAR",
    "outputId": "127834fb-2a7b-4026-9ade-10ce85093646"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Model 1 ...\n",
      "--------- Epoch 0 ---------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 186/186 [00:06<00:00, 29.84it/s, loss=0.717]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " train_loss  = 0.7169101940047357\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 56/56 [00:00<00:00, 67.78it/s, loss=0.852]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " val_loss  = 0.8518812443528857\n",
      "--------- Epoch 1 ---------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 186/186 [00:05<00:00, 31.41it/s, loss=0.687]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " train_loss  = 0.6873326035596992\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "100%|██████████| 56/56 [00:00<00:00, 63.62it/s, loss=0.727]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " val_loss  = 0.7269473033291953\n",
      "--------- Epoch 2 ---------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 186/186 [00:05<00:00, 31.39it/s, loss=0.687]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " train_loss  = 0.6870953594484637\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "100%|██████████| 56/56 [00:00<00:00, 64.11it/s, loss=0.692]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " val_loss  = 0.6917321628757885\n",
      "--------- Epoch 3 ---------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 186/186 [00:05<00:00, 31.74it/s, loss=0.686]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " train_loss  = 0.6864696036102951\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "100%|██████████| 56/56 [00:00<00:00, 63.17it/s, loss=0.689]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " val_loss  = 0.6894866728356907\n",
      "--------- Epoch 4 ---------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 186/186 [00:05<00:00, 31.74it/s, loss=0.686]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " train_loss  = 0.6860294213858984\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "100%|██████████| 56/56 [00:00<00:00, 68.95it/s, loss=0.694]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " val_loss  = 0.6937541716865131\n",
      "--------- Epoch 5 ---------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "100%|██████████| 186/186 [00:05<00:00, 31.82it/s, loss=0.686]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " train_loss  = 0.6860337542590275\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "100%|██████████| 56/56 [00:00<00:00, 68.22it/s, loss=0.692]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " val_loss  = 0.6917337509138244\n",
      "--------- Epoch 6 ---------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "100%|██████████| 186/186 [00:05<00:00, 31.72it/s, loss=0.686]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " train_loss  = 0.6862333727780209\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "100%|██████████| 56/56 [00:00<00:00, 70.11it/s, loss=0.692]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " val_loss  = 0.6916808528559548\n",
      "Eraly Stopping on Epoch 6\n",
      "Best Loss =  0.6894866728356907\n"
     ]
    }
   ],
   "source": [
    "params = dict()\n",
    "params['encoder_feat'] = encoder_feat\n",
    "params['window_size'] = window_size\n",
    "params['epochs'] = 100\n",
    "params['lr'] = 9e-4\n",
    "params['batch_size'] = 256\n",
    "params['device'] = 'cuda'\n",
    "params['verbose'] = True\n",
    "params['save_path'] = 'test2.pth'\n",
    "\n",
    "model = TestModel(params)  \n",
    "model.train(train_data, test_data,train_labels,test_labels, params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Gj6JqNY4j_fM",
    "outputId": "2af68631-016f-435d-82a8-db845eca808e"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 224/224 [00:01<00:00, 145.34it/s]\n"
     ]
    }
   ],
   "source": [
    "pred=model.predict(test_data)\n",
    "for i in range(len(pred)):\n",
    "  for j in range(5):\n",
    "    if pred[i][j]>0.5:\n",
    "      pred[i][j]=1\n",
    "    else:\n",
    "      pred[i][j]=-1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "duJRwnYOtaOd",
    "outputId": "c3b904ac-8b0e-4d4a-f696-81159ac4a833"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report, forward_returns  1\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.97      0.46      0.63     12908\n",
      "         1.0       0.15      0.88      0.26      1398\n",
      "\n",
      "    accuracy                           0.50     14306\n",
      "   macro avg       0.56      0.67      0.44     14306\n",
      "weighted avg       0.89      0.50      0.59     14306\n",
      "\n",
      "\n",
      "\n",
      "Classification report, forward_returns  2\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.98      0.51      0.67     12976\n",
      "         1.0       0.15      0.87      0.26      1330\n",
      "\n",
      "    accuracy                           0.55     14306\n",
      "   macro avg       0.56      0.69      0.47     14306\n",
      "weighted avg       0.90      0.55      0.63     14306\n",
      "\n",
      "\n",
      "\n",
      "Classification report, forward_returns  5\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.99      0.52      0.69     13660\n",
      "         1.0       0.08      0.89      0.15       646\n",
      "\n",
      "    accuracy                           0.54     14306\n",
      "   macro avg       0.54      0.70      0.42     14306\n",
      "weighted avg       0.95      0.54      0.66     14306\n",
      "\n",
      "\n",
      "\n",
      "Classification report, forward_returns  10\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.98      0.52      0.68     13718\n",
      "         1.0       0.07      0.80      0.12       588\n",
      "\n",
      "    accuracy                           0.53     14306\n",
      "   macro avg       0.53      0.66      0.40     14306\n",
      "weighted avg       0.95      0.53      0.66     14306\n",
      "\n",
      "\n",
      "\n",
      "Classification report, forward_returns  20\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.98      0.48      0.65     13775\n",
      "         1.0       0.05      0.75      0.10       531\n",
      "\n",
      "    accuracy                           0.49     14306\n",
      "   macro avg       0.52      0.62      0.37     14306\n",
      "weighted avg       0.95      0.49      0.63     14306\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "for i in range(5):\n",
    "  print(\"Classification report, forward_returns \",[1,2,5,10,20][i])\n",
    "  print(classification_report(pred[:,i],test_labels.iloc[:,i]))\n",
    "  print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wxR0z1nHClH1"
   },
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i1R3upoeZBJx"
   },
   "source": [
    "#### Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "IvWhji2sZTgx"
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "lZTLO0Ikbrit",
    "outputId": "a01411d7-bc15-4887-c616-f1b86993245c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report, forward_returns  1\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.74      0.74      0.74      5267\n",
      "         1.0       0.81      0.80      0.81      7093\n",
      "\n",
      "    accuracy                           0.78     12360\n",
      "   macro avg       0.77      0.77      0.77     12360\n",
      "weighted avg       0.78      0.78      0.78     12360\n",
      "\n",
      "\n",
      "\n",
      "Classification report, forward_returns  2\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.73      0.77      0.75      5502\n",
      "         1.0       0.81      0.77      0.79      6858\n",
      "\n",
      "    accuracy                           0.77     12360\n",
      "   macro avg       0.77      0.77      0.77     12360\n",
      "weighted avg       0.77      0.77      0.77     12360\n",
      "\n",
      "\n",
      "\n",
      "Classification report, forward_returns  5\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.59      0.78      0.67      4631\n",
      "         1.0       0.83      0.68      0.75      7729\n",
      "\n",
      "    accuracy                           0.71     12360\n",
      "   macro avg       0.71      0.73      0.71     12360\n",
      "weighted avg       0.74      0.71      0.72     12360\n",
      "\n",
      "\n",
      "\n",
      "Classification report, forward_returns  10\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.32      0.78      0.46      2559\n",
      "         1.0       0.91      0.57      0.70      9801\n",
      "\n",
      "    accuracy                           0.61     12360\n",
      "   macro avg       0.62      0.68      0.58     12360\n",
      "weighted avg       0.79      0.61      0.65     12360\n",
      "\n",
      "\n",
      "\n",
      "Classification report, forward_returns  20\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.31      0.68      0.43      2598\n",
      "         1.0       0.88      0.59      0.71      9762\n",
      "\n",
      "    accuracy                           0.61     12360\n",
      "   macro avg       0.59      0.64      0.57     12360\n",
      "weighted avg       0.76      0.61      0.65     12360\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "rf=RandomForestClassifier(max_depth=100)\n",
    "\n",
    "for i in range(5):\n",
    "  rf.fit(X_train,y_train.iloc[:,i])\n",
    "  pred=rf.predict(X_test)\n",
    "  print(\"Classification report, forward_returns \",[1,2,5,10,20][i])\n",
    "  print(classification_report(pred,y_test.iloc[:,i]))\n",
    "  print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5Br8KnfrZBNQ"
   },
   "source": [
    "#### Extremely randomized trees (Extra trees forest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "OS9q7QOGZT_W"
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import ExtraTreesClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "mNVf3dw-cA3i",
    "outputId": "f7db0a82-b151-4dfa-9614-d553d0bb4522"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report, forward_returns  1\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.68      0.74      0.71      4903\n",
      "         1.0       0.82      0.77      0.80      7457\n",
      "\n",
      "    accuracy                           0.76     12360\n",
      "   macro avg       0.75      0.76      0.75     12360\n",
      "weighted avg       0.76      0.76      0.76     12360\n",
      "\n",
      "\n",
      "\n",
      "Classification report, forward_returns  2\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.66      0.77      0.71      4995\n",
      "         1.0       0.83      0.73      0.78      7365\n",
      "\n",
      "    accuracy                           0.75     12360\n",
      "   macro avg       0.75      0.75      0.75     12360\n",
      "weighted avg       0.76      0.75      0.75     12360\n",
      "\n",
      "\n",
      "\n",
      "Classification report, forward_returns  5\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.46      0.76      0.57      3693\n",
      "         1.0       0.86      0.62      0.72      8667\n",
      "\n",
      "    accuracy                           0.66     12360\n",
      "   macro avg       0.66      0.69      0.64     12360\n",
      "weighted avg       0.74      0.66      0.67     12360\n",
      "\n",
      "\n",
      "\n",
      "Classification report, forward_returns  10\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.25      0.75      0.38      2134\n",
      "         1.0       0.91      0.55      0.68     10226\n",
      "\n",
      "    accuracy                           0.58     12360\n",
      "   macro avg       0.58      0.65      0.53     12360\n",
      "weighted avg       0.80      0.58      0.63     12360\n",
      "\n",
      "\n",
      "\n",
      "Classification report, forward_returns  20\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.19      0.71      0.30      1526\n",
      "         1.0       0.93      0.57      0.71     10834\n",
      "\n",
      "    accuracy                           0.59     12360\n",
      "   macro avg       0.56      0.64      0.50     12360\n",
      "weighted avg       0.84      0.59      0.66     12360\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "extra=ExtraTreesClassifier(max_depth=100)\n",
    "\n",
    "for i in range(5):\n",
    "  extra.fit(X_train,y_train.iloc[:,i])\n",
    "  pred=extra.predict(X_test)\n",
    "  print(\"Classification report, forward_returns \",[1,2,5,10,20][i])\n",
    "  print(classification_report(pred,y_test.iloc[:,i]))\n",
    "  print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7vxPU9hmZUHE"
   },
   "source": [
    "#### Gradient Boosting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jYHQPEnNXRaQ"
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import GradientBoostingClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "background_save": true
    },
    "id": "YbyRwZsmcLKV",
    "outputId": "1741e214-2436-4090-9973-861ec49854dd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report, forward_returns  1\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.67      0.66      0.67      5434\n",
      "         1.0       0.74      0.75      0.74      6926\n",
      "\n",
      "    accuracy                           0.71     12360\n",
      "   macro avg       0.71      0.70      0.71     12360\n",
      "weighted avg       0.71      0.71      0.71     12360\n",
      "\n",
      "\n",
      "\n",
      "Classification report, forward_returns  2\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.61      0.61      0.61      5852\n",
      "         1.0       0.65      0.65      0.65      6508\n",
      "\n",
      "    accuracy                           0.63     12360\n",
      "   macro avg       0.63      0.63      0.63     12360\n",
      "weighted avg       0.63      0.63      0.63     12360\n",
      "\n",
      "\n",
      "\n",
      "Classification report, forward_returns  5\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.45      0.59      0.51      4727\n",
      "         1.0       0.69      0.56      0.62      7633\n",
      "\n",
      "    accuracy                           0.57     12360\n",
      "   macro avg       0.57      0.57      0.57     12360\n",
      "weighted avg       0.60      0.57      0.58     12360\n",
      "\n",
      "\n",
      "\n",
      "Classification report, forward_returns  10\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.47      0.57      0.52      5158\n",
      "         1.0       0.64      0.54      0.59      7202\n",
      "\n",
      "    accuracy                           0.56     12360\n",
      "   macro avg       0.56      0.56      0.55     12360\n",
      "weighted avg       0.57      0.56      0.56     12360\n",
      "\n",
      "\n",
      "\n",
      "Classification report, forward_returns  20\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        -1.0       0.47      0.50      0.48      5399\n",
      "         1.0       0.59      0.56      0.57      6961\n",
      "\n",
      "    accuracy                           0.53     12360\n",
      "   macro avg       0.53      0.53      0.53     12360\n",
      "weighted avg       0.54      0.53      0.53     12360\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "gb=GradientBoostingClassifier(max_depth=100)\n",
    "\n",
    "for i in range(5):\n",
    "  gb.fit(X_train,y_train.iloc[:,i])\n",
    "  pred=gb.predict(X_test)\n",
    "  print(\"Classification report, forward_returns \",[1,2,5,10,20][i])\n",
    "  print(classification_report(pred,y_test.iloc[:,i]))\n",
    "  print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "I7ss5cfWr6dE"
   },
   "source": [
    "## Conclusions:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cMHlIj2egqAy"
   },
   "source": [
    "The easiest response to predict is the one with the highest accuracy for most of the models: 1-day returns which is totally predictable because the further we go into the future the more we have uncertainties and therefore it is harder to predict. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OCMxyKFIqweg"
   },
   "source": [
    "Decision Trees based classifiers had the best performance overall, this may be due to their ability to perform feature selection and thus choose the best features that describe best the behaviour of the target variable and due to their ability to beat overfitting with the bagging techniques.\n",
    "\n",
    "Sequence models based classifiers need more tunning to reach better performance as well as a feature selection mechanism to get rid of the useless features that may introduce noise and affect the performance of the model. Extra study is needed aswell to determin the right compression size for the different features comming out of the Autoencoder because a very low output dimension may lead to loss of information compared to the orginial features.\n",
    "\n",
    "The 5 targets to predict may be combined to work as a slope prediction that, if predicted with a certain high accuracy, can give a very accurate idea about the trend."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "9tGSkLkRsIWV",
    "lRCVm5YFsco5",
    "_l_mZz5Hs0Pc",
    "UCESflOcs9UR",
    "9sOTBtXuWywk",
    "t-gm-GPZXMuk"
   ],
   "name": "Copy of Final Project - Part 2.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}