{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction: Bayesian Linear Regression Project\n",
    "\n",
    "In this notebook, we will implement a complete machine learning project, focusing on Bayesian Inference methods, in particular, Bayesian Linear Regression. We will go through the entire machine learning process, cleaning the data, exploring it to find trends, establishing a baseline model, evaluating several machine learning approaches for comparisons, implementing Bayesian Linear Regression, interpreting the results, and presenting the results. Let's get started!\n",
    "\n",
    "## Dataset\n",
    "\n",
    "We are using data on student grades collected from a Portuguese secondary (high) school. This data is from the [UCI machine learning repository](https://archive.ics.uci.edu/ml/datasets/student+performance#), a great collection of datasets for model testing. The data includes academic and personal characteristics of the students as well as final grades. The objective is to predict the final grade from the student information which makes this a __supervised, regression task__. We have a set of training data with known labels, and we want the model to learn a mapping from the features (explanatory variables) to the target (the label) in this case the final grade. It is a regression task because the final grade is a continuous value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploratory Data Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pandas and numpy for data manipulation\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "np.random.seed(42)\n",
    "\n",
    " \n",
    "# Matplotlib and seaborn for plotting\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "import matplotlib\n",
    "matplotlib.rcParams['font.size'] = 16\n",
    "matplotlib.rcParams['figure.figsize'] = (9, 9)\n",
    "\n",
    "import seaborn as sns\n",
    "\n",
    "from IPython.core.pylabtools import figsize\n",
    "\n",
    "# Scipy helper functions\n",
    "from scipy.stats import percentileofscore\n",
    "from scipy import stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Standard ML Models for comparison\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.linear_model import ElasticNet\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.ensemble import ExtraTreesRegressor\n",
    "from sklearn.ensemble import GradientBoostingRegressor\n",
    "from sklearn.svm import SVR\n",
    "\n",
    "# Splitting data into training/testing\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "# Metrics\n",
    "from sklearn.metrics import mean_squared_error, mean_absolute_error, median_absolute_error\n",
    "\n",
    "# Distributions\n",
    "import scipy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.\n"
     ]
    }
   ],
   "source": [
    "# PyMC3 for Bayesian Inference\n",
    "import pymc3 as pm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read in Data and Examine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>school</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>address</th>\n",
       "      <th>famsize</th>\n",
       "      <th>Pstatus</th>\n",
       "      <th>Medu</th>\n",
       "      <th>Fedu</th>\n",
       "      <th>Mjob</th>\n",
       "      <th>Fjob</th>\n",
       "      <th>...</th>\n",
       "      <th>famrel</th>\n",
       "      <th>freetime</th>\n",
       "      <th>goout</th>\n",
       "      <th>Dalc</th>\n",
       "      <th>Walc</th>\n",
       "      <th>health</th>\n",
       "      <th>absences</th>\n",
       "      <th>G1</th>\n",
       "      <th>G2</th>\n",
       "      <th>Grade</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>GP</td>\n",
       "      <td>F</td>\n",
       "      <td>18</td>\n",
       "      <td>U</td>\n",
       "      <td>GT3</td>\n",
       "      <td>A</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>at_home</td>\n",
       "      <td>teacher</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>GP</td>\n",
       "      <td>F</td>\n",
       "      <td>17</td>\n",
       "      <td>U</td>\n",
       "      <td>GT3</td>\n",
       "      <td>T</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>at_home</td>\n",
       "      <td>other</td>\n",
       "      <td>...</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>11</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>GP</td>\n",
       "      <td>F</td>\n",
       "      <td>15</td>\n",
       "      <td>U</td>\n",
       "      <td>LE3</td>\n",
       "      <td>T</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>at_home</td>\n",
       "      <td>other</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>12</td>\n",
       "      <td>13</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>GP</td>\n",
       "      <td>F</td>\n",
       "      <td>15</td>\n",
       "      <td>U</td>\n",
       "      <td>GT3</td>\n",
       "      <td>T</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>health</td>\n",
       "      <td>services</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>GP</td>\n",
       "      <td>F</td>\n",
       "      <td>16</td>\n",
       "      <td>U</td>\n",
       "      <td>GT3</td>\n",
       "      <td>T</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>other</td>\n",
       "      <td>other</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>13</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 33 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  school sex  age address famsize Pstatus  Medu  Fedu     Mjob      Fjob  \\\n",
       "0     GP   F   18       U     GT3       A     4     4  at_home   teacher   \n",
       "1     GP   F   17       U     GT3       T     1     1  at_home     other   \n",
       "2     GP   F   15       U     LE3       T     1     1  at_home     other   \n",
       "3     GP   F   15       U     GT3       T     4     2   health  services   \n",
       "4     GP   F   16       U     GT3       T     3     3    other     other   \n",
       "\n",
       "   ...  famrel freetime  goout  Dalc  Walc health absences  G1  G2 Grade  \n",
       "0  ...       4        3      4     1     1      3        4   0  11    11  \n",
       "1  ...       5        3      3     1     1      3        2   9  11    11  \n",
       "2  ...       4        3      2     2     3      3        6  12  13    12  \n",
       "3  ...       3        2      2     1     1      5        0  14  14    14  \n",
       "4  ...       4        3      2     1     2      5        0  11  13    13  \n",
       "\n",
       "[5 rows x 33 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read in class scores\n",
    "df = pd.read_csv('data/student-mat.csv')\n",
    "\n",
    "# Filter out grades that were 0\n",
    "df = df[~df['G3'].isin([0, 1])]\n",
    "\n",
    "df = df.rename(columns={'G3': 'Grade'})\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(633, 33)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Describe for Numerical Columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>Medu</th>\n",
       "      <th>Fedu</th>\n",
       "      <th>traveltime</th>\n",
       "      <th>studytime</th>\n",
       "      <th>failures</th>\n",
       "      <th>famrel</th>\n",
       "      <th>freetime</th>\n",
       "      <th>goout</th>\n",
       "      <th>Dalc</th>\n",
       "      <th>Walc</th>\n",
       "      <th>health</th>\n",
       "      <th>absences</th>\n",
       "      <th>G1</th>\n",
       "      <th>G2</th>\n",
       "      <th>Grade</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>633.000000</td>\n",
       "      <td>633.000000</td>\n",
       "      <td>633.000000</td>\n",
       "      <td>633.000000</td>\n",
       "      <td>633.000000</td>\n",
       "      <td>633.000000</td>\n",
       "      <td>633.000000</td>\n",
       "      <td>633.000000</td>\n",
       "      <td>633.000000</td>\n",
       "      <td>633.000000</td>\n",
       "      <td>633.000000</td>\n",
       "      <td>633.000000</td>\n",
       "      <td>633.000000</td>\n",
       "      <td>633.000000</td>\n",
       "      <td>633.000000</td>\n",
       "      <td>633.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>16.720379</td>\n",
       "      <td>2.519747</td>\n",
       "      <td>2.320695</td>\n",
       "      <td>1.563981</td>\n",
       "      <td>1.941548</td>\n",
       "      <td>0.206951</td>\n",
       "      <td>3.932070</td>\n",
       "      <td>3.169036</td>\n",
       "      <td>3.184834</td>\n",
       "      <td>1.489731</td>\n",
       "      <td>2.268562</td>\n",
       "      <td>3.527646</td>\n",
       "      <td>3.751975</td>\n",
       "      <td>11.503949</td>\n",
       "      <td>11.756714</td>\n",
       "      <td>12.205371</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1.214044</td>\n",
       "      <td>1.133497</td>\n",
       "      <td>1.102284</td>\n",
       "      <td>0.750167</td>\n",
       "      <td>0.833284</td>\n",
       "      <td>0.579023</td>\n",
       "      <td>0.946411</td>\n",
       "      <td>1.047082</td>\n",
       "      <td>1.162014</td>\n",
       "      <td>0.912452</td>\n",
       "      <td>1.281776</td>\n",
       "      <td>1.448355</td>\n",
       "      <td>4.662003</td>\n",
       "      <td>2.682645</td>\n",
       "      <td>2.629715</td>\n",
       "      <td>2.656882</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>15.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>5.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>16.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>10.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>17.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>11.000000</td>\n",
       "      <td>12.000000</td>\n",
       "      <td>12.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>18.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>13.000000</td>\n",
       "      <td>13.000000</td>\n",
       "      <td>14.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>22.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>19.000000</td>\n",
       "      <td>19.000000</td>\n",
       "      <td>19.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              age        Medu        Fedu  traveltime   studytime    failures  \\\n",
       "count  633.000000  633.000000  633.000000  633.000000  633.000000  633.000000   \n",
       "mean    16.720379    2.519747    2.320695    1.563981    1.941548    0.206951   \n",
       "std      1.214044    1.133497    1.102284    0.750167    0.833284    0.579023   \n",
       "min     15.000000    0.000000    0.000000    1.000000    1.000000    0.000000   \n",
       "25%     16.000000    2.000000    1.000000    1.000000    1.000000    0.000000   \n",
       "50%     17.000000    2.000000    2.000000    1.000000    2.000000    0.000000   \n",
       "75%     18.000000    4.000000    3.000000    2.000000    2.000000    0.000000   \n",
       "max     22.000000    4.000000    4.000000    4.000000    4.000000    3.000000   \n",
       "\n",
       "           famrel    freetime       goout        Dalc        Walc      health  \\\n",
       "count  633.000000  633.000000  633.000000  633.000000  633.000000  633.000000   \n",
       "mean     3.932070    3.169036    3.184834    1.489731    2.268562    3.527646   \n",
       "std      0.946411    1.047082    1.162014    0.912452    1.281776    1.448355   \n",
       "min      1.000000    1.000000    1.000000    1.000000    1.000000    1.000000   \n",
       "25%      4.000000    3.000000    2.000000    1.000000    1.000000    2.000000   \n",
       "50%      4.000000    3.000000    3.000000    1.000000    2.000000    4.000000   \n",
       "75%      5.000000    4.000000    4.000000    2.000000    3.000000    5.000000   \n",
       "max      5.000000    5.000000    5.000000    5.000000    5.000000    5.000000   \n",
       "\n",
       "         absences          G1          G2       Grade  \n",
       "count  633.000000  633.000000  633.000000  633.000000  \n",
       "mean     3.751975   11.503949   11.756714   12.205371  \n",
       "std      4.662003    2.682645    2.629715    2.656882  \n",
       "min      0.000000    0.000000    5.000000    5.000000  \n",
       "25%      0.000000   10.000000   10.000000   10.000000  \n",
       "50%      2.000000   11.000000   12.000000   12.000000  \n",
       "75%      6.000000   13.000000   13.000000   14.000000  \n",
       "max     32.000000   19.000000   19.000000   19.000000  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Value Counts for Categorical Columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Column Name: school\n",
      "GP    421\n",
      "MS    212\n",
      "Name: school, dtype: int64\n",
      "\n",
      "Column Name: sex\n",
      "F    376\n",
      "M    257\n",
      "Name: sex, dtype: int64\n",
      "\n",
      "Column Name: address\n",
      "U    446\n",
      "R    187\n",
      "Name: address, dtype: int64\n",
      "\n",
      "Column Name: famsize\n",
      "GT3    443\n",
      "LE3    190\n",
      "Name: famsize, dtype: int64\n",
      "\n",
      "Column Name: Pstatus\n",
      "T    555\n",
      "A     78\n",
      "Name: Pstatus, dtype: int64\n",
      "\n",
      "Column Name: Mjob\n",
      "other       249\n",
      "services    135\n",
      "at_home     131\n",
      "teacher      70\n",
      "health       48\n",
      "Name: Mjob, dtype: int64\n",
      "\n",
      "Column Name: Fjob\n",
      "other       359\n",
      "services    175\n",
      "at_home      41\n",
      "teacher      35\n",
      "health       23\n",
      "Name: Fjob, dtype: int64\n",
      "\n",
      "Column Name: reason\n",
      "course        279\n",
      "home          146\n",
      "reputation    141\n",
      "other          67\n",
      "Name: reason, dtype: int64\n",
      "\n",
      "Column Name: guardian\n",
      "mother    444\n",
      "father    149\n",
      "other      40\n",
      "Name: guardian, dtype: int64\n",
      "\n",
      "Column Name: schoolsup\n",
      "no     566\n",
      "yes     67\n",
      "Name: schoolsup, dtype: int64\n",
      "\n",
      "Column Name: famsup\n",
      "yes    393\n",
      "no     240\n",
      "Name: famsup, dtype: int64\n",
      "\n",
      "Column Name: paid\n",
      "no     595\n",
      "yes     38\n",
      "Name: paid, dtype: int64\n",
      "\n",
      "Column Name: activities\n",
      "no     326\n",
      "yes    307\n",
      "Name: activities, dtype: int64\n",
      "\n",
      "Column Name: nursery\n",
      "yes    507\n",
      "no     126\n",
      "Name: nursery, dtype: int64\n",
      "\n",
      "Column Name: higher\n",
      "yes    569\n",
      "no      64\n",
      "Name: higher, dtype: int64\n",
      "\n",
      "Column Name: internet\n",
      "yes    489\n",
      "no     144\n",
      "Name: internet, dtype: int64\n",
      "\n",
      "Column Name: romantic\n",
      "no     404\n",
      "yes    229\n",
      "Name: romantic, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# Print the value counts for categorical columns\n",
    "for col in df.columns:\n",
    "    if df[col].dtype == 'object':\n",
    "        print('\\nColumn Name:', col,)\n",
    "        print(df[col].value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distribution of Grades"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    633.000000\n",
       "mean      12.205371\n",
       "std        2.656882\n",
       "min        5.000000\n",
       "25%       10.000000\n",
       "50%       12.000000\n",
       "75%       14.000000\n",
       "max       19.000000\n",
       "Name: Grade, dtype: float64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Grade'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11    104\n",
       "10     97\n",
       "13     82\n",
       "12     72\n",
       "14     63\n",
       "15     49\n",
       "16     36\n",
       "9      35\n",
       "8      35\n",
       "17     29\n",
       "18     15\n",
       "7      10\n",
       "6       3\n",
       "19      2\n",
       "5       1\n",
       "Name: Grade, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Grade'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 9412 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Bar plot of grades\n",
    "plt.bar(df['Grade'].value_counts().index, \n",
    "        df['Grade'].value_counts().values,\n",
    "         fill = 'navy', edgecolor = 'k', width = 1)\n",
    "plt.xlabel('Grade'); plt.ylabel('Count'); plt.title('Distribution of Final Grades');\n",
    "plt.xticks(list(range(5, 20)));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Grade Distribution by Different Categorical Variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 27972 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Grade distribution by address\n",
    "sns.kdeplot(df.ix[df['address'] == 'U', 'Grade'], label = 'Urban', shade = True)\n",
    "sns.kdeplot(df.ix[df['address'] == 'R', 'Grade'], label = 'Rural', shade = True)\n",
    "plt.xlabel('Grade'); plt.ylabel('Density'); plt.title('Density Plot of Final Grades by Location');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 35268 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Grade distribution by Guardian\n",
    "sns.kdeplot(df.ix[df['guardian'] == 'father', 'Grade'], label = 'Father', shade = True)\n",
    "sns.kdeplot(df.ix[df['guardian'] == 'mother', 'Grade'], label = 'Mother', shade = True)\n",
    "sns.kdeplot(df.ix[df['guardian'] == 'other', 'Grade'], label = 'Other', shade = True)\n",
    "plt.xlabel('Grade'); plt.ylabel('Density'); plt.title('Density Plot of Final Grades by Guardian');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 29020 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Grade distribution by internet\n",
    "sns.kdeplot(df.ix[df['internet'] == 'yes', 'Grade'], label = 'Internet', shade = True)\n",
    "sns.kdeplot(df.ix[df['internet'] == 'no', 'Grade'], label = 'No Internet', shade = True)\n",
    "plt.xlabel('Grade'); plt.ylabel('Density'); plt.title('Density Plot of Final Grades by Internet Access');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 27844 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Grade distribution by school\n",
    "sns.kdeplot(df.ix[df['school'] == 'GP', 'Grade'], label = 'GP', shade = True)\n",
    "sns.kdeplot(df.ix[df['school'] == 'MS', 'Grade'], label = 'MS', shade = True)\n",
    "plt.xlabel('Grade'); plt.ylabel('Count'); plt.title('Distribution of Final Grades by School');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "school  address\n",
       "GP      U          343\n",
       "        R           78\n",
       "MS      R          109\n",
       "        U          103\n",
       "Name: address, dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Look at distribution of schools by address\n",
    "schools = df.groupby(['school'])['address'].value_counts()\n",
    "schools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Grade Percentiles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate percentile for grades\n",
    "df['percentile'] = df['Grade'].apply(lambda x: percentileofscore(df['Grade'], x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 10528 characters]\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot percentiles for grades\n",
    "plt.figure(figsize = (8, 6))\n",
    "plt.plot(df['Grade'], df['percentile'], 'o')\n",
    "plt.xticks(range(0, 20, 2), range(0, 20, 2))\n",
    "plt.xlabel('Score'); plt.ylabel('Percentile'); plt.title('Grade Percentiles');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50th percentile score: 12\n",
      "Minimum Score needed for 90th percentile: 17\n"
     ]
    }
   ],
   "source": [
    "print('50th percentile score:', np.min(df.loc[df['percentile'] > 50, 'Grade']))\n",
    "print('Minimum Score needed for 90th percentile:', np.min(df.loc[df['percentile'] > 90, 'Grade']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Variable Correlations with Final Grade"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Numerical Correlations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "failures     -0.384569\n",
       "absences     -0.204230\n",
       "Dalc         -0.196891\n",
       "Walc         -0.178839\n",
       "traveltime   -0.129654\n",
       "goout        -0.111228\n",
       "freetime     -0.105206\n",
       "health       -0.096461\n",
       "age          -0.042505\n",
       "famrel        0.072888\n",
       "Fedu          0.204392\n",
       "studytime     0.249855\n",
       "Medu          0.278690\n",
       "G1            0.874777\n",
       "G2            0.942691\n",
       "percentile    0.985253\n",
       "Grade         1.000000\n",
       "Name: Grade, dtype: float64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Correlations of numerical values\n",
    "df.corr()['Grade'].sort_values()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Categorical Correlations using One-Hot Encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>school_GP</th>\n",
       "      <th>school_MS</th>\n",
       "      <th>sex_F</th>\n",
       "      <th>sex_M</th>\n",
       "      <th>address_R</th>\n",
       "      <th>address_U</th>\n",
       "      <th>famsize_GT3</th>\n",
       "      <th>famsize_LE3</th>\n",
       "      <th>Pstatus_A</th>\n",
       "      <th>Pstatus_T</th>\n",
       "      <th>...</th>\n",
       "      <th>activities_yes</th>\n",
       "      <th>nursery_no</th>\n",
       "      <th>nursery_yes</th>\n",
       "      <th>higher_no</th>\n",
       "      <th>higher_yes</th>\n",
       "      <th>internet_no</th>\n",
       "      <th>internet_yes</th>\n",
       "      <th>romantic_no</th>\n",
       "      <th>romantic_yes</th>\n",
       "      <th>Grade</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 44 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   school_GP  school_MS  sex_F  sex_M  address_R  address_U  famsize_GT3  \\\n",
       "0          1          0      1      0          0          1            1   \n",
       "1          1          0      1      0          0          1            1   \n",
       "2          1          0      1      0          0          1            0   \n",
       "3          1          0      1      0          0          1            1   \n",
       "4          1          0      1      0          0          1            1   \n",
       "\n",
       "   famsize_LE3  Pstatus_A  Pstatus_T  ...    activities_yes  nursery_no  \\\n",
       "0            0          1          0  ...                 0           0   \n",
       "1            0          0          1  ...                 0           1   \n",
       "2            1          0          1  ...                 0           0   \n",
       "3            0          0          1  ...                 1           0   \n",
       "4            0          0          1  ...                 0           0   \n",
       "\n",
       "   nursery_yes  higher_no  higher_yes  internet_no  internet_yes  romantic_no  \\\n",
       "0            1          0           1            1             0            1   \n",
       "1            0          0           1            0             1            1   \n",
       "2            1          0           1            0             1            1   \n",
       "3            1          0           1            0             1            0   \n",
       "4            1          0           1            1             0            1   \n",
       "\n",
       "   romantic_yes  Grade  \n",
       "0             0     11  \n",
       "1             0     11  \n",
       "2             0     12  \n",
       "3             1     14  \n",
       "4             0     13  \n",
       "\n",
       "[5 rows x 44 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Select only categorical variables\n",
    "category_df = df.select_dtypes('object')\n",
    "# One hot encode the variables\n",
    "dummy_df = pd.get_dummies(category_df)\n",
    "# Put the grade back in the dataframe\n",
    "dummy_df['Grade'] = df['Grade']\n",
    "dummy_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "higher_no           -0.343742\n",
       "school_MS           -0.227632\n",
       "Mjob_at_home        -0.158496\n",
       "reason_course       -0.138195\n",
       "internet_no         -0.131408\n",
       "address_R           -0.128350\n",
       "sex_M               -0.125755\n",
       "guardian_other      -0.100802\n",
       "schoolsup_yes       -0.098185\n",
       "reason_other        -0.092382\n",
       "activities_no       -0.078526\n",
       "paid_yes            -0.067146\n",
       "nursery_no          -0.056449\n",
       "romantic_yes        -0.052050\n",
       "Fjob_at_home        -0.049372\n",
       "Fjob_services       -0.042497\n",
       "Mjob_other          -0.035494\n",
       "Fjob_other          -0.021292\n",
       "famsize_GT3         -0.012957\n",
       "guardian_mother     -0.008042\n",
       "Pstatus_T           -0.001776\n",
       "famsup_no           -0.001581\n",
       "famsup_yes           0.001581\n",
       "Pstatus_A            0.001776\n",
       "Mjob_services        0.006210\n",
       "famsize_LE3          0.012957\n",
       "Fjob_health          0.026320\n",
       "reason_home          0.046639\n",
       "romantic_no          0.052050\n",
       "nursery_yes          0.056449\n",
       "guardian_father      0.066486\n",
       "paid_no              0.067146\n",
       "activities_yes       0.078526\n",
       "Mjob_health          0.092483\n",
       "schoolsup_no         0.098185\n",
       "sex_F                0.125755\n",
       "address_U            0.128350\n",
       "internet_yes         0.131408\n",
       "Fjob_teacher         0.160938\n",
       "Mjob_teacher         0.173851\n",
       "reason_reputation    0.185979\n",
       "school_GP            0.227632\n",
       "higher_yes           0.343742\n",
       "Grade                1.000000\n",
       "Name: Grade, dtype: float64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Correlations in one-hot encoded dataframe\n",
    "dummy_df.corr()['Grade'].sort_values()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Select 5 Most Correlated Variables with Final Score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Takes in a dataframe, finds the most correlated variables with the\n",
    "# grade and returns training and testing datasets\n",
    "def format_data(df):\n",
    "    # Targets are final grade of student\n",
    "    labels = df['Grade']\n",
    "    \n",
    "    # Drop the school and the grades from features\n",
    "    df = df.drop(columns=['school', 'G1', 'G2', 'percentile'])\n",
    "    \n",
    "    # One-Hot Encoding of Categorical Variables\n",
    "    df = pd.get_dummies(df)\n",
    "    \n",
    "    # Find correlations with the Grade\n",
    "    most_correlated = df.corr().abs()['Grade'].sort_values(ascending=False)\n",
    "    \n",
    "    # Maintain the top 5 most correlation features with Grade\n",
    "    most_correlated = most_correlated[:8]\n",
    "    \n",
    "    df = df.ix[:, most_correlated.index]\n",
    "    df = df.drop(columns = 'higher_no')\n",
    "    \n",
    "    # Split into training/testing sets with 25% split\n",
    "    X_train, X_test, y_train, y_test = train_test_split(df, labels, \n",
    "                                                        test_size = 0.25,\n",
    "                                                        random_state=42)\n",
    "    \n",
    "    return X_train, X_test, y_train, y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Grade</th>\n",
       "      <th>failures</th>\n",
       "      <th>higher_yes</th>\n",
       "      <th>Medu</th>\n",
       "      <th>studytime</th>\n",
       "      <th>Fedu</th>\n",
       "      <th>absences</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>619</th>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>323</th>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>201</th>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>410</th>\n",
       "      <td>17</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>426</th>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Grade  failures  higher_yes  Medu  studytime  Fedu  absences\n",
       "619     13         0           1     3          2     3         6\n",
       "323     10         0           1     4          1     3         0\n",
       "201     16         0           1     2          2     1         0\n",
       "410     17         0           1     2          2     2         0\n",
       "426     11         0           1     3          1     3         4"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train, X_test, y_train, y_test = format_data(df)\n",
    "X_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Rename Variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Rename variables in train and teste\n",
    "X_train = X_train.rename(columns={'higher_yes': 'higher_edu', \n",
    "                                  'Medu': 'mother_edu',\n",
    "                                  'Fedu': 'father_edu'})\n",
    "\n",
    "X_test = X_test.rename(columns={'higher_yes': 'higher_edu', \n",
    "                                  'Medu': 'mother_edu',\n",
    "                                  'Fedu': 'father_edu'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Grade</th>\n",
       "      <th>failures</th>\n",
       "      <th>higher_edu</th>\n",
       "      <th>mother_edu</th>\n",
       "      <th>studytime</th>\n",
       "      <th>father_edu</th>\n",
       "      <th>absences</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>619</th>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>323</th>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>201</th>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>410</th>\n",
       "      <td>17</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>426</th>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Grade  failures  higher_edu  mother_edu  studytime  father_edu  absences\n",
       "619     13         0           1           3          2           3         6\n",
       "323     10         0           1           4          1           3         0\n",
       "201     16         0           1           2          2           1         0\n",
       "410     17         0           1           2          2           2         0\n",
       "426     11         0           1           3          1           3         4"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(474, 7)\n",
      "(159, 7)\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape)\n",
    "print(X_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pairs Plot of Selected Variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 352768 characters]\n",
      "text/plain": [
       "<Figure size 1260x1260 with 56 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Calculate correlation coefficient\n",
    "def corrfunc(x, y, **kws):\n",
    "    r, _ = stats.pearsonr(x, y)\n",
    "    ax = plt.gca()\n",
    "    ax.annotate(\"r = {:.2f}\".format(r),\n",
    "                xy=(.1, .6), xycoords=ax.transAxes,\n",
    "               size = 24)\n",
    "    \n",
    "cmap = sns.cubehelix_palette(light=1, dark = 0.1,\n",
    "                             hue = 0.5, as_cmap=True)\n",
    "\n",
    "sns.set_context(font_scale=2)\n",
    "\n",
    "# Pair grid set up\n",
    "g = sns.PairGrid(X_train)\n",
    "\n",
    "# Scatter plot on the upper triangle\n",
    "g.map_upper(plt.scatter, s=10, color = 'red')\n",
    "\n",
    "# Distribution on the diagonal\n",
    "g.map_diag(sns.distplot, kde=False, color = 'red')\n",
    "\n",
    "# Density Plot and Correlation coefficients on the lower triangle\n",
    "g.map_lower(sns.kdeplot, cmap = cmap)\n",
    "g.map_lower(corrfunc);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create relation to the median grade column\n",
    "X_plot = X_train.copy()\n",
    "X_plot['relation_median'] = (X_plot['Grade'] >= 12)\n",
    "X_plot['relation_median'] = X_plot['relation_median'].replace({True: 'above', False: 'below'})\n",
    "X_plot = X_plot.drop(columns='Grade')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selected Variables Distribution by Relation to Median"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 142392 characters]\n",
      "text/plain": [
       "<Figure size 864x864 with 6 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(12, 12))\n",
    "# Plot the distribution of each variable colored\n",
    "# by the relation to the median grade\n",
    "for i, col in enumerate(X_plot.columns[:-1]):\n",
    "    plt.subplot(3, 2, i + 1)\n",
    "    subset_above = X_plot[X_plot['relation_median'] == 'above']\n",
    "    subset_below = X_plot[X_plot['relation_median'] == 'below']\n",
    "    sns.kdeplot(subset_above[col], label = 'Above Median', color = 'green')\n",
    "    sns.kdeplot(subset_below[col], label = 'Below Median', color = 'red')\n",
    "    plt.legend(); plt.title('Distribution of %s' % col)\n",
    "    \n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Establish Benchmarks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Metrics\n",
    "\n",
    "For this regression task, we will use two standard metrics:\n",
    "\n",
    "* Mean Absolute Error (MAE): Average of the absolute value of the difference between predictions and the true values\n",
    "* Root Mean Squared Error (RMSE): The square root of the average of the squared differences between the predictions and the true values.\n",
    "\n",
    "The mean absolute error is more interpretable, but the root mean squared error penalizes larger errors more heavily. Either one may be appropriate depending on the situation. \n",
    "[Here is a discussion](https://medium.com/human-in-a-machine-world/mae-and-rmse-which-metric-is-better-e60ac3bde13d)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate mae and rmse\n",
    "def evaluate_predictions(predictions, true):\n",
    "    mae = np.mean(abs(predictions - true))\n",
    "    rmse = np.sqrt(np.mean((predictions - true) ** 2))\n",
    "    \n",
    "    return mae, rmse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Naive Baseline\n",
    "\n",
    "For a regression task, a simple naive baseline is to guess the median value on the training set for all testing cases. If our machine learning model cannot better this simple baseline, then perhaps we should try a different approach! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Naive baseline is the median\n",
    "median_pred = X_train['Grade'].median()\n",
    "median_preds = [median_pred for _ in range(len(X_test))]\n",
    "true = X_test['Grade']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Median Baseline  MAE: 2.1761\n",
      "Median Baseline RMSE: 2.6777\n"
     ]
    }
   ],
   "source": [
    "# Display the naive baseline metrics\n",
    "mb_mae, mb_rmse = evaluate_predictions(median_preds, true)\n",
    "print('Median Baseline  MAE: {:.4f}'.format(mb_mae))\n",
    "print('Median Baseline RMSE: {:.4f}'.format(mb_rmse))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Standard Machine Learning Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluate several ml models by training on training set and testing on testing set\n",
    "def evaluate(X_train, X_test, y_train, y_test):\n",
    "    # Names of models\n",
    "    model_name_list = ['Linear Regression', 'ElasticNet Regression',\n",
    "                      'Random Forest', 'Extra Trees', 'SVM',\n",
    "                       'Gradient Boosted', 'Baseline']\n",
    "    X_train = X_train.drop(columns='Grade')\n",
    "    X_test = X_test.drop(columns='Grade')\n",
    "    \n",
    "    # Instantiate the models\n",
    "    model1 = LinearRegression()\n",
    "    model2 = ElasticNet(alpha=1.0, l1_ratio=0.5)\n",
    "    model3 = RandomForestRegressor(n_estimators=50)\n",
    "    model4 = ExtraTreesRegressor(n_estimators=50)\n",
    "    model5 = SVR(kernel='rbf', degree=3, C=1.0, gamma='auto')\n",
    "    model6 = GradientBoostingRegressor(n_estimators=20)\n",
    "    \n",
    "    # Dataframe for results\n",
    "    results = pd.DataFrame(columns=['mae', 'rmse'], index = model_name_list)\n",
    "    \n",
    "    # Train and predict with each model\n",
    "    for i, model in enumerate([model1, model2, model3, model4, model5, model6]):\n",
    "        model.fit(X_train, y_train)\n",
    "        predictions = model.predict(X_test)\n",
    "        \n",
    "        # Metrics\n",
    "        mae = np.mean(abs(predictions - y_test))\n",
    "        rmse = np.sqrt(np.mean((predictions - y_test) ** 2))\n",
    "        \n",
    "        # Insert results into the dataframe\n",
    "        model_name = model_name_list[i]\n",
    "        results.ix[model_name, :] = [mae, rmse]\n",
    "    \n",
    "    # Median Value Baseline Metrics\n",
    "    baseline = np.median(y_train)\n",
    "    baseline_mae = np.mean(abs(baseline - y_test))\n",
    "    baseline_rmse = np.sqrt(np.mean((baseline - y_test) ** 2))\n",
    "    \n",
    "    results.ix['Baseline', :] = [baseline_mae, baseline_rmse]\n",
    "    \n",
    "    return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = evaluate(X_train, X_test, y_train, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Visual Comparison of Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 39424 characters]\n",
      "text/plain": [
       "<Figure size 864x576 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "figsize(12, 8)\n",
    "matplotlib.rcParams['font.size'] = 16\n",
    "# Root mean squared error\n",
    "ax =  plt.subplot(1, 2, 1)\n",
    "results.sort_values('mae', ascending = True).plot.bar(y = 'mae', color = 'b', ax = ax)\n",
    "plt.title('Model Mean Absolute Error'); plt.ylabel('MAE');\n",
    "\n",
    "# Median absolute percentage error\n",
    "ax = plt.subplot(1, 2, 2)\n",
    "results.sort_values('rmse', ascending = True).plot.bar(y = 'rmse', color = 'r', ax = ax)\n",
    "plt.title('Model Root Mean Squared Error'); plt.ylabel('RMSE');\n",
    "\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mae</th>\n",
       "      <th>rmse</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Linear Regression</th>\n",
       "      <td>1.8859</td>\n",
       "      <td>2.27395</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ElasticNet Regression</th>\n",
       "      <td>2.09282</td>\n",
       "      <td>2.53656</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Random Forest</th>\n",
       "      <td>1.85529</td>\n",
       "      <td>2.32467</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Extra Trees</th>\n",
       "      <td>1.97657</td>\n",
       "      <td>2.4871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SVM</th>\n",
       "      <td>1.90262</td>\n",
       "      <td>2.30825</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Gradient Boosted</th>\n",
       "      <td>1.81272</td>\n",
       "      <td>2.22737</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Baseline</th>\n",
       "      <td>2.1761</td>\n",
       "      <td>2.67765</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           mae     rmse\n",
       "Linear Regression       1.8859  2.27395\n",
       "ElasticNet Regression  2.09282  2.53656\n",
       "Random Forest          1.85529  2.32467\n",
       "Extra Trees            1.97657   2.4871\n",
       "SVM                    1.90262  2.30825\n",
       "Gradient Boosted       1.81272  2.22737\n",
       "Baseline                2.1761  2.67765"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Gradient Boosted regressor is 16.70% better than the baseline.\n"
     ]
    }
   ],
   "source": [
    "print('The Gradient Boosted regressor is {:0.2f}% better than the baseline.'.format(\n",
    "    (100 * abs(results.loc['Gradient Boosted', 'mae'] - results.loc['Baseline', 'mae'])) / results.loc['Baseline', 'mae']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Formula from Ordinary Least Squares Linear Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Grade = 9.19 + -1.32 * failures + 1.86 * higher_edu + 0.26 * mother_edu + 0.58 * studytime + 0.03 * father_edu + -0.07 * absences'"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = LinearRegression()\n",
    "lr.fit(X_train.drop(columns='Grade'), y_train)\n",
    "\n",
    "ols_formula = 'Grade = %0.2f +' % lr.intercept_\n",
    "for i, col in enumerate(X_train.columns[1:]):\n",
    "    ols_formula += ' %0.2f * %s +' % (lr.coef_[i], col)\n",
    "    \n",
    "' '.join(ols_formula.split(' ')[:-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implementing Bayesian Linear Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Grade ~ failures + higher_edu + mother_edu + studytime + father_edu + absences'"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Formula for Bayesian Linear Regression (follows R formula syntax\n",
    "formula = 'Grade ~ ' + ' + '.join(['%s' % variable for variable in X_train.columns[1:]])\n",
    "formula"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Model in PyMC3 and Sample from Posterior\n",
    "\n",
    "We now build the model using the formula defined above and a normal distribution for the data likelihood. Then, we let a Markov Chain Monte Carlo algorithm draw samples from the posterior to approximate the posterior for each of the model parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Auto-assigning NUTS sampler...\n",
      "Initializing NUTS using jitter+adapt_diag...\n",
      "Sequential sampling (2 chains in 1 job)\n",
      "NUTS: [sd_log__, absences, father_edu, studytime, mother_edu, higher_edu, failures, Intercept]\n",
      "100%|██████████| 2500/2500 [00:27<00:00, 92.23it/s]\n",
      "100%|██████████| 2500/2500 [00:26<00:00, 95.17it/s]\n",
      "The acceptance probability does not match the target. It is 0.8918797997400376, but should be close to 0.8. Try to increase the number of tuning steps.\n"
     ]
    }
   ],
   "source": [
    "# Context for the model\n",
    "with pm.Model() as normal_model:\n",
    "    \n",
    "    # The prior for the model parameters will be a normal distribution\n",
    "    family = pm.glm.families.Normal()\n",
    "    \n",
    "    # Creating the model requires a formula and data (and optionally a family)\n",
    "    pm.GLM.from_formula(formula, data = X_train, family = family)\n",
    "    \n",
    "    # Perform Markov Chain Monte Carlo sampling\n",
    "    normal_trace = pm.sample(draws=2000, chains = 2, tune = 500, njobs=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Examine Bayesian Linear Regression Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Traceplot of All Samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Shows the trace with a vertical line at the mean of the trace\n",
    "def plot_trace(trace):\n",
    "    # Traceplot with vertical lines at the mean value\n",
    "    ax = pm.traceplot(trace, figsize=(14, len(trace.varnames)*1.8),\n",
    "                      lines={k: v['mean'] for k, v in pm.df_summary(trace).iterrows()})\n",
    "    \n",
    "    matplotlib.rcParams['font.size'] = 16\n",
    "    \n",
    "    # Labels with the median value\n",
    "    for i, mn in enumerate(pm.df_summary(trace)['mean']):\n",
    "        ax[i, 0].annotate('{:0.2f}'.format(mn), xy = (mn, 0), xycoords = 'data', size = 8,\n",
    "                          xytext = (-18, 18), textcoords = 'offset points', rotation = 90,\n",
    "                          va = 'bottom', fontsize = 'large', color = 'red')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 367032 characters]\n",
      "text/plain": [
       "<Figure size 1008x1166.4 with 16 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_trace(normal_trace);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 304612 characters]\n",
      "text/plain": [
       "<Figure size 864x1152 with 16 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pm.traceplot(normal_trace);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The left side of the traceplot is the marginal posterior: the values for the variable are on the x-axis with the probability for the variable (as determined by sampling) on the y-axis. The different colored lines indicate that we performed two chains of Markov Chain Monte Carlo. From the left side we can see that there is a range of values for each weight. The right side shows the different sample values drawn as the sampling process runs. \n",
    "\n",
    "Another method built into PyMC3 for examinig trace results is the forestplot which shows the distribution of each sampled parameter. This allows us to see the uncertainty in each sample. The forestplot is easily constructed from the trace using `pm.forestplot`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 29472 characters]\n",
      "text/plain": [
       "<Figure size 864x576 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pm.forestplot(normal_trace);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the forest plot we can see the most likely value of the parameter (the dot) as well as the 95% credible interval for the parameter. The `intercept` and `higher_edu` have larger uncertainty compared to the other variables. \n",
    "\n",
    "Another built in plotting method in PyMC3 is the posterior distribution of all the model parameters. These histograms allow us to see how the model result is a distribution for the parameters rather than a single value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 105124 characters]\n",
      "text/plain": [
       "<Figure size 1008x1008 with 8 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pm.plot_posterior(normal_trace, figsize = (14, 14), text_size=20);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variable: Intercept       Mean weight in model: 9.1974\n",
      "Variable: failures        Mean weight in model: -1.3166\n",
      "Variable: higher_edu      Mean weight in model: 1.8548\n",
      "Variable: mother_edu      Mean weight in model: 0.2636\n",
      "Variable: studytime       Mean weight in model: 0.5772\n",
      "Variable: father_edu      Mean weight in model: 0.0303\n",
      "Variable: absences        Mean weight in model: -0.0676\n",
      "Variable: sd_log__        Mean weight in model: 0.8244\n",
      "Variable: sd              Mean weight in model: 2.2817\n"
     ]
    }
   ],
   "source": [
    "# Print out the mean variable weight from the trace\n",
    "for variable in normal_trace.varnames:\n",
    "    print('Variable: {:15} Mean weight in model: {:.4f}'.format(variable, \n",
    "                                                                np.mean(normal_trace[variable])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interpretations of Weights\n",
    "\n",
    "Based on the sign and location of the weights, we can make the following inferences regarding the features in our dataset:\n",
    "\n",
    "* Previous class failures are negatively related to the students final grade\n",
    "* Higher education ambitions are positively related to the students grade\n",
    "* The mother's and father's education levels are positively related to the students final grade\n",
    "* Studying time per week is positively related to the students final grade\n",
    "* Absences are negatively related to the students final grade"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>sd</th>\n",
       "      <th>mc_error</th>\n",
       "      <th>hpd_2.5</th>\n",
       "      <th>hpd_97.5</th>\n",
       "      <th>n_eff</th>\n",
       "      <th>Rhat</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Intercept</th>\n",
       "      <td>9.197350</td>\n",
       "      <td>0.466233</td>\n",
       "      <td>0.009974</td>\n",
       "      <td>8.339046</td>\n",
       "      <td>10.129492</td>\n",
       "      <td>2505.0</td>\n",
       "      <td>0.999956</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>failures</th>\n",
       "      <td>-1.316566</td>\n",
       "      <td>0.200428</td>\n",
       "      <td>0.003174</td>\n",
       "      <td>-1.701078</td>\n",
       "      <td>-0.917525</td>\n",
       "      <td>3879.0</td>\n",
       "      <td>0.999768</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>higher_edu</th>\n",
       "      <td>1.854784</td>\n",
       "      <td>0.361739</td>\n",
       "      <td>0.006817</td>\n",
       "      <td>1.202987</td>\n",
       "      <td>2.607551</td>\n",
       "      <td>3202.0</td>\n",
       "      <td>1.000270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mother_edu</th>\n",
       "      <td>0.263578</td>\n",
       "      <td>0.125141</td>\n",
       "      <td>0.002255</td>\n",
       "      <td>0.019157</td>\n",
       "      <td>0.509765</td>\n",
       "      <td>2942.0</td>\n",
       "      <td>0.999802</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>studytime</th>\n",
       "      <td>0.577236</td>\n",
       "      <td>0.127546</td>\n",
       "      <td>0.002195</td>\n",
       "      <td>0.326653</td>\n",
       "      <td>0.827031</td>\n",
       "      <td>3301.0</td>\n",
       "      <td>0.999762</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>father_edu</th>\n",
       "      <td>0.030319</td>\n",
       "      <td>0.128503</td>\n",
       "      <td>0.002093</td>\n",
       "      <td>-0.227607</td>\n",
       "      <td>0.272364</td>\n",
       "      <td>3020.0</td>\n",
       "      <td>0.999836</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>absences</th>\n",
       "      <td>-0.067602</td>\n",
       "      <td>0.023550</td>\n",
       "      <td>0.000372</td>\n",
       "      <td>-0.114305</td>\n",
       "      <td>-0.022007</td>\n",
       "      <td>3512.0</td>\n",
       "      <td>0.999754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sd</th>\n",
       "      <td>2.281731</td>\n",
       "      <td>0.076456</td>\n",
       "      <td>0.001325</td>\n",
       "      <td>2.141054</td>\n",
       "      <td>2.438024</td>\n",
       "      <td>3405.0</td>\n",
       "      <td>1.001020</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                mean        sd  mc_error   hpd_2.5   hpd_97.5   n_eff  \\\n",
       "Intercept   9.197350  0.466233  0.009974  8.339046  10.129492  2505.0   \n",
       "failures   -1.316566  0.200428  0.003174 -1.701078  -0.917525  3879.0   \n",
       "higher_edu  1.854784  0.361739  0.006817  1.202987   2.607551  3202.0   \n",
       "mother_edu  0.263578  0.125141  0.002255  0.019157   0.509765  2942.0   \n",
       "studytime   0.577236  0.127546  0.002195  0.326653   0.827031  3301.0   \n",
       "father_edu  0.030319  0.128503  0.002093 -0.227607   0.272364  3020.0   \n",
       "absences   -0.067602  0.023550  0.000372 -0.114305  -0.022007  3512.0   \n",
       "sd          2.281731  0.076456  0.001325  2.141054   2.438024  3405.0   \n",
       "\n",
       "                Rhat  \n",
       "Intercept   0.999956  \n",
       "failures    0.999768  \n",
       "higher_edu  1.000270  \n",
       "mother_edu  0.999802  \n",
       "studytime   0.999762  \n",
       "father_edu  0.999836  \n",
       "absences    0.999754  \n",
       "sd          1.001020  "
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pm.df_summary(normal_trace)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linear Formula from Bayesian Inference using Mean of Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Grade =  9.20 * Intercept + -1.32 * failures + 1.85 * higher_edu + 0.26 * mother_edu + 0.58 * studytime + 0.03 * father_edu + -0.07 * absences + 0.82 * sd_log__ + 2.28 * sd'"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_formula = 'Grade = '\n",
    "for variable in normal_trace.varnames:\n",
    "    model_formula += ' %0.2f * %s +' % (np.mean(normal_trace[variable]), variable)\n",
    "\n",
    "' '.join(model_formula.split(' ')[:-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate Bayesian Model Using Mean of Model Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evalute the MCMC trace and compare to ml models\n",
    "def evaluate_trace(trace, X_train, X_test, y_train, y_test, model_results):\n",
    "    \n",
    "    # Dictionary of all sampled values for each parameter\n",
    "    var_dict = {}\n",
    "    for variable in trace.varnames:\n",
    "        var_dict[variable] = trace[variable]\n",
    "        \n",
    "    # Results into a dataframe\n",
    "    var_weights = pd.DataFrame(var_dict)\n",
    "    \n",
    "    # Means for all the weights\n",
    "    var_means = var_weights.mean(axis=0)\n",
    "    \n",
    "    # Create an intercept column\n",
    "    X_test['Intercept'] = 1\n",
    "    \n",
    "    # Align names of the test observations and means\n",
    "    names = X_test.columns[1:]\n",
    "    X_test = X_test.ix[:, names]\n",
    "    var_means = var_means[names]\n",
    "    \n",
    "    # Calculate estimate for each test observation using the average weights\n",
    "    results = pd.DataFrame(index = X_test.index, columns = ['estimate'])\n",
    "\n",
    "    for row in X_test.iterrows():\n",
    "        results.ix[row[0], 'estimate'] = np.dot(np.array(var_means), np.array(row[1]))\n",
    "        \n",
    "    # Metrics \n",
    "    actual = np.array(y_test)\n",
    "    errors = results['estimate'] - actual\n",
    "    mae = np.mean(abs(errors))\n",
    "    rmse = np.sqrt(np.mean(errors ** 2))\n",
    "    \n",
    "    print('Model  MAE: {:.4f}\\nModel RMSE: {:.4f}'.format(mae, rmse))\n",
    "    \n",
    "    # Add the results to the comparison dataframe\n",
    "    model_results.ix['Bayesian LR', :] = [mae, rmse]\n",
    "    \n",
    "    plt.figure(figsize=(12, 8))\n",
    "    \n",
    "    # Plot median absolute percentage error of all models\n",
    "    ax = plt.subplot(1, 2, 1)\n",
    "    model_results.sort_values('mae', ascending = True).plot.bar(y = 'mae', color = 'r', ax = ax)\n",
    "    plt.title('Model Mean Absolute Error Comparison'); plt.ylabel('MAE'); \n",
    "    plt.tight_layout()\n",
    "    \n",
    "    # Plot root mean squared error of all models\n",
    "    ax = plt.subplot(1, 2, 2)\n",
    "    model_results.sort_values('rmse', ascending = True).plot.bar(y = 'rmse', color = 'b', ax = ax)\n",
    "    plt.title('Model RMSE Comparison'); plt.ylabel('RMSE')\n",
    "    \n",
    "    return model_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model  MAE: 1.8859\n",
      "Model RMSE: 2.2741\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 42500 characters]\n",
      "text/plain": [
       "<Figure size 864x576 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "all_model_results = evaluate_trace(normal_trace, X_train, X_test, y_train, y_test, results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mae</th>\n",
       "      <th>rmse</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Linear Regression</th>\n",
       "      <td>1.8859</td>\n",
       "      <td>2.27395</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ElasticNet Regression</th>\n",
       "      <td>2.09282</td>\n",
       "      <td>2.53656</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Random Forest</th>\n",
       "      <td>1.85529</td>\n",
       "      <td>2.32467</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Extra Trees</th>\n",
       "      <td>1.97657</td>\n",
       "      <td>2.4871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SVM</th>\n",
       "      <td>1.90262</td>\n",
       "      <td>2.30825</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Gradient Boosted</th>\n",
       "      <td>1.81272</td>\n",
       "      <td>2.22737</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Baseline</th>\n",
       "      <td>2.1761</td>\n",
       "      <td>2.67765</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bayesian LR</th>\n",
       "      <td>1.88591</td>\n",
       "      <td>2.27411</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           mae     rmse\n",
       "Linear Regression       1.8859  2.27395\n",
       "ElasticNet Regression  2.09282  2.53656\n",
       "Random Forest          1.85529  2.32467\n",
       "Extra Trees            1.97657   2.4871\n",
       "SVM                    1.90262  2.30825\n",
       "Gradient Boosted       1.81272  2.22737\n",
       "Baseline                2.1761  2.67765\n",
       "Bayesian LR            1.88591  2.27411"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_model_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Make Predictions from Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test Observations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make a new prediction from the test set and compare to actual value\n",
    "def test_model(trace, test_observation):\n",
    "    \n",
    "    # Print out the test observation data\n",
    "    print('Test Observation:')\n",
    "    print(test_observation)\n",
    "    var_dict = {}\n",
    "    for variable in trace.varnames:\n",
    "        var_dict[variable] = trace[variable]\n",
    "\n",
    "    # Results into a dataframe\n",
    "    var_weights = pd.DataFrame(var_dict)\n",
    "    \n",
    "    # Standard deviation of the likelihood\n",
    "    sd_value = var_weights['sd'].mean()\n",
    "\n",
    "    # Actual Value\n",
    "    actual = test_observation['Grade']\n",
    "    \n",
    "    # Add in intercept term\n",
    "    test_observation['Intercept'] = 1\n",
    "    test_observation = test_observation.drop('Grade')\n",
    "    \n",
    "    # Align weights and test observation\n",
    "    var_weights = var_weights[test_observation.index]\n",
    "\n",
    "    # Means for all the weights\n",
    "    var_means = var_weights.mean(axis=0)\n",
    "\n",
    "    # Location of mean for observation\n",
    "    mean_loc = np.dot(var_means, test_observation)\n",
    "    \n",
    "    # Estimates of grade\n",
    "    estimates = np.random.normal(loc = mean_loc, scale = sd_value,\n",
    "                                 size = 1000)\n",
    "\n",
    "    # Plot all the estimates\n",
    "    plt.figure(figsize(8, 8))\n",
    "    sns.distplot(estimates, hist = True, kde = True, bins = 19,\n",
    "                 hist_kws = {'edgecolor': 'k', 'color': 'darkblue'},\n",
    "                kde_kws = {'linewidth' : 4},\n",
    "                label = 'Estimated Dist.')\n",
    "    # Plot the actual grade\n",
    "    plt.vlines(x = actual, ymin = 0, ymax = 5, \n",
    "               linestyles = '--', colors = 'red',\n",
    "               label = 'True Grade',\n",
    "              linewidth = 2.5)\n",
    "    \n",
    "    # Plot the mean estimate\n",
    "    plt.vlines(x = mean_loc, ymin = 0, ymax = 5, \n",
    "               linestyles = '-', colors = 'orange',\n",
    "               label = 'Mean Estimate',\n",
    "              linewidth = 2.5)\n",
    "    \n",
    "    plt.legend(loc = 1)\n",
    "    plt.title('Density Plot for Test Observation');\n",
    "    plt.xlabel('Grade'); plt.ylabel('Density');\n",
    "    \n",
    "    # Prediction information\n",
    "    print('True Grade = %d' % actual)\n",
    "    print('Average Estimate = %0.4f' % mean_loc)\n",
    "    print('5%% Estimate = %0.4f    95%% Estimate = %0.4f' % (np.percentile(estimates, 5),\n",
    "                                       np.percentile(estimates, 95)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Observation:\n",
      "Grade         12\n",
      "failures       0\n",
      "higher_edu     1\n",
      "mother_edu     2\n",
      "studytime      1\n",
      "father_edu     2\n",
      "absences       8\n",
      "Intercept      1\n",
      "Name: 39, dtype: int64\n",
      "True Grade = 12\n",
      "Average Estimate = 11.6763\n",
      "5% Estimate = 7.8771    95% Estimate = 15.3905\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 49520 characters]\n",
      "text/plain": [
       "<Figure size 576x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_model(normal_trace, X_test.iloc[41])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Observation:\n",
      "Grade         12\n",
      "failures       0\n",
      "higher_edu     1\n",
      "mother_edu     1\n",
      "studytime      2\n",
      "father_edu     1\n",
      "absences       6\n",
      "Intercept      1\n",
      "Name: 2, dtype: int64\n",
      "True Grade = 12\n",
      "Average Estimate = 12.0949\n",
      "5% Estimate = 8.5854    95% Estimate = 15.7475\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 48884 characters]\n",
      "text/plain": [
       "<Figure size 576x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_model(normal_trace, X_test.iloc[16])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Predictions for New Observation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make predictions for a new data point from the model trace\n",
    "def query_model(trace, new_observation):\n",
    "    \n",
    "    # Print information about the new observation\n",
    "    print('New Observation')\n",
    "    print(new_observation)\n",
    "    # Dictionary of all sampled values for each parameter\n",
    "    var_dict = {}\n",
    "    for variable in trace.varnames:\n",
    "        var_dict[variable] = trace[variable]\n",
    "        \n",
    "    # Standard deviation\n",
    "    sd_value = var_dict['sd'].mean()\n",
    "    \n",
    "    # Results into a dataframe\n",
    "    var_weights = pd.DataFrame(var_dict)\n",
    "    \n",
    "    # Align weights and new observation\n",
    "    var_weights = var_weights[new_observation.index]\n",
    "    \n",
    "    # Means of variables\n",
    "    var_means = var_weights.mean(axis=0)\n",
    "    \n",
    "    # Mean for observation\n",
    "    mean_loc = np.dot(var_means, new_observation)\n",
    "    \n",
    "    # Distribution of estimates\n",
    "    estimates = np.random.normal(loc = mean_loc, scale = sd_value,\n",
    "                                 size = 1000)\n",
    "    \n",
    "    # Plot the estimate distribution\n",
    "    plt.figure(figsize(8, 8))\n",
    "    sns.distplot(estimates, hist = True, kde = True, bins = 19,\n",
    "                 hist_kws = {'edgecolor': 'k', 'color': 'darkblue'},\n",
    "                kde_kws = {'linewidth' : 4},\n",
    "                label = 'Estimated Dist.')\n",
    "    # Plot the mean estimate\n",
    "    plt.vlines(x = mean_loc, ymin = 0, ymax = 5, \n",
    "               linestyles = '-', colors = 'orange', linewidth = 2.5)\n",
    "    plt.title('Density Plot for New Observation');\n",
    "    plt.xlabel('Grade'); plt.ylabel('Density');\n",
    "    \n",
    "    # Estimate information\n",
    "    print('Average Estimate = %0.4f' % mean_loc)\n",
    "    print('5%% Estimate = %0.4f    95%% Estimate = %0.4f' % (np.percentile(estimates, 5),\n",
    "                                       np.percentile(estimates, 95)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New Observation\n",
      "Intercept     1\n",
      "absences      1\n",
      "failures      0\n",
      "father_edu    1\n",
      "higher_edu    1\n",
      "mother_edu    4\n",
      "studytime     3\n",
      "dtype: int64\n",
      "Average Estimate = 13.8009\n",
      "5% Estimate = 10.1078    95% Estimate = 17.2885\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 37588 characters]\n",
      "text/plain": [
       "<Figure size 576x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "observation = pd.Series({'Intercept': 1, 'mother_edu': 4, 'failures': 0, \n",
    "                            'higher_edu': 1, 'studytime': 3,\n",
    "                            'father_edu': 1, 'absences': 1})\n",
    "query_model(normal_trace, observation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New Observation\n",
      "Intercept     1\n",
      "absences      4\n",
      "failures      2\n",
      "father_edu    3\n",
      "higher_edu    1\n",
      "mother_edu    2\n",
      "studytime     2\n",
      "dtype: int64\n",
      "Average Estimate = 9.9212\n",
      "5% Estimate = 6.0180    95% Estimate = 13.5235\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 38408 characters]\n",
      "text/plain": [
       "<Figure size 576x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "observation = pd.Series({'Intercept': 1, 'mother_edu': 2, 'failures': 2, \n",
    "                            'higher_edu': 1, 'studytime': 2,\n",
    "                            'father_edu': 3, 'absences': 4})\n",
    "query_model(normal_trace, observation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Variable Effects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to see the effect of changing one variable while holding the others constant, we can use the function `pm.plot_posterior_predictive_glm`. This takes a range of values to use for the variable, a linear model, and a number of samples. The function evaluates the linear model across the range of values for the number of samples. Each time, it draws a different set of parameters from the trace. This gives us an indication of the effect of a single variable and also the uncertainty in the model estimates. To see the effect of a single variable, we hold the others constant at their median values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Examines the effect of changing a single variable\n",
    "# Takes in the name of the variable, the trace, and the data\n",
    "def model_effect(query_var, trace, X):\n",
    "    \n",
    "    # Variables that do not change\n",
    "    steady_vars = list(X.columns)\n",
    "    steady_vars.remove(query_var)\n",
    "    \n",
    "    # Linear Model that estimates a grade based on the value of the query variable \n",
    "    # and one sample from the trace\n",
    "    def lm(value, sample):\n",
    "        \n",
    "        # Prediction is the estimate given a value of the query variable\n",
    "        prediction = sample['Intercept'] + sample[query_var] * value\n",
    "        \n",
    "        # Each non-query variable is assumed to be at the median value\n",
    "        for var in steady_vars:\n",
    "            \n",
    "            # Multiply the weight by the median value of the variable\n",
    "            prediction += sample[var] * X[var].median()\n",
    "        \n",
    "        return prediction\n",
    "    \n",
    "    figsize(6, 6)\n",
    "    \n",
    "    # Find the minimum and maximum values for the range of the query var\n",
    "    var_min = X[query_var].min()\n",
    "    var_max = X[query_var].max()\n",
    "    \n",
    "    # Plot the estimated grade versus the range of query variable\n",
    "    pm.plot_posterior_predictive_glm(trace, eval=np.linspace(var_min, var_max, 100), \n",
    "                                     lm=lm, samples=100, color='blue', \n",
    "                                     alpha = 0.4, lw = 2)\n",
    "    \n",
    "    # Plot formatting\n",
    "    plt.xlabel('%s' % query_var, size = 16)\n",
    "    plt.ylabel('Grade', size = 16)\n",
    "    plt.title(\"Posterior of Grade vs %s\" % query_var, size = 18)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 74520 characters]\n",
      "text/plain": [
       "<Figure size 432x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model_effect('mother_edu', normal_trace, X_train.drop(columns='Grade'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 73648 characters]\n",
      "text/plain": [
       "<Figure size 432x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model_effect('studytime', normal_trace, X_train.drop(columns='Grade'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 88412 characters]\n",
      "text/plain": [
       "<Figure size 432x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model_effect('absences', normal_trace, X_train.drop(columns='Grade'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 64652 characters]\n",
      "text/plain": [
       "<Figure size 432x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model_effect('failures', normal_trace, X_train.drop(columns='Grade'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 105416 characters]\n",
      "text/plain": [
       "<Figure size 432x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model_effect('father_edu', normal_trace, X_train.drop(columns='Grade'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Different Likelihood Prior Distribution\n",
    "\n",
    "We can perform the exact same Bayesian Linear Modeling using a Student's T-distribution as the prior for the data likelihood. A Student's T Distribution has more weight in the tails of the distribution so it is more robust to outliers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 34912 characters]\n",
      "text/plain": [
       "<Figure size 432x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# X values for plotting\n",
    "x = np.linspace(-5, 5, num = 500)\n",
    "\n",
    "# Generate pdf of normal distribution\n",
    "y_norm = scipy.stats.norm.pdf(x)\n",
    "\n",
    "# PDF of t-distribution with 2 degrees of freedom\n",
    "y_t = scipy.stats.t.pdf(x, df = 2)\n",
    "\n",
    "plt.plot(x, y_norm, 'b-', label = 'Normal')\n",
    "plt.plot(x, y_t, 'r-', label = 'T with 2 df')\n",
    "plt.legend(prop = {'size': 18}, loc = 1)\n",
    "plt.xlabel('x'); plt.ylabel('Probability'); plt.title('Normal vs T Distribution');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build Model and Perform Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Auto-assigning NUTS sampler...\n",
      "Initializing NUTS using jitter+adapt_diag...\n",
      "Sequential sampling (2 chains in 1 job)\n",
      "NUTS: [lam_log__, absences, father_edu, studytime, mother_edu, higher_edu, failures, Intercept]\n",
      "100%|██████████| 2500/2500 [00:26<00:00, 95.85it/s]\n",
      "100%|██████████| 2500/2500 [00:23<00:00, 104.61it/s]\n"
     ]
    }
   ],
   "source": [
    "# Context for model\n",
    "with pm.Model() as t_model:\n",
    "    # Family is Student's T in this case\n",
    "    family = pm.glm.families.StudentT(df = 2)\n",
    "    \n",
    "    # Formula, data, family\n",
    "    pm.GLM.from_formula(formula, data = X_train, family = family)\n",
    "    \n",
    "    # Sample from the posterior \n",
    "    t_trace = pm.sample(draws=2000, tune=500, njobs = -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 364492 characters]\n",
      "text/plain": [
       "<Figure size 1008x1166.4 with 16 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_trace(t_trace);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the Model\n",
    "\n",
    "__I'm not sure I implemented this part correclty!__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_model_t(trace, test_observation):\n",
    "    \n",
    "    var_dict = {}\n",
    "    for variable in trace.varnames:\n",
    "        var_dict[variable] = trace[variable]\n",
    "\n",
    "    # Results into a dataframe\n",
    "    var_weights = pd.DataFrame(var_dict)\n",
    "\n",
    "    # Actual Value\n",
    "    actual = test_observation['Grade']\n",
    "    \n",
    "    # Add in intercept term\n",
    "    test_observation['Intercept'] = 1\n",
    "    test_observation = test_observation.drop('Grade')\n",
    "    \n",
    "    # Align weights and test observation\n",
    "    var_weights = var_weights[test_observation.index]\n",
    "\n",
    "    # Means for all the weights\n",
    "    var_means = var_weights.mean(axis=0)\n",
    "\n",
    "    # Location of mean for observation\n",
    "    mean_loc = np.dot(var_means, test_observation)\n",
    "    \n",
    "    # Estimates of grade\n",
    "    estimates = mean_loc + np.random.standard_t(df = 2, size = 1000)\n",
    "\n",
    "    plt.figure(figsize(8, 8))\n",
    "    sns.distplot(estimates, hist = True, kde = True, bins = 19,\n",
    "                 hist_kws = {'edgecolor': 'k', 'color': 'darkblue'},\n",
    "                kde_kws = {'linewidth' : 4},\n",
    "                label = 'Estimated Dist.')\n",
    "    plt.vlines(x = actual, ymin = 0, ymax = 5, \n",
    "               linestyles = '--', colors = 'red',\n",
    "               label = 'True Grade',\n",
    "              linewidth = 2.5)\n",
    "    plt.vlines(x = mean_loc, ymin = 0, ymax = 5, \n",
    "               linestyles = '-', colors = 'orange',\n",
    "               label = 'Mean Estimate',\n",
    "              linewidth = 2.5)\n",
    "    \n",
    "    plt.legend(loc = 1)\n",
    "    plt.title('Density Plot for Test Observation');\n",
    "    plt.xlabel('Grade'); plt.ylabel('Density');\n",
    "    \n",
    "    print('True Grade = %d' % actual)\n",
    "    print('Average Estimate = %0.4f' % mean_loc)\n",
    "    print('5%% Estimate = %0.4f    95%% Estimate = %0.4f' % (np.percentile(estimates, 5),\n",
    "                                       np.percentile(estimates, 95)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True Grade = 17\n",
      "Average Estimate = 12.2513\n",
      "5% Estimate = 9.2249    95% Estimate = 14.8319\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 38280 characters]\n",
      "text/plain": [
       "<Figure size 576x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_model_t(t_trace, X_test.iloc[60])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Observation:\n",
      "Grade         17\n",
      "failures       0\n",
      "higher_edu     1\n",
      "mother_edu     4\n",
      "studytime      1\n",
      "father_edu     4\n",
      "absences       3\n",
      "Intercept      1\n",
      "Name: 595, dtype: int64\n",
      "True Grade = 17\n",
      "Average Estimate = 12.6022\n",
      "5% Estimate = 8.9748    95% Estimate = 16.3461\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 48960 characters]\n",
      "text/plain": [
       "<Figure size 576x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_model(normal_trace, X_test.iloc[60])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## New Observation Predictions\n",
    "\n",
    "__I'm not sure about these predictions!__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "def query_model_t(trace, new_observation):\n",
    "    \n",
    "    # Dictionary of all sampled values for each parameter\n",
    "    var_dict = {}\n",
    "    for variable in trace.varnames:\n",
    "        var_dict[variable] = trace[variable]\n",
    "        \n",
    "\n",
    "    \n",
    "    # Results into a dataframe\n",
    "    var_weights = pd.DataFrame(var_dict)\n",
    "    \n",
    "    # Align weights and new observation\n",
    "    var_weights = var_weights[new_observation.index]\n",
    "    # Means of variables\n",
    "    var_means = var_weights.mean(axis=0)\n",
    "    \n",
    "    # Mean for observation\n",
    "    mean_loc = np.dot(var_means, new_observation)\n",
    "    \n",
    "    # Distribution of estimates\n",
    "    estimates = mean_loc +  np.random.standard_t(df = 2, size = 1000)\n",
    "    \n",
    "\n",
    "    plt.figure(figsize(8, 8))\n",
    "    sns.distplot(estimates, hist = True, kde = True, bins = 19,\n",
    "                 hist_kws = {'edgecolor': 'k', 'color': 'darkblue'},\n",
    "                kde_kws = {'linewidth' : 4},\n",
    "                label = 'Estimated Dist.')\n",
    "    plt.vlines(x = mean_loc, ymin = 0, ymax = 5, \n",
    "               linestyles = '-', colors = 'orange', linewidth = 2.5)\n",
    "    plt.title('Density Plot for New Observation');\n",
    "    plt.xlabel('Grade'); plt.ylabel('Density');\n",
    "    \n",
    "    print('Average Estimate = %0.4f' % mean_loc)\n",
    "    print('5%% Estimate = %0.4f    95%% Estimate = %0.4f' % (np.percentile(estimates, 5),\n",
    "                                       np.percentile(estimates, 95)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New Observation\n",
      "Intercept     1\n",
      "absences      1\n",
      "failures      0\n",
      "father_edu    1\n",
      "higher_edu    1\n",
      "mother_edu    4\n",
      "studytime     3\n",
      "dtype: int64\n",
      "Average Estimate = 13.8009\n",
      "5% Estimate = 10.1702    95% Estimate = 17.3781\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 38432 characters]\n",
      "text/plain": [
       "<Figure size 576x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "observation = pd.Series({'Intercept': 1, 'mother_edu': 4, 'failures': 0, \n",
    "                            'higher_edu': 1, 'studytime': 3,\n",
    "                            'father_edu': 1, 'absences': 1})\n",
    "query_model(normal_trace, observation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Estimate = 13.3678\n",
      "5% Estimate = 10.5739    95% Estimate = 16.3109\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 28332 characters]\n",
      "text/plain": [
       "<Figure size 576x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "observation = pd.Series({'Intercept': 1, 'mother_edu': 4, 'failures': 0, \n",
    "                            'higher_edu': 1, 'studytime': 3,\n",
    "                            'father_edu': 1, 'absences': 1})\n",
    "query_model_t(t_trace, observation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is some variation between the t-distribution estimates and those from the normal distribution for the data likelihood.Choosing appropriate priors is one of the hardest aspect of Bayesian Modeling, but we can get around that by having more data. As the amount of data the model learns from increases, the prior has less of an effect because each time the posterior is updated based on the new data. Essentially machine learning models perform inference with no priors, basing the final model entirely on the data. In the case of limited samples, Bayesian Inference can be a better method for building models because it provides a reasonable estimate in situations with few data points (as long as the prior is reasonable)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusions\n",
    "\n",
    "In this notebook we looked at using Bayesian Linear Regression to predict student performance based on six factors. Rather than specify probabilities for the Bayesian network which is basically impossible for continuous variables, we framed the problem as a machine learning task. In addition to the standard machine learning models that learn from observations, we also used Bayesian Linear Regression to create a model mapping the features (student characteristics) to the targets (final grade). The advantages of Bayesian Linear Regression are that if we use sensible priors, we can still get a decent estimate with few samples, and the final weights are not a single number, but a distribution componsed of every sample drawn during the sampling run. We can then make predictions using all the sampled weights to form a distribution of expected values rather than a single answer. \n",
    "\n",
    "The Bayesian  Linear Regression did not perform as well as the other methods in terms of the two metrics we choose. This might not be the ideal case for a Bayesian inference approach but we saw that Bayesian Linear Regression produced intuitive estimates for the model weights and gave predictions for new students that align with our expectations for the factors influencing student performance. To summarize, although Bayesian Linear Regression did not outperform the standard machine learning methods, it gave us a chance to learn another tool for use in evaluating and making sense of data. It's always a positive to have more skills that you can deploy as needed!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
