{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overfitting vs. Underfitting\n",
    "\n",
    "Exploring a fundamental problem in modeling. This notebook will look at a simple example showing the problem of overfitting and underfitting as well as how to address it via cross-validation. This example is based on the [scikit-learn exercise](http://scikit-learn.org/stable/auto_examples/model_selection/plot_underfitting_overfitting.html). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports \n",
    "\n",
    "We will use numpy and pandas, two of the most common libraries for data manipulation. We use scikit-learn, a popular machine learning library, for creating and evaluating the models. Matplotlib is used for model visualization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Numpy and pandas as usual\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# Scikit-Learn for fitting models\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "# For plotting in the notebook\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# Default parameters for plots\n",
    "matplotlib.rcParams['font.size'] = 12\n",
    "matplotlib.rcParams['figure.titlesize'] = 16\n",
    "matplotlib.rcParams['figure.figsize'] = [9, 7]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate a relationship\n",
    "\n",
    "First, we need a \"true\" relationship. We define a curve, in this case a sine curve to serve as our process that generates the data. As the real-world is never perfectly celan however, we also need to add some noise into the observations. This is done by adding a small random number to each value. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the random seed for reproducible results\n",
    "np.random.seed(42)\n",
    "\n",
    "# \"True\" generating function representing a process in real life\n",
    "def true_gen(x):\n",
    "    y = np.sin(1.2 * x * np.pi) \n",
    "    return(y)\n",
    "\n",
    "# x values and y value with a small amount of random noise\n",
    "x = np.sort(np.random.rand(120))\n",
    "y = true_gen(x) + 0.1 * np.random.randn(len(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training and Testing Sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Random indices for creating training and testing sets\n",
    "random_ind = np.random.choice(list(range(120)), size = 120, replace=False)\n",
    "xt = x[random_ind]\n",
    "yt = y[random_ind]\n",
    "\n",
    "# Training and testing observations\n",
    "train = xt[:int(0.7 * len(x))]\n",
    "test = xt[int(0.7 * len(x)):]\n",
    "\n",
    "y_train = yt[:int(0.7 * len(y))]\n",
    "y_test = yt[int(0.7 * len(y)):]\n",
    "\n",
    "# Model the true curve\n",
    "x_linspace = np.linspace(0, 1, 1000)\n",
    "y_true = true_gen(x_linspace)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 32324 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1e54d1b2828>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Visualize observations and true curve\n",
    "plt.plot(train, y_train, 'ko', label = 'Train'); \n",
    "plt.plot(test, y_test, 'ro', label = 'Test')\n",
    "plt.plot(x_linspace, y_true, 'b-', linewidth = 2, label = 'True Function')\n",
    "plt.legend()\n",
    "plt.xlabel('x'); plt.ylabel('y'); plt.title('Data');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Polynomial Model\n",
    "\n",
    "We want to try and capture the data using a polynomial function. A polynomial is defined by the degree, or the highest power to for the x-values. A line has a degree of 1 because it is of the form $y = b_1*x + b_0$ where $b_1$ is the slope and $b_0$ is the intercept. A third degree polynomial would have the form $y = b_3 * x^3 + b_2 * x^2 + b_1 * x + b_0$ and so on. The higher the degree of the polynomial, the more flexible the model. A more flexible model is prone to overfitting because it can can \"bend\" to follow the training data.  \n",
    "\n",
    "The following function creates a polynomial with the specified number of degrees and plots the results. We can use these results to determine the optimal degrees to achieve the right balance between over and underfitting. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def fit_poly(train, y_train, test, y_test, degrees, plot='train', return_scores=False):\n",
    "    \n",
    "    # Create a polynomial transformation of features\n",
    "    features = PolynomialFeatures(degree=degrees, include_bias=False)\n",
    "    \n",
    "    # Reshape training features for use in scikit-learn and transform features\n",
    "    train = train.reshape((-1, 1))\n",
    "    train_trans = features.fit_transform(train)\n",
    "    \n",
    "    # Create the linear regression model and train\n",
    "    model = LinearRegression()\n",
    "    model.fit(train_trans, y_train)\n",
    "    \n",
    "    # Calculate the cross validation score\n",
    "    cross_valid = cross_val_score(model, train_trans, y_train, scoring='neg_mean_squared_error', cv = 5)\n",
    "    \n",
    "    # Training predictions and error\n",
    "    train_predictions = model.predict(train_trans)\n",
    "    training_error = mean_squared_error(y_train, train_predictions)\n",
    "    \n",
    "    # Format test features\n",
    "    test = test.reshape((-1, 1))\n",
    "    test_trans = features.fit_transform(test)\n",
    "    \n",
    "    # Test set predictions and error\n",
    "    test_predictions = model.predict(test_trans)\n",
    "    testing_error = mean_squared_error(y_test, test_predictions)\n",
    "    \n",
    "    # Find the model curve and the true curve\n",
    "    x_curve = np.linspace(0, 1, 100)\n",
    "    x_curve = x_curve.reshape((-1, 1))\n",
    "    x_curve_trans = features.fit_transform(x_curve)\n",
    "    \n",
    "    # Model curve\n",
    "    model_curve = model.predict(x_curve_trans)\n",
    "    \n",
    "    # True curve\n",
    "    y_true_curve = true_gen(x_curve[:, 0])\n",
    "    \n",
    "    # Plot observations, true function, and model predicted function\n",
    "    if plot == 'train':\n",
    "        plt.plot(train[:, 0], y_train, 'ko', label = 'Observations')\n",
    "        plt.plot(x_curve[:, 0], y_true_curve, linewidth = 4, label = 'True Function')\n",
    "        plt.plot(x_curve[:, 0], model_curve, linewidth = 4, label = 'Model Function')\n",
    "        plt.xlabel('x'); plt.ylabel('y')\n",
    "        plt.legend()\n",
    "        plt.ylim(-1, 1.5); plt.xlim(0, 1)\n",
    "        plt.title('{} Degree Model on Training Data'.format(degrees))\n",
    "        plt.show()\n",
    "        \n",
    "    elif plot == 'test':\n",
    "        # Plot the test observations and test predictions\n",
    "        plt.plot(test, y_test, 'o', label = 'Test Observations')\n",
    "        plt.plot(x_curve[:, 0], y_true_curve, 'b-', linewidth = 2, label = 'True Function')\n",
    "        plt.plot(test, test_predictions, 'ro', label = 'Test Predictions')\n",
    "        plt.ylim(-1, 1.5); plt.xlim(0, 1)\n",
    "        plt.legend(), plt.xlabel('x'), plt.ylabel('y'); plt.title('{} Degree Model on Testing Data'.format(degrees)), plt.show();\n",
    "    \n",
    "    # Return the metrics\n",
    "    if return_scores:\n",
    "        return training_error, testing_error, -np.mean(cross_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Try Model with Different Degrees"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Degrees = 1 -> Underfitting\n",
    "\n",
    "In this case, a linear model cannot accurate learn the relationship between x and y and will underfit the data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 37844 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1e54d1d5390>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fit_poly(train, y_train, test, y_test, degrees = 1, plot='train')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 33112 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1e54d1d51d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fit_poly(train, y_train, test, y_test, degrees = 1, plot='test')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Degrees = 25 -> Overfitting\n",
    "\n",
    "We can go in the completely opposite direction and create a model that overfits the data. This model has too much flexibility and learns the training data too closely. As the training data has some amount of noise, it will end up capturing that noise and will be misled by that noise when it tries to make predictions on the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 43540 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1e5465f5588>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fit_poly(train, y_train, test, y_test, plot='train', degrees = 25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 33708 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1e54b8b56d8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fit_poly(train, y_train, test, y_test, degrees=25, plot='test')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Degrees = 5 -> Balanced Model\n",
    "\n",
    "Now that we have seen the two extremes, we can take a look at a model that does a good job of both accounting for the data while not following it too closely. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 38256 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1e54d108b00>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fit_poly(train, y_train, test, y_test, plot='train', degrees = 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 32396 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1e54d1bc518>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fit_poly(train, y_train, test, y_test, degrees=5, plot='test')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cross Validation\n",
    "\n",
    "To pick the optimal model, we need to use a validation set. Cross validation is even better than a single validation set because it uses numerous validation sets created from the training data. In this case, we are using 5 different validation sets. The model that performs best on the cross validation is usually the optimal model because it has shown that it can learn the relationships while not overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Range of model degrees to evaluate\n",
    "degrees = [int(x) for x in np.linspace(1, 40, 40)]\n",
    "\n",
    "# Results dataframe\n",
    "results = pd.DataFrame(0, columns = ['train_error', 'test_error', 'cross_valid'], index = degrees)\n",
    "\n",
    "# Try each value of degrees for the model and record results\n",
    "for degree in degrees:\n",
    "    degree_results = fit_poly(train, y_train, test, y_test, degree, plot=False, return_scores=True)\n",
    "    results.ix[degree, 'train_error'] = degree_results[0]\n",
    "    results.ix[degree, 'test_error'] = degree_results[1]\n",
    "    results.ix[degree, 'cross_valid'] = degree_results[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10 Lowest Cross Validation Errors\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>degrees</th>\n",
       "      <th>cross_valid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4</td>\n",
       "      <td>0.010549</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>0.010637</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7</td>\n",
       "      <td>0.010665</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6</td>\n",
       "      <td>0.010887</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>0.011182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>3</td>\n",
       "      <td>0.011695</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>9</td>\n",
       "      <td>0.011757</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>11</td>\n",
       "      <td>0.011769</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>10</td>\n",
       "      <td>0.011902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>12</td>\n",
       "      <td>0.012642</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   degrees  cross_valid\n",
       "0        4     0.010549\n",
       "1        5     0.010637\n",
       "2        7     0.010665\n",
       "3        6     0.010887\n",
       "4        8     0.011182\n",
       "5        3     0.011695\n",
       "6        9     0.011757\n",
       "7       11     0.011769\n",
       "8       10     0.011902\n",
       "9       12     0.012642"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('10 Lowest Cross Validation Errors\\n')\n",
    "train_eval = results.sort_values('cross_valid').reset_index(level=0).rename(columns={'index': 'degrees'})\n",
    "train_eval.ix[:,['degrees', 'cross_valid']] .head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Minimum Cross Validation Error occurs at 4 degrees.\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 33044 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1e54d4c7240>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(results.index, results['cross_valid'], 'go-', ms=6)\n",
    "plt.xlabel('Degrees'); plt.ylabel('Cross Validation Error'); plt.title('Cross Validation Results');\n",
    "plt.ylim(0, 0.2);\n",
    "print('Minimum Cross Validation Error occurs at {} degrees.\\n'.format(int(np.argmin(results['cross_valid']))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Final Model\n",
    "\n",
    "The model with the lowest cross validation error had four degrees. Therefore, we will use a 4th degree polynomial for the final model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 38836 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1e54d1e72e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fit_poly(train, y_train, test, y_test, degrees=4, plot='train')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 32672 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1e54d1d54e0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fit_poly(train, y_train, test, y_test, degrees=4, plot='test')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate Models \n",
    "\n",
    "The next step is to examine the scores for the models. We will use a range of values to see how the performance on the training and testing set compares. A model with much lower errors on the training data than the testing data is overfit. A model with high error on the training data (which will lead to high testing error as well) is underfitting because it does not even learn the training data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Quantitative Comparison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10 Lowest Training Errors\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>degrees</th>\n",
       "      <th>train_error</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>35</td>\n",
       "      <td>0.006121</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>36</td>\n",
       "      <td>0.006142</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>40</td>\n",
       "      <td>0.006192</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>39</td>\n",
       "      <td>0.006272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>31</td>\n",
       "      <td>0.006367</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>27</td>\n",
       "      <td>0.006405</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>32</td>\n",
       "      <td>0.006412</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>28</td>\n",
       "      <td>0.006432</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>33</td>\n",
       "      <td>0.006441</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>29</td>\n",
       "      <td>0.006467</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   degrees  train_error\n",
       "0       35     0.006121\n",
       "1       36     0.006142\n",
       "2       40     0.006192\n",
       "3       39     0.006272\n",
       "4       31     0.006367\n",
       "5       27     0.006405\n",
       "6       32     0.006412\n",
       "7       28     0.006432\n",
       "8       33     0.006441\n",
       "9       29     0.006467"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('10 Lowest Training Errors\\n')\n",
    "train_eval = results.sort_values('train_error').reset_index(level=0).rename(columns={'index': 'degrees'})\n",
    "train_eval.ix[:,['degrees', 'train_error']] .head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10 Lowest Testing Errors\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>degrees</th>\n",
       "      <th>test_error</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5</td>\n",
       "      <td>0.009482</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6</td>\n",
       "      <td>0.010215</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4</td>\n",
       "      <td>0.010483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8</td>\n",
       "      <td>0.010609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7</td>\n",
       "      <td>0.010618</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>9</td>\n",
       "      <td>0.010754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>14</td>\n",
       "      <td>0.011216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>10</td>\n",
       "      <td>0.011321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>11</td>\n",
       "      <td>0.011342</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>13</td>\n",
       "      <td>0.011387</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   degrees  test_error\n",
       "0        5    0.009482\n",
       "1        6    0.010215\n",
       "2        4    0.010483\n",
       "3        8    0.010609\n",
       "4        7    0.010618\n",
       "5        9    0.010754\n",
       "6       14    0.011216\n",
       "7       10    0.011321\n",
       "8       11    0.011342\n",
       "9       13    0.011387"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('10 Lowest Testing Errors\\n')\n",
    "train_eval = results.sort_values('test_error').reset_index(level=0).rename(columns={'index': 'degrees'})\n",
    "train_eval.ix[:,['degrees', 'test_error']] .head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Visual Comparison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 32724 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1e54d422518>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Minimum Training Error occurs at 35 degrees.\n",
      "Minimum Testing Error occurs at 5 degrees.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "plt.plot(results.index, results['train_error'], 'b-o', ms=6, label = 'Training Error')\n",
    "plt.plot(results.index, results['test_error'], 'r-*', ms=6, label = 'Testing Error')\n",
    "plt.legend(loc=2); plt.xlabel('Degrees'); plt.ylabel('Mean Squared Error'); plt.title('Training and Testing Curves');\n",
    "plt.ylim(0, 0.05); plt.show()\n",
    "\n",
    "print('\\nMinimum Training Error occurs at {} degrees.'.format(int(np.argmin(results['train_error']))))\n",
    "print('Minimum Testing Error occurs at {} degrees.\\n'.format(int(np.argmin(results['test_error']))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
